<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Xandr Engineering on Medium]]></title>
        <description><![CDATA[Stories by Xandr Engineering on Medium]]></description>
        <link>https://medium.com/@Xandr.Tech?source=rss-f77f4074e4d6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*9SjZpu_T0IQ0u24ZXBgZUA.png</url>
            <title>Stories by Xandr Engineering on Medium</title>
            <link>https://medium.com/@Xandr.Tech?source=rss-f77f4074e4d6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 20:18:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@Xandr.Tech/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Malicious Content Identification]]></title>
            <link>https://medium.com/xandr-tech/malicious-content-identification-f0c0359b5c75?source=rss-f77f4074e4d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0c0359b5c75</guid>
            <category><![CDATA[random]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[hackathons]]></category>
            <dc:creator><![CDATA[Xandr Engineering]]></dc:creator>
            <pubDate>Fri, 10 Aug 2018 13:23:36 GMT</pubDate>
            <atom:updated>2018-08-10T13:23:36.904Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Hackathon Success!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yU9QdL2OKFtOqJ4UeVxetg.jpeg" /></figure><p>This post is written by <a href="https://www.linkedin.com/in/eliot-huh-a97b84165/"><em>Eliot Huh</em></a><em>, </em><a href="https://www.linkedin.com/in/keithpallo/"><em>Keith Pallo</em></a><em>, </em><a href="https://www.linkedin.com/in/chenny-ren-523a1215a/"><em>Chenny Ren</em></a><em>, and </em><a href="https://www.linkedin.com/in/hartek-sabharwal/"><em>Hartek Sabharwal</em></a><em>. Each was a member of Team Early Bird — the submission that won the AppNexus Challenge at the 2018 New York DeveloperWeek Hackathon. DeveloperWeek Hackathon’s represent the nation’s largest challenge-driven hackathon series and are hosted several times throughout the year around the country. The 2018 NY Hackathon hosted over 35 different teams with 150+ hackers. In addition to winning the AppNexus challenge, Team Early Bird also placed in the Top 5 Overall Teams at the Hackathon.</em></p><h3>Forming the Team</h3><p>If you have been to any widely “open” Hackathon before, you know that the first hour at these events is one of the most interesting sights for anyone to behold. Given that directions and challenges are usually posted ahead of time, the environment created by the diversity of participants is truly amazing. One is likely to find a group of individuals who just met standing adjacent to a dedicated team planted at a workstation. The members of the first group may be trying to determine if an idea might work while the others may be ceremoniously licking their chops as they begin to implement a project that had been in ideation for days.</p><p>On June 19th at 9:00 a.m. on a Saturday, the scene at the Brooklyn Expo Center was no different. This was New York’s much anticipated DeveloperWeek Hackathon, and the chaos created by hundreds of hackers trying to either find a project or feverishly start making progress was coming to a peak. This particular hackathon had several different challenges that hackers could participate in, which is not always the case. Hence, conversations focused on what challenges one would like to participate in. From these conversations our team quickly emerged — we were all extremely interested in working on the Malicious Content challenge presented by AppNexus.</p><p>The prompt was relatively straightforward, but by no means simple. Given a set of malicious creatives (image advertisements with dubious intentions), find a way to determine if any newly presented advertisement is malicious or not. With two CS undergrads entering their second year at university, one member entering a Master’s Program in Cybersecurity, and another entering a Master’s Program in Artificial Intelligence, we were all very excited for a variety of reasons. Not only was there a clearly defined and objectively measured goal, the latitude of approaches we could take was huge! This challenge also presented a way to use machine learning at the intersection of cyber-security, and determining “bad content” represents one of the most pervasive problems facing several leading technology companies today. Also, we were proud to be working with an innovative and invested partner. While researching the challenge we found several blog posts and white papers presented by AppNexus on this topic, <a href="https://www.appnexus.com/sites/default/files/whitepapers/appnexus_viralcontentwhitepaper_december2017.pdf">including one specifically on fraudulent ads</a>. Brought together by our enthusiasm, we forged on to accomplishing our task.</p><h3>Challenge Overview</h3><p>Our first step was to examine the data. The challenge presented two distinct types of malicious content — Misleading Claims and Tabloid Advertorials. Their definitions and an example of each are as follows:</p><p><strong>Tabloid Advertorial</strong>: Images where the content of the creative does not match that of the landing page. In other words, it would be used to trick the user into clicking something.</p><p><strong>Misleading Claims</strong>: The claim purported by the creative is unsubstantiated or provably false. In other words, claims that simply aren’t true. The most classic example would be Snake Oil — a magical mixture that can cure all ailments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jm5xiJJPEbmT9VPkF46kxQ.png" /></figure><p>Given a set of about two hundred ads labeled as either Tabloid Advertorial or Misleading Claim, we began to research quantitative methods that would create representations of each advertisement. Almost immediately, it became clear that we would need to use machine learning. Any possible representation which we could define ourselves simply wouldn’t be good enough to achieve our desired accuracy in finding novel malicious advertisements. Machine learning would alleviate this challenge — instead of having to explicitly tell our algorithm what to analyze, our program would learn what to look for by examining the dataset. The methods we researched seemed promising and readily available. However, a few questions quickly arose no matter what kind of method we were thinking of applying. First, regardless of the desired ML algorithm we implemented, almost all of them would need a comparable set of non-malicious ads. We would come to call these “quality advertisements”. These ads would not only help in training our system, but they would also be useful in testing. Additionally, each model we researched had several drawbacks and watch-outs listed. Which one would be best? Why would we use one over the other for this specific purpose?</p><h3>Project Ideation</h3><p>With these core questions floating in our minds, we decided to take action and find some “quality ads”, or novel advertisements that were not malicious. This seemingly easy task proved rather difficult! Almost every which way we tried, it seems that querying for “Real Ads” actually produced image results of “Fake Ads” given the recent controversy around this area. This led us to think: What exactly is Malicious Content? Our team tried a traditional brainstorm, and a word cloud resulting from our conversation looked like someone had simply randomly picked terms from a dictionary. Phrases like “Big Text”, “Theft”, and “Large percentages” were mixed together in a way that was seemingly unrelated. Stumped, we thought hard and came up with two unique methods for achieving our goal.</p><p>First — we noticed that querying for ads from Fortune 500 companies produced a large number of high quality advertisements. However, we did not want to introduce bias into our training data by only selecting content from companies that we knew. So, we turned to a random walk method of sorts — we used a specific online <a href="http://www.domainconverters.com/calc/random-company-generator/#calc">company name generator</a> to create a list of major corporations. From there we used a <a href="https://chrome.google.com/webstore/detail/fatkun-batch-download-ima/nnjjahlikiabnchcpehcpkdeckfgnohf">Google Chrome extension</a> to download a large sample of that company’s advertisements, while also removing any ads that still seemed potentially malicious. We then repeated this process for several companies.</p><p>Additionally, we thought of mediums that were not likely to produce malicious content due to their nature. So, we sourced images from providers that have a much lower sample of advertisements with higher scrutiny — Print Advertising! This idea came to us when recollecting about some of our favorite shows, and thinking about the opening episode of the hit “Mad Men”. Throughout the episode Don Draper is stumped on creating a new advertisement campaign for a cigarette company. This was particularly challenging given that significant recent research had revealed the negative effects of smoking. If an advertiser such as Mr. Draper was going to think that hard about one ad, then surely those presented were going to be within our quality bucket. Using the same method as above we sourced an additional set of images further diversifying our “quality ads”.</p><p>Now, the next step was to partition our data into training and test sets and apply a machine learning algorithm. We thought that a use case might help us in determining which ML algorithm, or which set of ML algorithms to use. After searching for some time, we realized that we had just personally encountered a problem that could be solved — creating a custom definition of malicious content! Since malicious content can mean so many different things in different contexts, we wanted to create an intuitive system that would allow any user to “tune” their own definition.</p><p>This system would have several potential use cases. One example would be for an application, such as Pinterest, that would allow users to upload large numbers of images which could potentially be undesirable. An additional use case would be for a tool that an ad provider, someone like AppNexus, could present to sources that show ad content (publishers). These sources could then tune their own definition of malicious content to what their individual business needs were.</p><p>For example, an independent news outlet may want to rely on an entirely different definition of malicious content than a retail clothing startup simply due to the vastly different audiences that each is trying to serve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/0*a6d2pzR0YrsEcpsk" /></figure><p>In order to create this tool, our approach would be to combine two different ML systems to detect malicious content. First, we would would train two distinct ML algorithms which would independently determine if new content was either quality or malicious. Then, we run both algorithms on each new image, and combine the results. The system would fail content if either marked an image as malicious — effectively creating the most conservative content administrator based upon both sets of results (statistically speaking, the union of the two sets). This would be the default definition of malicious content. From here, a user would then be able to select what types of images they wanted to allow from the set of advertisements that previously failed on only one of our ML algorithms (now the disjoint union of the two sets). Finally, both ML algorithms would independently retrain themselves with the new user input, creating a definition of undesirable content according to that specifics user’s inputs.</p><p>Hence, our system would intuitively figure out what advertisements the end user would want to catch — and it would do so extremely quickly.</p><p><strong>Step 0</strong>: Create two distinct ML algorithm’s to determine malicious content. Then, test new creatives using both algorithms independently and create most conservative result</p><p><strong>Step 1</strong>: Allow users to decide on photo’s that are not malicious if they failed on one ML system</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GxSmXayyPDcSW6me" /></figure><p><strong>Step 2</strong>: Retrain original ML systems based on user input</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/0*BlD7oxG3NVpo_xq_" /></figure><p>We called ourselves Team Early Bird — we wanted to not only allow those using our tool to catch worms, which in our case was unwanted advertisement content, but we would help users get the RIGHT worms based on their own individual goals.</p><p>With this in mind, we turned our focus to creating the ML algorithms that would independently determine malicious content. After looking at the different pro’s and con’s, we decided on a text-based system and a purely image-based classifier to create a strong mix of results. The thinking behind this was that our intuition around a text-based system was relatively strong — the ads were likely to include some “look here” words. However, we did not have a good idea of what an image classifier would really use to distinguish these ads. Obviously the system would develop pattern detection, but we scratched our heads when really thinking hard about it. We hoped that combining these two vastly different methods would lead us to some interesting results — so we pressed onward.</p><h3>Fundamental System 1 — Text Analysis</h3><p>As humans, our first impression of malicious intent generally came from the generic, spammy text in the ads, but we were not sure how this would come into play in the analysis. First, we extracted the text from the images using Tesseract in Python, one of the best open source optical character recognition (OCR) engines available. However, it alone failed to read the ads accurately enough to generate worthwhile data. The strings were not only unreliable, but many images did not produce a single fully formed word. Huddled around one laptop, we struggled to comprehend how any analysis would use the sparse phrases such as “Look,,, “ and “ st0Ck going upwrd,” almost reminiscent of early 2000’s texting. To fix it, we programmatically resized and enhanced the images before processing. Along with some additional text cleaning, such as only allowing ASCII characters, our output improved substantially.</p><p>Our next move was to extract useful features from the text using natural language processing (NLP), a rapidly developing field in CS which allows computers to understand, on some level, human language. One technique in NLP is sentiment analysis, which takes a sentence and outputs whether it is mostly positive, negative, or neutral. Generally, such a program uses a database of positive and negative words and phrases, while also keeping track of sentence structure and negations. A staggeringly effective implementation is included in the Stanford CoreNLP library, and it is the one deployed in our model. Our hypothesis was that malicious ads would use stronger, more forceful language to compel users to click, while non-malicious ads would use more neutral language to better deliver a message and avoid negative language to preserve positive associations with the brand.</p><p>Having spent hours setting up our development environment, fiddling around with data, reading blog posts and research articles, we were all forced to go home. Standing on the R train platform, we overheard two separate teams mention both Tesseract and NLP, so the best team was going to be the one with the smartest implementation and deepest optimization, not the one that threw out the most buzzwords and library names. Our subterranean reconnaissance further confirmed that we would all be working late into the night.</p><p>The pure sentiment analysis turned out to be fairly effective. The data showed that 62% of non-malicious ads were neutral in sentiment, compared to just 22% of tabloid and misleading ads. We were afraid that the program would get the sentiments wrong since the core methodology of NLP is far from perfect and often misses the nuances of real human language, like sarcasm. However, the claims in our set of malicious ads were simple enough that this did not happen. An ML model using the polarity of sentiment to classify ads as either malicious or non-malicious achieved about 65% accuracy. However, sentiment analysis was less effective in differentiating the two types of malicious ads, an additional goal of the challenge, so finer-grained features were needed.</p><p>Looking at the results we thought to ourselves, if we were the computer, what else would we want? All of the analysis we had done simplified the ads into one basic number, while any robust ML model would need lots of data and lots of columns. We needed metrics that were just as substantive but with more granularity. After reviewing the diverse set of coursework accumulated between us we landed on the following; “How about some basic statistics?” Other than just the sentiment, it would likely be useful to know things like the most common words and how often they appeared compared to peers in different ad types. This led us to complete a simple n-gram analysis. First, we calculated the most common words present in both types of malicious ads. This required tokenization of the text and the removal of stop words, which are phrases such as “the” or “is,” that are common in advertisements of all kinds but semantically are virtually meaningless. This was done with NLTK, another NLP library. We learned about this from an online Kaggle competition dealing with a similar task. We repeated the process for tabloid and misleading ads separately.</p><p>This was an effective strategy because although both types of ads were malicious, the language each used to draw in users was distinct. Tabloid ads commonly used words like “rumors,” “reason,” and “confirms”, while misleading ads used words like “trick,” “free,” and “millions.” For each ad, we also computed two scores based on the frequency of keywords in tabloid and misleading ads, with more weight given to more popular words. An ML clustering model using sentiments and these scores achieved about 80% accuracy in differentiating all three types of ads.</p><p>Finally, we repeated the preceding analysis with bigrams, which is NLP nomenclature for two consecutive words. This was meaningful because certain groups of words become much more useful next to others than on their own. The most common bigrams in malicious ads included phrases like “read more,” “simple trick,” “discover how,” and “hollywood’s elite” — all clearly resembling a spammy ad.</p><p>This brought us to our second day at the expo center, and the pace was feverish. Our only real breaks were taken during lengthy software downloads and compilation of code. Some of us attended conferences about the latest developments in the industry, like “Blockchain Framework Exonum 0.8 for Developers.” Some of us, the ones who didn’t believe in the blockchain hype, collected company merchandise and dined on AppNexus’s platter of fine cheeses. Some of us, the ones repulsed by cheese and blockchain, continued to look at code, perhaps finding an errant comma half an hour into code compilation. And Eliot won an RC helicopter.</p><p>For the final machine learning component tying it all together, we experimented with both a random forest model and a support vector machine (SVM) . A random forest ideally allows for the minimization of overfitting and would avoid significant consideration of hyperparameters. It showed promising cross validation scores. However, we ultimately went with an SVM because of our small number of features (7) and small sample size (398). It gave us a staggering 90% accuracy in categorizing tabloid, misleading, and non-malicious ads, and 95% accuracy in categorizing ads as malicious or non-malicious. Below is an image to represent our text classifier system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/0*3-IFVrutlB_dKMhZ" /></figure><p>Our code and sets of images for this fundamental system can be <a href="https://github.com/tectonic8/DevWeek2018Hackathon/blob/master/text%20analysis.ipynb">found on GitHub</a>.</p><h3>Fundamental System 2 — Image Classification</h3><p>Given that pure image classification is a very hard thing to think about intuitively, we were very interested to see how our results would turn out and what the ML process would really entail. We desired to do a rather simple image classification using a Convolutional Neural Network (CNN) — a deep learning model that we could implement using python. We learned that building a CNN from scratch would require training on several thousand different types of images to be accurate enough for production. Quickly our team realized that our combined compute power of 4 laptops would not be enough! Desperately we started looking into using cloud computing services like AWS and Azure. We dove into the documentation, but it became apparent that actually getting to use significant resources power that we needed was not going to be easy. Would this be impossible to do? Should we look into a different simpler system? These were questions we thought about, but thankfully we found a potential solution — modifying a pre-trained model! To understand this and the inherent advantages provided, we can examine a very simplified version of a CNN.</p><p>Essentially, a CNN utilizes the concepts of neurons from biology to very effectively detect patterns. This is likely no shock, as this concept has been displayed in major news outlets prominently for the past few years. You often hear that these deep learning models have many different layers. But what does that mean and how does it relate to the brain? These layers can intuitively be thought of as different systems that detect a very specific type of small pattern. For example — one layer may look to identify a specific feature in someone’s face, such as someone’s eyes. In actuality, these layers detect patterns that are much smaller, such as one tiny portion of a letter, but the idea is the same — small groups of pattern detectors. These layers are then grouped together, often using simplification processes like pooling (using subsets of data for calculation simplification) in order to detect something much larger and more difficult. Continuing on from our previous example, a CNN may try to detect if an image contains a human face. This is called the final classification because it combines results from the previous groups of small pattern detectors to determine one final attribute, which may have a significant amount of variance. This is essentially what your own brain tries to do — you look for small patterns, like hair, wrinkles, ears, and so on, and from the aggregation of all this data, are able to make a judgement call. No one, or one set of features, defines a face, but intuitively you just sort of “know”.</p><p><strong>Figure: Simplified CNN Example</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/0*3Bzh-VhMtUZpST-v" /><figcaption>(Source: <a href="https://www.mathworks.com/videos/introduction-to-deep-learning-what-are-convolutional-neural-networks--1489512765771.html">mathworks.com</a>)</figcaption></figure><p>The way that these layers (or more simply thought of as small pattern detectors) are formed is by looking at thousands of images and tuning the system through a process known as backpropagation. This was troubling as we had a fraction of this image set from our newly created database. So, we turned towards pre-trained models, and determined that using Tensorflow from Google would greatly help us in this issue.</p><p>Tensorflow is an open source library for numerical computation, specializing in machine learning applications. TensorFlow introduces the field of image recognition and uses a pre-trained model (Inception) for recognizing images. The model we would import would have been already trained on the <a href="http://www.image-net.org/challenges/LSVRC/">ImageNet Large Visual Recognition Challenge</a> dataset — a massive database. Models from Tensorflow are powerful as they can differentiate between 1,000’s of different classes, ranging from Dalmatians to dishwashers, and are very sophisticated. However, they take only a few minutes to actually download. We would be able to take advantage of Tensorflow by retraining the last step of the process in one specific model — simply telling the chosen algorithm to classify the images into tabloid advertorials, misleading claims, or quality ads. This way we would get almost all of the advantages of a CNN without having to fully build out all of the smaller pattern systems that were built in the previous layers!</p><p>So, we installed Tensorflow and retrained a <a href="https://www.tensorflow.org/hub/modules/image">MobileNet</a> (a small efficient convolutional neural network) to fit our needs. First, we configured the MobileNet by setting up the input resolution and relative size of the model as a fraction of the largest possible MobileNet.</p><p>Before starting the training, we also installed a module called TensorBoard. <a href="https://www.tensorflow.org/guide/summaries_and_tensorboard">TensorBoard</a> is a graphical monitoring and inspection tool included with Tensorflow. Hence, it would allow us to monitor the training progress. With our environment set up, through a few simple commands we retrained the imported model to solely classify images into our three distinct categories.</p><p>After all the training steps were completed, we ran a final script to test evaluation accuracy on a set of images that were kept separate from the training set. After running the script on several images our results were relatively positive. Depending on how the images were partitioned in the training process and the particular photos that were being tested — we achieved between 45% and 55% overall accuracy in placing our advertisements into the correct group. These lower results were likely due to a low sample of images to test on, but they also represented a massive improvement from the baseline of randomly guessing (~33%). Our second ML algorithm concept was proved, and thus our two core systems had been developed. Since virtually all of the development process takes place in the command line, the code to implement this system has not been included in the Github repo. However, this <a href="https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/#0">Google TensorFlow tutorial</a> was the overall basis that we used in order to train our ML system.</p><h3>Laying out Next Steps</h3><p>Once these two systems were trained, the time for code freeze was coming. Unfortunately, we did not have an opportunity to link our results together — but what was unique about our group was that we had created the use cases already. Our last step was to showcase our work — we crafted a simple Powerpoint, <a href="https://github.com/tectonic8/DevWeek2018Hackathon/releases/tag/1.0">viewable on GitHub</a>, that would allow us to walk our judges through our project ideation, core systems and an idea for a tool that could use the classification algorithms that we had implemented.</p><p>Before we knew it, the first round of evolution was upon us! Teams of judges quickly swarmed around the entire hackathon, selecting groups systematically to present their work. Almost instantly our turn came — we had about two minutes to present all that we had done. Scrolling through the presentation while side-by-side displaying our code on another laptop, we confidently introduced our challenge, the specific problem we would address, and then dove deep into our progress thus far. It ended up taking around ten minutes, but we suppose the project was compelling enough to warrant quintupling their time limit. This much was confirmed when we had the judge come to us privately afterwards, dropping the dour poker face. Afterwards, we presented directly to AppNexus, which was delightful as the judge was not only engaged with our solution to a prompt that he actually worked on, but also proposed some thoughtful questions. How would our system affect AppNexus’ clients? What would we do for an override if a significant portion of quality ads were being blocked?</p><p>After this experience, we returned to our station. The time went by in a whirlwind and it was a breath of fresh air for the first time in the last 36 hours or so. While pondering the questions mentioned, our team received a phone call. “You have been selected as a top 5 team in the entire hackathon.” Thrilled, we immediately began to clean up our code and presentation and before we knew it, we appeared on stage at the Hackathon showcasing our work to the entire event. Thanks to serious rehearsal, we exceeded our time limit by a slightly smaller factor.</p><p>We sat back down to see the other four teams. Since their projects were inspiringly creative and technically sophisticated, we talked about how this had been a good experience even without the win. (Admittedly, we regretted not having <a href="https://blog.netapp.com/wp-content/uploads/2017/05/Machine-Learning.jpeg">cooler visuals</a>.) A random audience member said he liked us the best, which was recognition enough at that point. But astonishingly, he wasn’t the only one; half hour after our presentation, we were also selected as the AppNexus challenge winner! Overall, the experience was not only fast-paced and exhilarating, but we could not have asked for a more invested and innovative partner than AppNexus.</p><p>With more time, we would have liked to use more NLP techniques to get a better grasp of text meaning in the advertisements. Examples include combining different forms of words, using word2vec to consider the similarity of words, using a parser to analyze sentence structure, and more advanced NLP to find specific emotions that are crafted in the advertisement. We would also plan to perform a grid-search to optimize the hyperparameters on the SVC, and further process the images for better OCR text extraction (binarize, remove noise). Also, we would have liked to create a much larger database of images for both testing and training our image classifier. Finally, we would have loved to combine the results from both systems and actually test out our own inputs to create a Team Early Bird version of malicious content.</p><p>Thank you AppNexus for presenting the challenge at the NYC DeveloperWeek Hackathon and for allowing us to be part of this blog!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0c0359b5c75" width="1" height="1" alt=""><hr><p><a href="https://medium.com/xandr-tech/malicious-content-identification-f0c0359b5c75">Malicious Content Identification</a> was originally published in <a href="https://medium.com/xandr-tech">Xandr-Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Celebrating !!Con 2018 Not with a Whimper, but a Bang]]></title>
            <link>https://medium.com/xandr-tech/celebrating-con-2018-not-with-a-whimper-but-a-bang-1e0193ad9748?source=rss-f77f4074e4d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e0193ad9748</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[conference]]></category>
            <category><![CDATA[random]]></category>
            <category><![CDATA[bangbangcon]]></category>
            <dc:creator><![CDATA[Xandr Engineering]]></dc:creator>
            <pubDate>Wed, 30 May 2018 19:01:50 GMT</pubDate>
            <atom:updated>2018-05-30T19:01:50.195Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This post is written by </em><a href="https://www.clippings.me/adrianachavez"><em>Adriana Chavez</em></a><em>, a writer for AppNexus and former AppNexus Girls Who Code student. When not writing or coding, she can be found reading literary fiction or walking her rescue dog in the streets of New York.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9NehEbMyC3miJDBsB7oEsw.jpeg" /></figure><p>Two weeks ago the fifth annual !!Con (pronounced bang-bang con) was held at AppNexus’s headquarters in New York. Attendees gathered in Flatiron for the two day conference to see some of the most innovative and surprising ways programming is currently being used.</p><p>The conference officially began the morning of May 12th and ran throughout the day and into the 13th. I attended the event as a programmer excited to learn more about tech, but also as a writer and a creative who is always trying to find the humanity and intersectionality in the field.</p><p>The event broke up throughout the weekend into sessions of lightning talks — 10 minute presentations each on a different subject. This format allowed the audience to hear from a wide range of speakers throughout the conference while not being overwhelmed by one topic for too long.</p><p>Keynote speakers Mimi Onuoha and Liz Fong-Jones opened each day of lightning talks on Saturday and Sunday, respectively. Mimi — a researcher, artist, and adjunct professor — and Liz — a Staff Site Reliability Engineer at Google — both show how we can bring our own interests and causes we care about to the work we do. Liz’s talk focused on her love of video games and how other unusual forms of experience can be an advantage at work. While part of Mimi’s work is based in using tech as a form of social critique — she utilizes data collection and computational categorization to examine the ways people are categorized and represented. Together they each brought an energy to the event that opened the audience up to a day full of tech and new possibilities.</p><p>When I walked into !!Con for the first time I was immediately greeted with posters of stick figures throughout the hall reading “Welcome!” followed by signs for gender-neutral restrooms.</p><p>Providing an inclusive and accepting environment for attendees is as vital to the conference’s mission as producing an educative tech event. In addition to gender-inclusive bathrooms, the conference also had amenities like CART live captioning; a lactation room provided by AppNexus; and an alcohol-free, all ages environment. The conference matches these accommodations with a strict <a href="http://bangbangcon.com/conduct.html">code of conduct</a> and no harassment policy.</p><p>The result of these parameters makes !!Con feel less like an annual event, and more like a community of similar minded people with the same values and love for tech coming together.</p><p>“It’s actually the community that’s the most incredible [part of !!Con]. The talks are fantastic but I think they’re the product of people who think deeply about what they’re doing and enjoy it. So it’s really reflective,” said Aria Stewart, a systems engineer and programmer. This is the third year in a row they’ve attended !!Con.</p><p>Talks ranged drastically throughout the weekend, from discussions on how synthetic filesystems work (Omar Rizwan), to the data structure of Pokemon (Jan Mitsuko Cash), to how you can watch Star Wars using Postgress (Will Leinweber).</p><p>Speakers captivated the crowd throughout the two days by either showcasing an unexpected way they are using tech, or by connecting tech to a seemingly unrelated topic. The most surprising part of the conference was that these diverse lightning talks all had the same ability to leave attendees viewing tech in an entirely new light.</p><p>In the talk, <em>Fast, but not too fast! What 17th century windmills can teach us about database migrations,</em> Walden Hillen connected the mechanics of windmills to the way his company WeTransfer runs. He explained that these 400 year old windmills naturally know to catch less wind during storms to prevent breaking- similar to how the developers at WeTransfer must brace themselves for possible crashes or “storms” while managing billions of file transfers.</p><p>Technology is behind everything we see; the way it can be used and the processes behind certain technologies relate more to our everyday lives than we may think. This theme from the weekend especially struck a chord with me because of my mixed background.</p><p>One !!Con attendee spoke to me about her takeaways from Emily Nakashima’s talk about visual programming languages.</p><p>“I really liked [what Emily said about] looking at life in terms of pictures instead of language and how that… open[s] up your mind to all types of possibilities,” the attendee said.</p><p>She connected what she learned about the functions of VPLs to how she thinks of language in her own life. “I’m multilingual… often when I’m trying to switch from one language to another it’s more a feeling I’m trying to transmit.. than an actual word-for-word translation. So [the talk] really resonated because images can do that in some ways more accurately than words,” she said. She told me the talk allowed her to see technology from a more creative and meaningful angle.</p><p>Another attendee, Travis McDemus, who’s worked separately in both audio production and in programming, particularly enjoyed Vince Allen’s talk on Sunday. Vince’s talk, <em>The Man Comes Around: and so does his sound!,</em> showed how he employed machine learning to break down the audio of a Johnny Cash song. “That’s actually part of the intersection of music and programming that I’m unfamiliar with… It’s very amusing to see all the ways people are trying to apply technology,” McDemus said.</p><p>Between breaks, attendees eagerly huddled into groups to talk about the lightning talks they just heard. Hands motioned emphatically to express themselves during conversations or reached out to other people to introduce themselves. The noise in the room echoed around us and practically made the floor vibrate.</p><p>In times when people can feel stuck in the world of programming, !!Con reminds us exactly how fun, creative, and amazing modern technology can be. On Sunday evening, !!Con’s MC Erty Seidohl helped close the event: “We hope !!Con is a reminder of why you became a developer… We hope… that you leave this venue inspired, engaged, and fired up about making the world a better place through the work that you do.”</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e0193ad9748" width="1" height="1" alt=""><hr><p><a href="https://medium.com/xandr-tech/celebrating-con-2018-not-with-a-whimper-but-a-bang-1e0193ad9748">Celebrating !!Con 2018 Not with a Whimper, but a Bang</a> was originally published in <a href="https://medium.com/xandr-tech">Xandr-Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AppNexus Common Framework: it’s out! (also how to print integers faster)]]></title>
            <link>https://medium.com/xandr-tech/appnexus-common-framework-its-out-also-how-to-print-integers-faster-ceb72ac171a1?source=rss-f77f4074e4d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/ceb72ac171a1</guid>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[c]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Xandr Engineering]]></dc:creator>
            <pubDate>Sat, 23 Dec 2017 17:23:50 GMT</pubDate>
            <atom:updated>2017-12-23T17:23:50.075Z</atom:updated>
            <content:encoded><![CDATA[<p>In the words of a friend and former colleague:</p><p>Two years of my life in one repository…. Congrats <a href="http://twitter.com/pkhuong">@pkhuong</a> <a href="http://twitter.com/arexus">@arexus</a> and all! <a href="https://t.co/jPFnYrc5V4">https://t.co/jPFnYrc5V4</a></p><p>— <a href="https://twitter.com/johnwittrock/status/943251391459528704">@johnwittrock</a></p><p>After trying to package some internals for more than two years, we recently got the OK to open source the <a href="https://github.com/appnexus/acf">AppNexus Common Framework</a> under the Apache License 2.0, despite its incomplete state.</p><p>ACF contains the base data structure and runtime library code we use to build production services, in C. Some of it is correctly packaged, most of it just has the raw files from our internal repository. Ironically, after settling on the project’s name, we decided not to publish the most “framework-y” bits of code: it’s not clear that anyone else would benefit. The data structures are in C, and tend to be read-optimised, with perhaps some support for non-blocking single-writer/multi-reader concurrency. There’s also some non-blocking algorithms to support the data structures, and some basic HTTP server code that we found useful to run CPU-intensive or mixed CPU/network-intensive services.</p><p>Publishing our internal code took a long time because we were trying to open a project that didn’t exist yet, despite being composed of code that we use every day. <a href="http://www.appnexus.com/">AppNexus</a> doesn’t sell code or binaries. Like many other companies, AppNexus sells services backed by in-house code. Our internal code base is full of informal libraries (I would be unable to make sense of the code base if it wasn’t organised that way), but enforcing a clean separation between pseudo-libraries is a lot of extra work for questionable value.</p><p>These fuzzy demarcations are made worse by the way we imported some ideas directly from Operating Systems literature, in order to support efficient concurrent operations. That had a snowball effect: everything, even basic data structures, ends up indirectly depending on framework-y code that’s specialised for our use case. The usual initial offenders are the safe memory reclamation module, and the tracking memory allocator (with a bump pointer mode); both go deep in internals that probably don’t make sense outside of AppNexus.</p><p>Back in 2015, we looked at our internal support code (i.e., code that doesn’t directly run the business) and decided we should share it. We were — and still are — sure that other people face similar challenges, and exchanging ideas, if not directly trading code, can only be good for us and for coders at large. We tried to untangle the internal “Common” support library from the rest of the code base, and to decouple it from the more opinionated parts of our code, while keeping integration around (we need it) but purely opt-in.</p><p>That was hard. Aiming for a separate <em>shared object</em> and a Debian package made it even harder than it had to be. The strong separation between packaged ACF code and the rest of repo added a lot of friction, and the majority of the support code stayed in-tree.</p><p>Maybe we made a mistake when we tried to “librarify” our internals. We want a library of reusable code; it doesn’t mean we need a literal shared object. I’m reminded of the two definitions of portable code: code sprinkled with platform conditionals, or code that can be made to run on a different platform with minimal effort. Most of the time, I’d rather have the latter. Especially when code mostly runs on a single platform, or is integrated in few programs, I’d rather reduce overhead for the common case, while making reuse <em>possible</em> and easy enough that others can benefit.</p><p>And that’s how we got the ACF effort out of the door: we accepted that the result would not be as polished as our favourite open source libraries, and that most of the code wouldn’t even be packaged or fully disentangled from internals. Far from an ideal state, but it’s closer to our goals than keeping the project private and on the backburner. We got it out by “feature” boxing the amount of work — paring it down to figuring out what would never be useful to others, and tracking down licenses and provenance — before pushing the partial result out to a public repository. Unsurprisingly, once that was done, we completed more work on ACF in a few days than we have in the past year.</p><p>Now that <a href="https://github.com/appnexus/acf">ACF</a> is out, we still have to figure out the best way to help others co-opt our code, to synchronise the public repository with our internal repository, and, in my dreams, accept patches for the public repo and have them also work for the internal one. In the end, what’s important is that the code is out there with a clear license, and that someone with similar problems can easily borrow our ideas, if not our code.</p><p>If anyone else is in the same situation, please give yourself the permission to open source something that’s not yet fully packaged. As frustrating as such projects can be, it has to be better than keeping it closed.</p><h3>How to print integers faster</h3><p>The integer to string conversion file (<a href="https://github.com/appnexus/acf/blob/master/src/an_itoa.c">an_itoa</a>) implemented a few cute ideas to solve a depressingly common problem. I like it so much that it&#39;s the first file I cleaned up and packaged after making the repository public.</p><p>We wrote that code in July 2014. Back then, we had an application with a moderate deployment (a couple racks on three continents) that was approaching full capacity… while more machines were in the pipeline, a quick perf run showed it was spending a lot of time converting strings to integers and back. We already had a fast-ish string to integer function. Converting machine integers back to string however, is a bit more work, and took up around 20% of total CPU time.</p><p>Of course, the real solution here is to not have this problem. We shouldn’t have been using a human-readable format like JSON in the first place. We had realised the format would be a problem a long time ago, and were actually in the middle of the transition, after a first temporary fix (replacing a piece of theoretically reconfigurable Javascript that was almost never reconfigured with hardcoded C that performed the same JSON manipulation). But, there we were, in the middle of this slow transition involving terabytes of valuable persistent data, and we needed another speed boost until protobuf was ready to go.</p><p>When you’re stuck with C code that was manually converted, line by line, from Javascript, you don’t want to try and make high level changes to the code. The only reasonable quick win was to make conversion from integer to string faster.</p><p>Human readable formats resulting in wasted CPU cycles to print integers is a common problem, and we quickly found a few promising approaches or libraries. Our baseline was the radix-10 code in <a href="https://github.com/client9/stringencoders/blob/master/src/modp_numtoa.c#L52">stringencoders</a>. This <a href="https://tia.mat.br/posts/2014/06/23/integer_to_string_conversion.html">post about Lwan</a> suggested using radix-10, but generating strings backward instead of reversing like the stringencoders library. Facebook apparently hit a similar problem in 2013, which lead to <a href="https://www.facebook.com/notes/facebook-engineering/three-optimization-tips-for-c/10151361643253920">this solution</a> by Andrei Alexandrescu. The Facebook code is a combination of two key ideas: radix-100 encoding, and determining the size of the string ahead of time with galloping search to generate the result backward, directly where it should go.</p><p>Radix-100 made sense, although I wasn’t a fan of the 200-byte lookup table. I was also dubious of the galloping search; it’s a lot of branches, and not necessarily easy to predict. The kind of memmove we need to fixup after conversion is small and easy to specialise on x86, so we might not need to know the number of digits at all.</p><p>I then looked at the microbenchmarks in Andrei’s post, and they made it look like the code was either tested on integers with a fixed number of digits (e.g., only 4-digit integers), or randomly picked with uniform probability over a large range.</p><p>If the number of digits is fixed, the branchiness of galloping search isn’t an issue. When sampling uniformly… it’s also not an issue: most integers are large! If I pick an integer at random in [0, 1e6), 90% of the integers have 6 digits, 99% 5 or 6, etc.</p><p>Sometimes, uniform selection is representative of the real workload (e.g., random uids or sequential object ids). Often, not so much. In general, small numbers are more common; e.g., small counts can be expected to roughly follow an exponential.</p><p>I was also worried about data cache footprint with the larger lookup table for radix-100 encoding, but then realised we were converting integers in tight loops, so the lookup table should usually be hot. That also meant we could afford a lot of instruction bytes; a multi-KB atoi function wouldn&#39;t be acceptable, but a couple hundred bytes was fine.</p><p>Given these known solutions, <a href="https://twitter.com/johnwittrock/">John</a> and I started doodling for a bit. Clearly, the radix-100 encoding was a good idea. We now <em>had</em> to know if we could do better.</p><p>Our first attempt was to find the number of decimal digits more quickly than with the galloping search. It turns out that approximating log_10 is hard, and we gave up ;)</p><p>We then realised we didn’t need to know the number of decimal digits. If we generated the string in registers, we could find the length after the fact, slide characters with bitwise shifts, and directly write to memory.</p><p>I was still worried about the lookup table: the random accesses in the 200 byte table for radix-100 encoding could hurt when converting short arrays of small integers. I was more comfortable with some form of arithmetic that would trade best-case speed for consistent, if slightly sub-optimal, performance. As it turns out, it’s easy to convert values between 0 and 100 to <a href="https://en.wikipedia.org/wiki/Binary-coded_decimal">unpacked BCD</a> with a reciprocal multiplication by 1/10 and some in-register bit twiddling. Once we have a string of BCD bytes, we can add &#39;0&#39; vertically to convert to ASCII characters.</p><p>The radix-100 approach is nice because it chops up the input two digits at a time; the makespan for a given integer is roughly half as long, since modern CPUs have plenty of execution units for the body.</p><p>The dependency graph for radix-10 encoding of 12345678 looks like the following, with 7 serial steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*X8SoXjdNUtfIGKIg2i0EdQ.png" /></figure><p>Going for radix-100 halves the number of steps, to 4; the steps are still serial, except for the conversion of integers in [0, 100) to strings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*lND-4l4SCY1ROJL0Rap03A.png" /></figure><p>Could we expose even more <a href="https://en.wikipedia.org/wiki/Instruction-level_parallelism">ILP</a> than the radix-100 loop?</p><p>The trick is to divide and conquer: divide by 10000 (1e4) before splitting each group of four digits with a radix-100 conversion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/426/1*sLlc5CPav_NoTW5yOnqjAw.png" /></figure><p>True recursive encoding gives us fewer steps, and 2 of the 3 steps can execute in parallel. However, that might not always be worth the trouble for small integers, and we know that small numbers are common. So, even if we have a good divide-and-conquer approach for larger integers, we must also implement a fast path for small integers.</p><p>The fast path for small integers (or the most significant limb of larger integers) converts a 2 or 4 digit integer to unpacked BCD, finds the number of leading zeros, converts the BCD to ASCII by adding &#39;0&#39; (0x30) to each byte, and shifts out any leading zero; we assume that trailing noise is acceptable, and it&#39;s all NUL bytes anyway.</p><p>For 32-bit integers <a href="https://github.com/appnexus/acf/blob/master/src/an_itoa.c#L236">an_itoa</a> (really an_uitoa) looks like:</p><pre>if number &lt; 100:<br>     execute specialised 2-digit function</pre><pre>if number &lt; 10000:<br>     execute specialised 4-digit function<br> <br>partition number with first 4 digits, next 4 digits, and remainder.<br> <br>convert first 2 groups of 4 digits to string.<br>if the number is &lt; 1e8:  # remainder is 0!<br>     shift out leading zeros, print string.<br>else:<br>     print remainder # at most 100, since 2^32 &lt; 1e10<br>     print strings for the first 2 groups of 4 digits.</pre><p>The 64 bit version, <a href="https://github.com/appnexus/acf/blob/master/src/an_itoa.c#L342">an_ltoa</a> (really an_ultoa) is more of the same, with differences when the input number exceeds 1e8.</p><h3>Is it actually faster?</h3><p>I had already concluded that cache footprint was mostly not an issue, but it’s still important to make sure we don’t get anything <em>too</em> big.</p><ul><li>an_itoa: 400 bytes.</li><li>an_ltoa: 880 bytes</li><li>fb_itoa: 426 bytes + 200 byte LUT</li><li>fb_constant_itoa (without the galloping search): 172 bytes + 200 byte LUT</li><li>lwan_itoa (radix-10, backward generation): 60 bytes.</li><li>modp_uitoa10: 91 bytes.</li></ul><p>The galloping search in Facebook’s converter takes a lot of space (there’s a ton of conditional branches, and large numbers must be encoded somewhere). Even without the lookup table, an_itoa is <em>smaller</em> than fb_itoa, and an_ltoa (which adds code for &gt; 32 bit integers) is only 254 bytes larger than fb_itoa. Facebook&#39;s galloping search attempts to make small integers go faster by checking for them first; if we convert small numbers, we don&#39;t expect to use all ~250 bytes in the galloping search. That&#39;s also true for an_itoa and an_ltoa: the code is setup such that larger numbers jump forward over specialised subroutines for small integers. Small integers thus fall through to only execute code at the beginning of the functions. 400 or 800 bytes are sizable footprints compared to the 60 or 90 bytes of the radix-10 functions, but acceptable when called in tight loops.</p><p>Now that we feel like the code and data sizes are reasonable (something that microbenchmarks rarely highlight), we can look at speed.</p><p>I first ran the conversion with random integers in each digit count class from 1 digit (i.e., numbers in [0, 10)) to 19 (numbers in [1e8, 1e9)). The I$ was hot, but the routines were not warmed on a set of numbers in the same class (more realistic that way).</p><p>We have data for:</p><ul><li>an_itoa, our 32 bit conversion routine;</li><li>an_ltoa, our 64 bit conversion routine;</li><li>fb_constant_itoa, Facebook&#39;s code, with the galloping search stubbed out;</li><li>fb_itoa, Facebook&#39;s radix-100 code;</li><li>itoa, GNU libc conversion (via sprintf);</li><li>lw_itoa, <a href="https://lwan.ws/">Lwan</a>&#39;s backward radix-10 converter;</li><li>modp, stringencoder&#39;s radix-10 / strreverse <a href="https://github.com/client9/stringencoders/blob/master/src/modp_numtoa.c#L52">converter</a>.</li></ul><p>I included fb_constant_itoa to serve as a lower bound on the radix-100 approach: the conversion loop stops as soon as it hits 0 (same as fb_itoa), but the data is written at a fixed offset, like lw_itoa does. In both cases, we&#39;d need another copy to slide the part of the output buffer that was populated with characters over the unused padding (Facebook&#39;s code solves the problem by determining where the output should end before converting).</p><p>Some functions only go up to UINT32_MAX, in which case we have no data after 9 digits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*h_mYbxhvoPaFIsIW1Z8-ZQ.png" /></figure><p>The solid line is the average time per conversion (in cycles), over 10K data points, while the shaded region covers the 10th percentile to the 90th percentile.</p><p>(GNU) libc conversion is just wayy out there. The straightforward modp (stringencoders) code overlaps with Facebook&#39;s itoa; it&#39;s slightly slower, but so much smaller.</p><p>We then have two incomplete string encoders: neither fb_constant_itoa nor lw_itoa generate their output where it should go. They fill a buffer from the end, and something else (not benchmarked) is responsible for copying the valid bytes where they belong. If an incomplete implementation suffices, Lwan&#39;s radix-10 approach is already competitive with (arguably faster than) the Facebook code. The same backward loop, but in radix-100, is definitely faster than Facebook&#39;s full galloping search/radix-100 converter.</p><p>Finally, we have an_itoa and an_ltoa, that are neck and neck with one another, faster than both modp and fb_itoa on small and large integers, and even comparable with or faster than the incomplete converters. Their runtime is also more reliable (less variance) than modp&#39;s and fb_itoa&#39;s: modp pays for the second variable length loop in strreverse, and fb_itoa for the galloping search. There are more code paths in an_itoa and an_ltoa, but no loop, so the number of (unpredictable) conditional branches is lower.</p><p>What have we learned from this experiment?</p><ol><li>It’s easy to improve on (g)libc’s sprintf. That makes sense, since that code is so generic. However, in practice, we only convert to decimal, some hex, even less octal, and the rest is noise. Maybe we can afford to special case these bases.</li><li>The double traverse in modp_uitoa10 hurts. It does make sense to avoid that by generating backward, ideally in the right spot from the start.</li><li>Radix-100 encoding is a win over radix-10 (fb_constant_itoa is faster than lwan_itoa).</li><li>Using registers as buffers while generating digits is a win (an_itoa and an_ltoa are faster for small values).</li><li>Divide and conquer is also a win (an_ltoa is flatter for large integers).</li></ol><p>With results that made sense for an easily understood microbenchmark, I decided to try a bunch of distributions. Again, the code was hot, the predictors lukewarm, and we gathered 10K cycle counts per distribution/function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*w5Dz69ZAmxZX_BDVE4rtgw.png" /></figure><p>Clearly, we can disregard glibc’s sprintf (itoa).</p><p>The first facet generated integers by choosing uniformly between 100, 1000, ¹⁰⁴, …, ¹⁰⁸. That’s a semi-realistic tweak on the earlier dataset, which generated a bunch of numbers in each size class, and serves as an easily understood worst-case for branch prediction. Both an_itoa and an_ltoa are faster than the other implementations, and branchier implementations (fb_itoa and modp) show their variance. Facebook&#39;s fb_itoa isn&#39;t even faster than modp&#39;s radix-10/strreverse encoder. The galloping search really hurts (fb_constant_itoa, without that component, is slightly faster than then comparable radix-10 lw_itoa).</p><p>The second facet is an even harder case for branch predictors: random exponentials (pow(2, 64.0 * random() / RAND_MAX)), to simulate real-world counts. Both an_itoa and an_ltoa are faster than the other implementations, although an_ltoa less so: an_itoa only handles 32 bit integers, so it deals with less entropy. Between the 32-bit implementations, an_itoa is markedly faster and more consistent than lw_itoa (which is also incomplete) and modp. Full 64-bit converters generally exhibit more variance in runtime (their input is more randomised), but an_ltoa is still visibly faster than fb_itoa, and even than the incomplete fb_constant_itoa. We also notice that fb_itoa&#39;s runtimes are more spread out than fb_constant_itoa: the galloping search adds overhead in time, but also a lot of variance. That makes me think that the Facebook code is more sensitive than others to difference in data distribution between microbenchmarks and production.</p><p>The third facet should be representative of printing internal sequential object ids: uniform integers in [0, 256K). As expected, every approach is tighter than with the exponential distribution (most integers are large). The an_itoa/an_ltoa options are faster than the rest, and it&#39;s far from clear that fb_itoa is preferable to even modp. The range was also chosen because it&#39;s somewhat of a worst case for an_itoa: the code does extra work for values between 10⁴ and 10⁸ to have more to do before the conditional branch for x &lt; 1e8. That <em>never</em> pays off in the range tested here. However, even with this weakness, an_itoa still seems preferable to fb_itoa, and even to the simpler modp_uitoa10.</p><p>The fourth facet (first of the second row) shows what happens when we choose random integers in [0, 20). That test case is interesting because it&#39;s small, thus semi-representative of counts for rare events, and because it needs 1 or 2 digits with equal probability. Everything does pretty well, and runtime distributions are overall tight; branch predictors can do a decent job when there are only two options. I&#39;m not sure why there&#39;s such a difference between an_itoa and an_ltoa&#39;s distribution. The code for any value less than 100 is identical at the C level, but there are small difference in code generation… but I can&#39;t pinpoint where the difference might come from.</p><p>The fifth facet, for random integers in [100, 200) is similar, with a bit more variance.</p><p>The sixth facet generates unix timestamps around a date in 2014 with uniform selection plus or minus one million second. It’s meant to be representative of printing timestamps. Again, an_itoa and an_ltoa are faster than the rest, with an_itoa being slightly faster and more consistent. Radix-100 (fb_constant_itoa) is faster and more consistent than radix-10 (lw_itoa), but it&#39;s not clear if fb_itoa is preferable to modp. The variance for modp is larger than for the other implementations, even fb_itoa: that&#39;s the cost of a radix-10 loop and of the additional strreverse.</p><p>This set of results shows that conditional branches are an issue when converting integers to strings, and that the important of branches strongly depends on the distribution. The Facebook’s approach, with a galloping search for the number of digits, seems particularly sensitive to the distribution. Running fb_itoa in production because it does well in microbenchmark is thus only a good idea if we know that the microbenchmark is representative of production.</p><p>Bigger numbers take more time to convert, but the divide and conquer approach of an_itoa and an_ltoa is consistently faster at the high end, while the unrolled SIMD within a register approach does well for small numbers.</p><h3>So, if you ever find yourself bottlenecked on s[n]printf</h3><blockquote>The correct solution to the “<em>itoa</em> is too slow&quot; problem is simple: don&#39;t do that. After all, If there&#39;s no special requirement, we find <a href="https://developers.google.com/protocol-buffers/">Protobuf</a> does very well as a &quot;better JSON.&quot;</blockquote><p>However, once you find yourself in this bad spot, it’s trivial to do better than generic libc conversion code. That makes it a dangerously fun problem in a way… especially given that the data distribution can matter so much. No benchmark is perfect, but various implementations are affected differently by flaws in microbenchmarks. It’s thus essential not to overfit on the benchmark data, probably even more important than improving performance by another factor of 10% or 20% (doing 4–5x better than libc code is already a given). That’s why I prefer integer conversion code with more consistent cycle counts: there’s less room for differences due to the distribution of data.</p><p>Finally, if, like 2014-AppNexus, you find yourself converting a lot of integers to strings in tight loops, <a href="https://github.com/appnexus/acf/blob/master/src/an_itoa.c">try </a><a href="https://github.com/appnexus/acf/blob/master/src/an_itoa.c">an_itoa or </a><a href="https://github.com/appnexus/acf/blob/master/src/an_itoa.c">an_ltoa</a>! The <a href="https://github.com/appnexus/acf">whole repository is Apache 2.0</a>, and it should be easy to copy and paste all the dependencies to pare it down to two files. Also, note that the functions use their destination array (up to 10 bytes for an_itoa, and 20 for an_ltoa) as scratch space, even for integers that need fewer digits.</p><p>Written by <strong>Dr. Paul khuong</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ceb72ac171a1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/xandr-tech/appnexus-common-framework-its-out-also-how-to-print-integers-faster-ceb72ac171a1">AppNexus Common Framework: it’s out! (also how to print integers faster)</a> was originally published in <a href="https://medium.com/xandr-tech">Xandr-Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Meet Van Gogh: The Algorithm that Makes Data Segmenting Simple]]></title>
            <link>https://medium.com/xandr-tech/meet-van-gogh-the-algorithm-that-makes-data-segmenting-simple-f24496dd401c?source=rss-f77f4074e4d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/f24496dd401c</guid>
            <category><![CDATA[intern]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Xandr Engineering]]></dc:creator>
            <pubDate>Fri, 03 Nov 2017 17:44:41 GMT</pubDate>
            <atom:updated>2018-01-24T22:56:38.626Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YHsjpLVS-M9IvPAl.jpg" /></figure><p><em>This post is part of our Summer Intern Blog Series! Each of our most recent class of interns wrote a blog post on their biggest accomplishments and lessons of the summer. This installment comes from Tyrone, a rising senior studying Computer Science at Boston University. He enjoys rock climbing, listening to jazz, and getting overly excited when talking about good food options in Boston.</em></p><p>In college, many computer science projects are small to medium sized affairs, designed to be learning exercises that practice a specific set of concepts or test knowledge retention. Rarely will a well-polished project blossom into something larger. While my previous internships introduced me to the work environment of large-scale collaborative research effort, my summer at AppNexus was my first foray into working at a company that maintains large scale applications impacting millions of people around the globe.</p><p>AppNexus transacts billions of impressions daily. When a user clicks on a website, they can be associated with specific <strong>data segments</strong> that contain information about, among other things, geographic location and demographics. Advertisers seeking to glean insights from segments would naturally be interested in the number of users per segment. Another useful query would be to find the set of users belonging to a specific set of segments, with the assumption that users in similar segments likely have similar product tastes and ad preferences.</p><h3>Segmenting: Not as easy as it sounds</h3><p>It’s not difficult to think up simple algorithms to do unique set operations on small data sets. The problem is these algorithms don’t scale well. Counting the number of distinct elements in a set requires keeping track of each unique element being iterated over. For a platform dealing with billions of users a day, this solution is infeasible. Luckily, not knowing the <em>exact</em> number of users becomes less of an issue if we can accept some degree of error while still producing a reasonably accurate estimate. Over the summer, my project was to generate that estimate, creating a ‘sketch’ of the data stored in a data structure aptly titled a <strong>data sketch</strong>.</p><p>The main advantage of sketches is their sublinear space complexity. Instead of storing all the unique items from an incoming data stream, the sketch uses a hashing algorithm to produce an estimate. At the cost of some accuracy, we get reduced memory usage and significant performance speedups. As a bonus, there exist data sketch libraries implemented in Go, the language we would be using to build our application.</p><p>We named the project Van (Go)gh after the artist himself, so his artistic talent in sketching would bleed over to coding our Go data sketches. Accordingly, my first order of business was to get a handle on the fundamentals of Go.</p><p>Next, we ironed out Van Gogh’s architecture. We settled on running two different Go server types dubbed <strong>collectors</strong> and <strong>aggregators</strong>. For a given incoming ad request, a collector will extract a list of segments belonging to the user who sent the request and update a data sketch tracking the unique segment. The aggregator periodically pulls down and merges sketches coming from collectors to store a time series of segment counts, which is exposed to clients through a web API.</p><h3>Van Gogh in action</h3><p>The API lets you query for specific segments as a URL parameter, returning the unique count number. To show something more exciting than a blank white screen with a number on it, I also added a web visualization that compares time series data across multiple segments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/969/0*-eyrAvpmn390byVy.png" /></figure><p>Above is a graph of mock data displaying multiple user segment counts over time, created using the d3.js framework. Although this example was generated randomly, one could easily imagine adding real data to track, for example, segment growth in response to a targeted ad campaign.</p><p>One major design consideration I wrestled with was the need for Van Gogh to have some level of concurrency. The collectors needed to extract segment data from hundreds of thousands of incoming ad requests. All of this would be written to the collector’s global sketch and subsequently read across a network by the aggregator. Here we run into the classic readers-writers problem from concurrency. Luckily, Go provides built-in primitives to handle concurrent resource access. We implemented a simple data structure to wrap all sketch read/write operations in Go’s version of locks, preventing other threads from accessing the sketch at the same time.</p><p>This summer has been a wild ride, and I’m grateful to the community that made me feel welcome in a chaotic, unfamiliar city. As I worked through the difficulties involved in building my project, I was always able to turn to a friendly team member or slack a resident expert for advice. At the same time, having a multi-faceted, self-contained project allowed me to continually challenge myself and foster the ability to work independently. I hope the experience I’ve gained at AppNexus will aid me far into the future.</p><p><em>Originally published at </em><a href="https://blog.appnexus.com/2017/meet-van-gogh-algorithm-makes-data-segmenting-simple/?utm_source=linkedin&amp;utm_medium=social&amp;utm_content=Oktopost-linkedin-profile&amp;utm_campaign=Oktopost-Impressionist+-+Intern+blog+-+Tyrone+-+Segmenting"><em>blog.appnexus.com</em></a><em> on November 3, 2017.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f24496dd401c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/xandr-tech/meet-van-gogh-the-algorithm-that-makes-data-segmenting-simple-f24496dd401c">Meet Van Gogh: The Algorithm that Makes Data Segmenting Simple</a> was originally published in <a href="https://medium.com/xandr-tech">Xandr-Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Five Tips for Working with Redux in Large Applications]]></title>
            <link>https://medium.com/xandr-tech/five-tips-for-working-with-redux-in-large-applications-89452af4fdcb?source=rss-f77f4074e4d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/89452af4fdcb</guid>
            <category><![CDATA[front-end-development]]></category>
            <category><![CDATA[frontend]]></category>
            <category><![CDATA[adtech]]></category>
            <category><![CDATA[redux]]></category>
            <dc:creator><![CDATA[Xandr Engineering]]></dc:creator>
            <pubDate>Fri, 09 Jun 2017 15:52:14 GMT</pubDate>
            <atom:updated>2018-01-25T01:51:55.878Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*U2DmhXYumRyXH6X1.png" /></figure><p>Redux is an excellent tool for managing the “state” of an application. The unidirectional flow of data and the focus on immutable state makes reasoning about changes to the state simple. Each update to our state is caused by a dispatched action, which causes our reducer function to return a new state with the desired changes. Many of the user interfaces we create with Redux at AppNexus deal with large amounts of data and very complex user interactions as our customers manage their advertisements or publishing inventory on our platform. Over the course of developing these interfaces, we have arrived at some helpful rules and tips to keep Redux manageable. The following points of discussion should help anyone working with Redux on large, data intensive applications:</p><ul><li><a href="#2032">Section 1:</a> Using indices and selectors for storing and accessing state</li><li><a href="#b3eb">Section 2</a>: Separation of state between data objects, edits to those objects, and other UI state</li><li><a href="#f43d">Section 3</a>: Sharing of state between multiple screens in a Single Page Application, and when not to</li><li><a href="#fe11">Section 4</a>: Reusing common reducers between different places in state</li><li><a href="#cb70">Section 5</a>: Best practices for connecting React components to Redux state</li></ul><h3>1. Store data with an index. Access it with selectors.</h3><p>Choosing the right data structure can make a big difference for our application’s organization and performance. Storing serializable data from an API will greatly benefit from being stored in an index. An index is a javascript object in which the keys are the ids of the data objects we’re storing, and the value is the actual data objects themselves. This pattern is very similar to using a hashmap to store data, and we get the same benefits in terms of lookup time. This is likely unsurprising to those well-versed in Redux. Indeed, Redux’s creator Dan Abramov recommends this data structure in his <a href="https://egghead.io/lessons/javascript-redux-persisting-the-state-to-the-local-storage">Redux tutorial</a>.</p><p>Imagine you have a list of data objects fetched from a REST API, e.g. data from the /users service. Let’s assume that we decided to simply store the plain array in our state, just as it is in the response. What happens when we need to retrieve a specific user object? We would need to iterate over all the users in state. If there are many users, this could be a costly operation. What if we wanted to keep track of a subset of users, perhaps selected and unselected users? We either need to store the data in two separate arrays, or keep track of the indices in the main array of the selected and unselected users.</p><p>Instead, we decide to refactor our code to store the data in an index. We would store the data in our reducer like so:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d41b6723b9449ec445a3ad6168e20208/href">https://medium.com/media/d41b6723b9449ec445a3ad6168e20208/href</a></iframe><p>So how does this data structure help us with these problems? If we need to lookup a specific user object, we simply access the state like so: const user = state.usersById[userId]. This method of access does not require us to iterate over the whole list, saving us time and simplifying our retrieval code.</p><p>At this point you may be wondering how we actually accomplish rendering a simple list of users with these data structures. To do so, we will use a selector, which is a function that takes the state and returns your data. A simple example would be a function to get all the users in our state:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/86fdbe1726b2e1929d0bd8870de00dea/href">https://medium.com/media/86fdbe1726b2e1929d0bd8870de00dea/href</a></iframe><p>In our view code, we call that function with our state to produce the list of users. Then we can iterate over those users and produce our view. We could make another function to get just the selected users from our state like so:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e7de5ce7bcf86d902215f008711c83ac/href">https://medium.com/media/e7de5ce7bcf86d902215f008711c83ac/href</a></iframe><p>The selector pattern also increases our code’s maintainability. Imagine that later on we wish to change the shape of our state. Without selectors, we will be required to update all of our view code as well to match the new state shape. As the number of view components increases, the burden of changing state shape increases drastically. To avoid this problem, we will use selectors to access state in our views. If the underlying state shape changes, we just update the selector to access state in the correct way. All of the consuming components will still get their data, and we don’t have to update them. For all these reasons, large Redux applications will benefit from the index and selector data storage pattern.</p><h3>2. Separate canonical state from view and edit state</h3><p>Real-world Redux applications usually need to fetch some kind of data from another service, such as a REST API. When we receive that data, we dispatch an action with the payload of whatever data we got back. We refer to data returned from a service as “canonical state” — i.e. the current correct state of the data as it is stored in our database. Our state also contains other kinds of data such as the state of user interface components or of the application as a whole. The first time we retrieve some canonical data from our API, we might be tempted to store it in the same reducer file as the rest of our state for a given page. Although this approach may be convenient, it is difficult to scale when you need to fetch many kinds of data from a variety of sources.</p><p>Instead, we will separate out the canonical state into its own reducer file. This approach encourages better code organization and modularity. Scaling reducer files vertically (adding more lines of code) is less maintainable than scaling them horizontally (adding more reducer files to the combineReducers call). Breaking reducers out into their own files makes it easier to reuse those reducers (more on that in <a href="http://techblog.appnexus.com/#section3">Section 3</a>). Additionally, it discourages developers from adding non-canonical state into the data object reducers.</p><p>Why not store other kinds of state with canonical state? Imagine we have the same list of users that we fetched from our REST API. Using the index storage pattern, we would store the data in our reducer like so:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/46c7175c200d23818e0137e5c287e4cf/href">https://medium.com/media/46c7175c200d23818e0137e5c287e4cf/href</a></iframe><p>Now imagine that our UI allows the users to be edited in the view. When the edit icon is clicked for a user, we need to update our state so that the view renders the edit controls for that user. Instead of keeping our view state out of the canonical state, we just decide to put it in as a new field on the objects stored in the users/by-id index. Now our state might look something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ba7ad09d92623537c688b6e860f72bb9/href">https://medium.com/media/ba7ad09d92623537c688b6e860f72bb9/href</a></iframe><p>We make some edits, click the submit button, and the changes are PUT back to our REST service. The service returns back the new state of that object. But how do we merge our new canonical state back into our store? If we just set the new object for the user at their id key in the users/by-id index, then our isEditing flag will no longer be there. We now need to manually specify which fields on the API payload we need to put back into the store. This complicates our update logic. You may have multiple booleans, strings, arrays, or other new fields necessary for UI state that would get appended on to the canonical state. In this situation it is easy for to add a new action for modifying canonical state but forget to reset the other UI fields on the object, resulting in an invalid state. Instead we ought to keep canonical data in its own independent data store in the reducer, and keep our actions simpler and easier to reason about.</p><p>Another benefit to keeping edit state separate is that if the user cancels their edit we can easily reset back to the canonical state. Imagine we have clicked the edit icon for a user, and have edited the name and email address of the user. Now imagine we don’t want to keep these changes, so we click the cancel button. This should cause the changes we made in the view to revert back to their previous state. However, since we overwrote our canonical state with the editing state, we no longer have the old state of the data. We would be forced to re-fetch the data from our REST API to get the canonical state again. Instead, let’s store the editing state in another place in state. Now our state might look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8311d9ce788654e4e8f700135c949fbe/href">https://medium.com/media/8311d9ce788654e4e8f700135c949fbe/href</a></iframe><p>Since we now have a copy of both the editing state of the object and the canonical state of the object, resetting back to after clicking cancel is easy. We simply show the canonical state in the view instead of the editing state, and no further calls to the REST API are necessary. As a bonus, we’re still tracking the edit state in our store. If we decide that we did want to keep our edits, we can just click the edit button again and now the edit state is shown with our old changes. Overall, keeping edit and view state separate from the canonical state both provides a better developer experience in terms of code organization and maintainability, as well as a better user experience for interacting with our form.</p><h3>3. Share state between views judiciously</h3><p>Many applications may start off with a single store and a single user interface. As we grow our application to scale out our features, we will need to manage state between multiple different views and stores. In order to scale out our Redux application, it may help to create one top level reducer per page. Each page and top level reducer corresponds to one view in our application. For example, the users screen will fetch users from our API and store them in the users reducer, and another page which tracks the domains for the current user will fetch and store data from our domain API. The state might look something like this now:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7f4dd00bd081e8dfad425aebce726f53/href">https://medium.com/media/7f4dd00bd081e8dfad425aebce726f53/href</a></iframe><p>Organizing our pages like this will help keep the data behind our views decoupled and self-contained. Each page keeps track of its own state, and our reducer files can even be co-located with our view files. As we continue to expand our application, we may discover the need to share some state between two views which both depend on that data. Consider the following when thinking about sharing state:</p><ul><li>How many views or other reducers will depend on this data?</li><li>Does each page need its own copy of the data?</li><li>How frequently does the data change?</li></ul><p>For example, our application needs to display some information about the currently logged-in user on every page. We need to fetch their user information from the API, and store it in our reducer. We know that every page is going to depend on this data, so it doesn’t seem to fit with our one reducer per page strategy. We know that each page doesn’t necessarily need a unique copy of the data, since most pages will not be fetching other users or modifying the current user. Also, the data about the currently logged-in user is unlikely to change unless they are editing themselves on the users page.</p><p>Sharing the current user state between our pages seems like a good call, so we will pull it out into its own top level reducer in its own file. Now the first page the user visits will check if the current user reducer has loaded, and fetch the data from the API if not. Any view that is connected to the Redux store can view information about the current logged-in user.</p><p>What about cases that don’t make sense to share state? Let’s consider another example. Imagine that each domain belonging to a user also has a number of subdomains. We add a subdomain page to the application which shows a list of all the users subdomains. The domains page also has the option to display the subdomains for a given selected domain. Now we have two pages that both depend on the subdomain data. We also know that domains can change on a somewhat frequent basis — users may add, remove, or edit domains and subdomains at any time. Each page will also probably need its own copy of the data. The subdomain page will allow for reading or writing to the subdomain API, and will also potentially need to paginate through multiple pages of data. The domain screen by contrast will only need to fetch subsets of the subdomains at a time (the subdomains for a given selected domain). It seems clear this isn’t a good use case for sharing our subdomain state between these views. Each page should store its own copy of the subdomain data.</p><h3>4. Reuse common reducer functions across state</h3><p>After writing a few reducer functions, we may decide to try reusing our reducer logic between different places in state. For example, we may create a reducer to fetch users from our API. The API only returns 100 users at a time, and we may have thousands of users in our system or more. To address this, our reducer will also need to keep track of which page of data is currently being displayed. Our fetch logic will read from the reducer to determine the pagination parameters to send with the next API request (such as page_number). Later on when we need to fetch the list of domains, we will end up writing the same exact logic to fetch and store the domains, just with a different API endpoint and a different object schema. The pagination behavior remains the same. The savvy developer will realize we can probably modularize this reducer and share the logic between any reducers that need to paginate.</p><p>Sharing reducer logic can be a little tricky in Redux. By default, all the reducer functions are called when a new action is dispatched. If we share a reducer function in multiple other reducer functions, then when we dispatch our action it will cause <em>all</em> of those reducers to fire. This isn’t the behavior we want for reusing our reducers, though. When we fetch the users and get a total count of 500, we don’t want the domain’s count to change to 500 as well.</p><p>We recommend two different ways to accomplish this, both using special scopes or prefixes for types. The first way involves passing a scope inside your payload in an action. The action uses the type to infer the key in state to update. For illustrative purposes, let’s imagine that we have a web page containing several different sections, all of which load asynchronously from different API endpoints. Our state to track loading might look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/79acb1de40c37acec1f7d41b2b8c3580/href">https://medium.com/media/79acb1de40c37acec1f7d41b2b8c3580/href</a></iframe><p>Given such a state, we will need reducers and actions to set the loading state for each section of the view. We could write out four different reducer functions with four different actions — each using their own unique action type. That’s a lot of repeated code! Instead, let’s try using a scoped reducer and action. We create just one action type SET_LOADING, and a reducer function like so:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f2d7c103f01b2ecbc7124ef5206524ba/href">https://medium.com/media/f2d7c103f01b2ecbc7124ef5206524ba/href</a></iframe><p>We also need to provide a scoped action creator function to call our scoped reducer. The action would look something like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9248bdd0d48e63dd665711a182ca92c6/href">https://medium.com/media/9248bdd0d48e63dd665711a182ca92c6/href</a></iframe><p>By using a scoped reducer like this, we eliminate the need to repeat our reducer logic across multiple actions and reducer functions. This significantly decreases the amount of code repetition and helps us to write smaller action and reducer files. If we need to add another section to our view, we simply add a new key in our initial state and make another dispatch call with a different scope passed to setLoading. This solution works great when we have several similar collocated fields that need to be updated in the same way.</p><p>Sometimes though we need to share reducer logic between multiple different places in state. Instead of using one reducer and action to set multiple fields in one place in state, we want a reusable reducer function that we can call combineReducers with to plug into different places in state. This reducer will be returned by calls to a reducer factory function, which returns a new reducer function with that type prefix added.</p><p>A great example for reusing reducer logic is when it comes to pagination information. Going back to our fetching users example, our API might contain thousands of users or more. Most likely our API will provide some information for paginating through multiple pages of users. Perhaps the API response we receive looks something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/42c4ba479a7a631b07d0ce02fdd599e6/href">https://medium.com/media/42c4ba479a7a631b07d0ce02fdd599e6/href</a></iframe><p>If we want the next page of data, we would make a GET request with the startElement=100 query parameter. We could just build a reducer function for each API service we interact with, but that would repeat the same logic across many places in our code. Instead, we will create a standalone pagination reducer. This reducer will be returned from a reducer factory which takes a prefix type and returns a new reducer function:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e3c55a7de80d5057b5019c7b5f91abed/href">https://medium.com/media/e3c55a7de80d5057b5019c7b5f91abed/href</a></iframe><p>The reducer factory paginationReducerFor takes the prefix type which will be added to all of the types that this reducer matches on. The factory returns a new reducer, with its types prefixed. Now when we dispatch an action like USERS_SET_PAGINATION it will only cause the pagination reducer for users to update. The domains pagination reducer will remain unchanged. This effectively allows us to reuse common reducer functions in multiple places in our store. For sake of completeness, here is an action creator factory to go along with our reducer factory, also using a prefix:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/93ed159f2b521334276b44db3220c8f5/href">https://medium.com/media/93ed159f2b521334276b44db3220c8f5/href</a></iframe><h3>5. React Integration and Wrap up</h3><p>Some Redux applications may never need to render a view to users (like an API), but most of the time you will want some kind of view to render your data inside of. The most popular library for rendering UIs with Redux is React, and that is the one we will choose to demonstrate how to integrate Redux. We can use the strategies we learned in the above sections to make our lives easier when creating our view code. To do this integration, we will use the react-redux <a href="https://github.com/reactjs/react-redux">library</a>.</p><p>One useful pattern in UI integration is using selectors to access the data in state from our view components. A convenient place to use selectors in react-redux is in the mapStateToProps function. This function is passed into the call to the connect function (the function you call to connect your React component to the Redux store). This is the place where you will map the data in state to the props that your component receives. This is the perfect place to use a selector to retrieve the data from state, and pass to the component as props. An example integration might look like the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9961ff6a5feee18d4dc203313f3906b9/href">https://medium.com/media/9961ff6a5feee18d4dc203313f3906b9/href</a></iframe><p>The integration between React and Redux also provides us a handy location for wrapping our actions in scopes or types. We have to hook up our component’s handlers to actually call the store’s dispatch with our action creators. To accomplish this in react-redux we use the mapDispatchToProps function, which is also passed to the call to connect. This mapDispatchToProps function is the place we normally call the Redux bindActionCreators function to bind each action to the dispatch method from the store. While we’re at it, we can also bind the scope to the actions like we showed in <a href="http://techblog.appnexus.com/#section4">Section 4</a>. For example, if we wanted to use the scoped reducer pattern with the paginator for our Users page, we would write the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7f5b441fd8741122f8148fa13b245629/href">https://medium.com/media/7f5b441fd8741122f8148fa13b245629/href</a></iframe><p>Now from the perspective of our UsersPage component, it just receives the list of users and other pieces of state as props, as well as the bound action creators. The component doesn’t need to be aware of which scoped action it needs or how to access the state; we have handled these concerns instead at the integration level. This allows us to create very decoupled components which don’t necessarily depend on the inner workings of our state. Hopefully by following the patterns discussed here, we can all create Redux applications in a scalable, maintainable, and reasonable way.</p><p><strong>Further Reading:</strong></p><ul><li><a href="http://redux.js.org/">Redux</a> the state management library being discussed</li><li><a href="https://github.com/reactjs/reselect">Reselect</a> a library for creating selectors</li><li><a href="https://github.com/paularmstrong/normalizr">Normalizr</a> a library for normalizing JSON data against a schema, useful for storing data in indices</li><li><a href="https://github.com/gaearon/redux-thunk">Redux-Thunk</a> a middleware library for async actions in Redux</li><li><a href="https://github.com/redux-saga/redux-saga">Redux-Saga</a> another middleware for async actions using ES2016 generators</li></ul><p>Written by <strong>Chris Dopuch</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=89452af4fdcb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/xandr-tech/five-tips-for-working-with-redux-in-large-applications-89452af4fdcb">Five Tips for Working with Redux in Large Applications</a> was originally published in <a href="https://medium.com/xandr-tech">Xandr-Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Renovating Microservice UIs]]></title>
            <link>https://medium.com/xandr-tech/renovating-microservice-uis-8f639c7b8ffe?source=rss-f77f4074e4d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f639c7b8ffe</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[ui]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[frontend]]></category>
            <category><![CDATA[resin]]></category>
            <dc:creator><![CDATA[Xandr Engineering]]></dc:creator>
            <pubDate>Mon, 01 May 2017 00:00:00 GMT</pubDate>
            <atom:updated>2017-06-13T19:39:50.294Z</atom:updated>
            <content:encoded><![CDATA[<p>AppNexus Console has evolved from a simple PHP application to a highly complex platform, built using a micro-services architecture. During this transition, we learned that scaling applications is not easy. We had to find a new approach that would simplify our architecture, reduce deployment times, and enable testing release builds on production environments. We chose to build our solution using a combination of <a href="https://nodejs.org/en/">Node.JS</a>, <a href="http://expressjs.com/">Express</a>, and <a href="https://www.docker.com/">Docker</a> — creating an AppNexus special sauce that we call Resin UI.</p><p>Before I explain the goodness that is Resin UI, it is worthwhile to go through a brief history of Console.</p><h3>AppNexus Console 1.0</h3><p>At the beginning, Console was simple. It was built on PHP using <a href="https://framework.zend.com/">Zend Framework</a> (an open source PHP application library), and it served templated HTML for each page in Console. This was easy for a single engineer, who worked on both PHP and HTML, to maintain. As the web development evolved, so did Console. It moved from raw HTML to a collection of JavaScript modules that formed the UI. The code base wasn’t the only thing that changed: the development team scaled up from a single engineer to over 50, in just five years.</p><p>As AppNexus grew, the UI team was split into two dedicated teams to better align with our customers. We now had two teams sharing one monolithic code base which presented code management, ownership, maintenance and deployment issues. Our solution was to split the JavaScript modules by team, where each team would be responsible for their own code base. However, we still experienced issues with deployment and code management, which pushed us to come up with a better approach. We decided that the best solution was to move to a micro-service architecture.</p><h3>The Era of Micro-Services</h3><p>With AppNexus’ migration to a micro-service architecture, each team became responsible for their own application, on its own servers, using Virtual Machines (VMs). Each VM had its own virtual CPU, memory, hard drive and OS. This gave teams the flexibility to control their own app and release process, while allowing our dedicated DevOps team to maintain the systems that virtualized the services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/0*_yIsUvP2W11xggHP.png" /></figure><p>With the migration, we came to the conclusion that we needed a centralized service to control Console authentication, styling and configurations. We also required a router to control traffic and direct it to the corresponding app depending on the page a client visited in Console. These were simple services that we built on <a href="https://nodejs.org/en/">Node.JS</a>, and they integrated easily with all our apps; new and old alike.</p><h3>Resin to the Rescue</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/0*4uMZFwcuIEs40RfK.png" /></figure><p>As the number of engineers and apps increased, we required more and more VMs. It quickly reached a point where we had too many VMs to maintain, and it became apparent that we lacked a standard way to build apps. This prompted one of our teams to come up with a simple solution — Resin. Inspired by Console services, Resin was a simple Node.JS server framework and toolset that used <a href="https://www.docker.com/">Docker</a>, a new type of virtualization system. Docker requires noticeably less overhead than VMs do. It does so by sharing its operating system across instances — this is known as “containerizing” the app. Resin would now serve up a web application built with React.JS that integrated nicely in the AppNexus ecosystem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/443/0*2d1EiTZw9GgRbAw-.png" /></figure><h3>Resin UI</h3><p>This brings us to today. As the new Resin architecture matured, I realized that we could take it a step further and move to a single server architecture with abstraction around external services. The single server system hosts multiple UI applications which would allow us to collapse the number of containers required to run UI applications and thus, reduce our hardware costs. <strong>Resin UI is born!</strong></p><p>It was relatively easy for us to convert container apps into the single server Resin UI architecture since most of them were single page apps that relied on little, or no server code. Motivated by <a href="http://emberjs.com/">Ember’s</a> built-in deployment, I designed Resin UI such that when we release a Resin UI app, the generated JavaScript, CSS, and image files, get uploaded to a CDN. Resin then tells the server what files to access, and where they can be accessed. With this new architecture, all the server must to do is serve up HTML.</p><p>Our web engineers can now focus on building web applications without having to worry about server maintenance. Since the app is Resin based, and comes with the same build commands, Resin UI is a seamless transition from an engineer’s point-of-view.</p><p>Applications built on Resin UI can be developed locally (with built-in support to AppNexus APIs and services required for development) and releases now take less than 30 seconds! In contrast, our monolithic PHP UI required a development server box and took about an hour to two hours to release. Resin UI has freed teams from a fixed release schedule, empowering them to release ad-hoc with limited overhead. We are delivering new features and bug fixes quicker and more often to customers, providing them a much better Console experience.</p><p>As AppNexus continues to grow, engineers have continued to see and improve the whole system (one of our core values) — even parts of the development ecosystem that clients will never see. With innovations such as Resin UI, we are working to make Console faster and ensuring that our customers get the best experience possible!</p><p>Written by <strong>Alejandro Gonzalez Sole</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f639c7b8ffe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/xandr-tech/renovating-microservice-uis-8f639c7b8ffe">Renovating Microservice UIs</a> was originally published in <a href="https://medium.com/xandr-tech">Xandr-Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Facebook Infrastructure at Scale]]></title>
            <link>https://medium.com/xandr-tech/facebook-infrastructure-at-scale-c6cd98fb9686?source=rss-f77f4074e4d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/c6cd98fb9686</guid>
            <category><![CDATA[facebook]]></category>
            <category><![CDATA[tech-talk]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[tech-events]]></category>
            <dc:creator><![CDATA[Xandr Engineering]]></dc:creator>
            <pubDate>Thu, 28 Jul 2016 00:00:00 GMT</pubDate>
            <atom:updated>2017-06-12T00:09:07.493Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*CTq3Uo7OC1AST1XUEQ6l2w.png" /></figure><p>Learn about the creation and evolution of the Release To Production (RTP) Team at Facebook. The RTP team works with Facebook hardware designers, software developers, performance and capacity engineering specialists, and external hardware manufacturers to ensure that Facebook hardware will meet the needs of their growing business. At this tech talk talk, Facebook Hardware Engineer Yashar Bayani shared their story, along with challenges they faced along the way which led to changes in how they test their hardware prior to mass production.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fme9Fhxk3y9Q%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dme9Fhxk3y9Q&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fme9Fhxk3y9Q%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ee6c7c5e08ab66f3456622a4d59ce56b/href">https://medium.com/media/ee6c7c5e08ab66f3456622a4d59ce56b/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6cd98fb9686" width="1" height="1" alt=""><hr><p><a href="https://medium.com/xandr-tech/facebook-infrastructure-at-scale-c6cd98fb9686">Facebook Infrastructure at Scale</a> was originally published in <a href="https://medium.com/xandr-tech">Xandr-Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Test Environments Enabled Organizational Growth]]></title>
            <link>https://medium.com/xandr-tech/how-test-environments-enabled-organizational-growth-6c649ad2ea86?source=rss-f77f4074e4d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c649ad2ea86</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[tech-talk]]></category>
            <category><![CDATA[tech-events]]></category>
            <dc:creator><![CDATA[Xandr Engineering]]></dc:creator>
            <pubDate>Mon, 06 Jun 2016 00:00:00 GMT</pubDate>
            <atom:updated>2017-06-12T00:09:18.063Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*qqqDl85MXggnu_xdnkT8Rw.png" /></figure><p>Watch this tech talk to learn how our DevOps team has built a scalable infrastructure (using Docker and other technologies) to automate the creation of scaled-down replicas of the production AppNexus platform and make them available company-wide as an on-demand service. Featuring AppNexus Engineering Team Lead Leevar Williams.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Ff2ohl4GBjyc%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Df2ohl4GBjyc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Ff2ohl4GBjyc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/96acc55034d25e4003e5fd51f2e9311c/href">https://medium.com/media/96acc55034d25e4003e5fd51f2e9311c/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c649ad2ea86" width="1" height="1" alt=""><hr><p><a href="https://medium.com/xandr-tech/how-test-environments-enabled-organizational-growth-6c649ad2ea86">How Test Environments Enabled Organizational Growth</a> was originally published in <a href="https://medium.com/xandr-tech">Xandr-Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to build, manage and scale an effective engineering organization ft.]]></title>
            <link>https://medium.com/xandr-tech/how-to-build-manage-and-scale-an-effective-engineering-organization-tech-talks-appnexus-ft-3f637bb21e54?source=rss-f77f4074e4d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f637bb21e54</guid>
            <category><![CDATA[tech-events]]></category>
            <category><![CDATA[tech-talk]]></category>
            <category><![CDATA[stackoverflow]]></category>
            <category><![CDATA[scale]]></category>
            <dc:creator><![CDATA[Xandr Engineering]]></dc:creator>
            <pubDate>Thu, 05 May 2016 00:00:00 GMT</pubDate>
            <atom:updated>2017-06-12T00:09:30.513Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to build, manage and scale an effective engineering organization ft. Stack Overflow CEO Joel Spolsky</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KbQVXD36PRwlFDGP.png" /></figure><p>Think hiring the best engineers is tough? What about keeping engineers engaged once they are a part of your organization? Watch the latest installment of our Tech Talks series to hear how Joel Spolsky, founder and CEO of Stack Overflow, and Igor Shindel, CIO of AppNexus, think about these issues and discuss what it takes to build and manage a world-class engineering organization.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FzTEmPwkJXrA%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DzTEmPwkJXrA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FzTEmPwkJXrA%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/3261058318ff3915e6fdaba3e379bed1/href">https://medium.com/media/3261058318ff3915e6fdaba3e379bed1/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f637bb21e54" width="1" height="1" alt=""><hr><p><a href="https://medium.com/xandr-tech/how-to-build-manage-and-scale-an-effective-engineering-organization-tech-talks-appnexus-ft-3f637bb21e54">How to build, manage and scale an effective engineering organization ft.</a> was originally published in <a href="https://medium.com/xandr-tech">Xandr-Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Securing our Big Data Platform: Part 3]]></title>
            <link>https://medium.com/xandr-tech/securing-our-big-data-platform-part-3-98c8f723455a?source=rss-f77f4074e4d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/98c8f723455a</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Xandr Engineering]]></dc:creator>
            <pubDate>Sun, 10 Apr 2016 00:00:00 GMT</pubDate>
            <atom:updated>2017-06-13T19:40:26.161Z</atom:updated>
            <content:encoded><![CDATA[<p>As we’ve seen in our past couple of posts —<a href="https://medium.com/appnexus-tech/securing-our-big-data-platform-part-1-a19d8460155d"> Securing our Big Data Platform: Part 1</a> and <a href="https://medium.com/appnexus-tech/securing-our-big-data-platform-part-2-5a0eafc713a4">Securing our Data Platform: Part 2</a> — we’ve been hard at work beefing up the security of our Big Data Platform. We had one other important piece of the puzzle to solve before we dotted our i’s on this security effort — <strong>Delegation</strong>.</p><p>Delegation is the process of letting a user act on the behalf of another user. Delegation allows user X to perform actions on behalf of user Y by granting the privileges of user Y upon user X. Thus, through delegation we are able to account for a user’s actions in our services.</p><h3>Why delegation?</h3><p>Delegation enables our systems to act on behalf of users without actually having to manage or store the user’s credentials. This is a huge win as we do not have to fret about securely storing our users’ credentials in our systems. Also, as we’d mentioned earlier, another important reason for delegation is the need to provide accountability in our system for every action. Every action needs to be tied to its rightful owner and this information should be available for potential auditability in the future.</p><p>Here is an overview of our system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LNz5jVAmtN1gCqEe.png" /></figure><p>Our Job API service is the gatekeeper to accessing the underlying infrastructure services like Hadoop and the various databases. The Job API service is fronted by a UI service that simplifies the end user’s interactions with the Data Platform. The UI service is a website built on Flask, a Python web framework, and it authenticates all users before allowing them to access the system.</p><p>Almost all of our users exclusively use the UI to manage their data platform jobs. Jobs are created, deleted, edited and viewed from the UI.</p><p>The UI service interacts with the Job API service using HTTP Restful APIs over an SSL connection. In addition, the UI service has its own set of credentials (a system user with a password) that allows it to authenticate with the Job API service before executing an API call. When a user performs an action, it is important for the Job API service to know the identity of the end user who is making the call, in order to verify the permissions of the user, and to log this information for accountability. Without proper delegation, all API calls to the Job API service would be executed using UI service’s identity. This not only meant that it looked like only one user ever used our system, but also every end user was permitted to perform any action that was permissible for the system user.</p><p>This is where delegation steps in. The idea is to let the UI service use its own credentials to authenticate with the backend services but the backend service performs the requested action as if the original user had made the request directly.</p><h3>How do we do delegation?</h3><p>We certainly didn’t want the UI service to know nor manage the user’s credentials by passing them to the Job API service. Each time an API call is made, we needed another way to pass the user’s identity and also have a way to verify the authenticity of the user’s identity.</p><h4>Authentication tokens</h4><p>Every user who interacts with our UI is authenticated with our Backend Auth service. This process enables us to verify the identity of the user. The Backend Auth service provides a wrapper around our company’s Active Directory service using the LDAP protocol. Thus, our users only need to use their regular LDAP username and password when they authenticate with the UI service.</p><p>The auth service will also generate a token for each successful authentication. The authentication token is a cryptographic hash of the user’s identity and some other information. Each authentication token is unique to a user.</p><p>This token is returned in the response and is also cached in the auth service, with a mapping to the user, for whom it was generated.</p><p>Upon receipt of this auth token, the UI service will set the token in the user’s browser’s cookie. Subsequently all interactions between the user and the UI service are authenticated using this auth token in the cookie.</p><h3>Plugging it all together</h3><p>When the UI service needs to make a call to the backend Job API service to retrieve information about jobs, it also includes the user’s name and their authentication token (from the cookie in the request) in its request to the backend Job API service.</p><p>We use custom headers in the HTTP request to send this information through.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/851965f4382501043a310d59bf5e8801/href">https://medium.com/media/851965f4382501043a310d59bf5e8801/href</a></iframe><p>The Job API service first authenticates the incoming request, which is using the UI service’s credentials.</p><p>It then parses the above headers and calls the Backend Auth service to verify the authenticity of the token and that it does indeed belong to the specified user.</p><p>The auth service consults its cache, and verifies the token.</p><p>If the auth service rejects the token, then the request is failed by the Job API service and an appropriate error is displayed by the UI service on the user’s browser.</p><p>Upon successful verification, the Job API service will now perform all the actions as if it had been directly called by the user.</p><p>All jobs and actions performed via the Job API service are authorized in terms of LDAP Groups, as described in <a href="http://techblog.appnexus.com/blog/2016/01/21/securing-our-big-data-platform-part-2/">our previous post</a>. Every user belongs to one or more LDAP groups and permissions for each action are granted at a group level.</p><p>Thus, if a user tries to perform an action that is not authorized by their group, the Job API service will fail the operation citing insufficient authorization.</p><p>In the case when a user makes the API call directly to the Job API service, usually programmatically, these delegation headers will be absent in the request. In such a scenario, we use the calling user’s identity, which is provided through their credentials in the request, to perform the requested action.</p><p>This diagram illustrates the sequence of interactions in our systems with proper delegation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gMEq7ErVVWtJDgno.png" /></figure><p>Securing our data platform is a continuous effort, and we need to be constantly vigilant to keep the bad actors out. By investing over the past half a year in beefing up the security of our platform, we’ve made it a little bit harder for the bad actors to exploit the system.</p><p>Stay tuned for more updates from the security space as we continue to improve upon and secure our platform.</p><p>Written by <strong>Sriharish Koka</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=98c8f723455a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/xandr-tech/securing-our-big-data-platform-part-3-98c8f723455a">Securing our Big Data Platform: Part 3</a> was originally published in <a href="https://medium.com/xandr-tech">Xandr-Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>