<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jake Cahill on Medium]]></title>
        <description><![CDATA[Stories by Jake Cahill on Medium]]></description>
        <link>https://medium.com/@kaelscion?source=rss-cd58749cce23------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*VgEpgChZgBvbmzNa4m0LFg.png</url>
            <title>Stories by Jake Cahill on Medium</title>
            <link>https://medium.com/@kaelscion?source=rss-cd58749cce23------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 22:56:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kaelscion/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Bye Bye 403: Building A Filter-Resistant Web Crawler Part III: User Agents]]></title>
            <link>https://medium.datadriveninvestor.com/bye-bye-403-building-a-filter-resistant-web-crawler-part-iii-user-agents-5df330ce253?source=rss-cd58749cce23------2</link>
            <guid isPermaLink="false">https://medium.com/p/5df330ce253</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Jake Cahill]]></dc:creator>
            <pubDate>Thu, 07 Mar 2019 05:35:00 GMT</pubDate>
            <atom:updated>2019-03-30T05:13:07.922Z</atom:updated>
            <content:encoded><![CDATA[<figure><a href="http://www.track.datadriveninvestor.com/1B9E"><img alt="" src="https://cdn-images-1.medium.com/max/700/0*PFl-M6yvxEif0Qls" /></a></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/1*rVARZ4J9qNkzZCiF7PV5Jg.png" /><figcaption>Bye Bye 403 Part 3: User Agents</figcaption></figure><p>So we’ve covered what web scraping is. We’ve also covered how to cycle proxy information to disguise our IP Address. And if you haven’t 1) Shame on you, 2) Why would you read a post with “Part III” without having read parts “I” and “II” and 3) Cheekiness aside, here are links to the other articles:</p><p>Part I: <a href="https://medium.com/datadriveninvestor/bye-bye-403-building-a-filter-resistant-web-crawler-part-i-what-is-web-scraping-442b268f64f3">https://medium.com/datadriveninvestor/bye-bye-403-building-a-filter-resistant-web-crawler-part-i-what-is-web-scraping-442b268f64f3</a></p><p>Part II: <a href="https://medium.com/datadriveninvestor/bye-bye-403-building-a-filter-resistant-web-crawler-part-ii-building-a-proxy-list-49ffe437f458">https://medium.com/datadriveninvestor/bye-bye-403-building-a-filter-resistant-web-crawler-part-ii-building-a-proxy-list-49ffe437f458</a></p><p>But there is another step in masking the scraper that we now must consider. To illustrate, think of it this way:</p><p>Imagine that somebody is bugging you on any social media platform and you’ve blocked them. Let’s say you’re an influencer so you get hundreds of follows every day and it is really hard to do a “profile check” of every follower to make sure that they are not the person who you’ve blocked. What would be the easiest way for you to make a preliminary guess? Obviously, checking the name of the new followers would be your first line of defense. Well, what if somebody with the same name and profile picture started following you? Would the fact that their profile snippet listed them as being from a different country than before keep you from blocking them right away? Not likely. In all likelihood, it would make you more suspicious of this person because they are being deceitful.</p><p><a href="https://www.datadriveninvestor.com/2018/11/12/web-scraping-hedge-fund-strategy/">Web Scraping and Hedge Funds&#39; Alternative Data Strategy - Data Driven Investor</a></p><p>Think of a web server bot filter as the “influencer with an annoying follower”. By cycling our IP address, we change our respective location ID, and “profile pic”, but we are still listed as a connection from the same type of laptop, with the same OS version, from the same browser, etc. It’s pretty easy to peg and filter. So what is a budding web scraper to do? Cycle your User-Agent.</p><p>A User-Agent is essentially a field in your HTTP/S headers that tells the web server the browser version, host OS version, host machine form factor, and some other small tidbits about every request that comes through. This serves two purposes: diagnostics of the web content (to see if a certain bug is only reported on certain OSes, Browsers, or some combination of the two) and security (to see who’s connected and what are they connecting from to track possible offenders during and/or after a security incident). So in layman’s terms, a User-Agent tells the web server about the connecting machine, whereas an IP address tells it more about where you’re coming from. Make sense? Good!</p><p>To begin cycling User-Agents, head <a href="https://deviceatlas.com/blog/list-of-user-agent-strings#desktop">here</a> for an up-to-date collection of desktop User-Agents:</p><p><strong><em>Pro Tip: We can also scrape the mobile versions of websites by changing the User Agent. That is actually how the “show desktop site” option on Mobile Browsers works. The browser will spoof your User Agent as if it came from a desktop device despite the fact that you are on mobile, thus showing you the “desktop version” of any content. The same works in reverse for scraping mobile-only content</em></strong></p><p>Now, we could just copy/paste all of these user-agents into a file, but this is a web scraping post series after all so instead, create a new project file and call it user_agent.py</p><pre>from bs4 import BeautifulSoup</pre><pre>import random<br>import requests</pre><pre>class UserAgent:</pre><pre>    ua_source_url=&#39;https://deviceatlas.com/blog/list-of-user-agent-strings#desktop&#39;</pre><pre>    def __init__(self):<br>        self.new_ua = random.choice(self.get_ua_list())</pre><p>So first we’re making the UserAgent class. If you’ve made it this far into the series you can probably put together that I am a big fan of encapsulation. This is largely due to coming from the Network Engineering and IT Technical worlds before I started writing code where, it is assumed, that an incident <strong>WILL</strong> wake you up <strong>EVERY TIME</strong> you are on call at no earlier than 4 AM. Because of this, I am a huge fan of creating things that are easily debugged. Getting paged at 4, getting to the office by 4:30, and leaving by 4:45 to actually have some semblance of a morning with my family is preferred to taking 3 hours to fix a simple issue that was buried under layers of junk. Because if it takes me 3 hours, then I have solved my problem by 7:30. You know, just in time to start “normal” work for the day…</p><p>That being said, the UserAgent class is going to handle all the user-agenty things for us! The class variable we’ve created ua_source_url is outside of the __init__ function, as you might have noticed. Why is that? Because everything inside __init__ only “exists” after an instance of the class is called (aka instantiated) in your code. Class level variables can be referenced directly and more importantly, are shared among all instances of that class. Meaning when we multithread this bad boy, we won&#39;t have to store that particular string in memory more than once.</p><p>Within our __init__ function is the new_ua property which we want to be different for each instance of this class but not be pulled into existence until the class is created.</p><p><em>Side Note: For a further understanding of </em><em>__init__, what it does, and how to use it, stick around. There is a post coming on that very soon!</em></p><p>The new_ua property picks a random user agent from a list of user agents. Where will we get that list you ask? From the code your about to type:</p><pre>def get_ua_list(self, source=ua_source_url):<br>    r = requests.get(source)<br>    soup = BeautifulSoup(r.content, &quot;html.parser&quot;)<br>    tables = soup.find_all(&#39;table&#39;)<br>    return [table.find(&#39;td&#39;).text for table in tables]</pre><p>So what have we done here? Well, we have scraped the page above (as we do) for the user-agent information listed on that page. As with all of our little scraping scripts, it started with a Requests get() function, fed the response content to our HTML parser (BeautifulSoup), found the table markup for each user-agent, and extracted the text of the data of each table. Now that this function is complete, does the __init__ make a bit more sense? If not, allow me to explain:</p><p>As soon as we create an instance of the UserAgent class ANYWHERE in our code, it will run this function without us needing to touch it. When the function completes, it will have assigned the list of potential UserAgents to the property new_ua. How is this relevant? Take a look at the terminal screen capture below which shows the output of the following code:</p><pre>i=1<br>while i &lt;= 4:<br>    print(UserAgent().new_ua)<br>    i+=1</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/0*RuAKLvu16CpBPwhl.png" /></figure><p>As you can see, we do not need to write the user agents to a file, which involves opening the file, reading it, then closing it again, relying on disk memory. Instead, we read this list into RAM (the faster, easier-to-clean storage) and retrieve it only when we need it, getting a different UserAgent each time through the random.choice() function</p><p><strong><em>Pro Tip: For those of you who are unfamiliar with </em></strong><strong><em>random.choice(), it basically does the same thing when selecting from a list or iterable as </em></strong><strong><em>random.randint(0, len(sample_list)). Simply put, it selects a pseudo-random item from any iterable data structure you pass to it. In instances like this, </em></strong><strong><em>random.choice() is my preference because both approaches do roughly the same thing and </em></strong><strong><em>.choice() is fewer keystrokes.</em></strong></p><p>Alright, Mavericks (Season of the Drifter anyone 😉) you’ve got the tools to fool 99% of bot filters. But do you know how to use them? Head out into the Wild and find out, then report back. I’d love to hear your stories of the datasets your gathering and what you plan to do with them! Successes and failures are equally welcome because this completes the first set of countermeasures we’re going to discover. These are referred to (by me at least) as the “hard skills” of web scraping. The skills we need to develop our knowledge. But how are your instincts? Find out next week in part four when we break into the intermediate stuff: Timing and Throttling!</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fb2a0d6%3Fas_embed%3Dtrue&amp;dntp=1&amp;display_name=Upscribe&amp;url=https%3A%2F%2Fupscri.be%2Fb2a0d6%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/0707f5c806284d01a4a13c7b13a91ce3/href">https://medium.com/media/0707f5c806284d01a4a13c7b13a91ce3/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5df330ce253" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/bye-bye-403-building-a-filter-resistant-web-crawler-part-iii-user-agents-5df330ce253">Bye Bye 403: Building A Filter-Resistant Web Crawler Part III: User Agents</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Leadership: That Concept that Actually *Doesn’t* Mean That You’re In Charge]]></title>
            <link>https://medium.datadriveninvestor.com/leadership-that-concept-that-actually-doesnt-mean-that-you-re-in-charge-caea9208d154?source=rss-cd58749cce23------2</link>
            <guid isPermaLink="false">https://medium.com/p/caea9208d154</guid>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[leadership]]></category>
            <dc:creator><![CDATA[Jake Cahill]]></dc:creator>
            <pubDate>Fri, 14 Dec 2018 06:24:26 GMT</pubDate>
            <atom:updated>2018-12-14T15:02:19.824Z</atom:updated>
            <content:encoded><![CDATA[<figure><a href="http://www.track.datadriveninvestor.com/P12O"><img alt="" src="https://cdn-images-1.medium.com/max/700/0*3QtunKYcE2KdO9pn" /></a></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lYz-Sf4wNfCx4Clv" /><figcaption>Photo by <a href="https://unsplash.com/@weilstyle?utm_source=medium&amp;utm_medium=referral">Ethan Weil</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*KzKrLfz0C6FDnVF-hDcLHA.png" /><figcaption>Leadership in Technology</figcaption></figure><p><strong><em>A true leader is not one who is in charge, but one who is held responsible. They must understand and accept the burden that success should reflect their team, and failure will reflect their leadership. It is a burden to bear, not a prize to be won. All good leaders seek to create change that they can pass on to their successors, allowing them to accomplish the goals of a concept they showed to the world.</em></strong></p><blockquote><strong>Originally published on the Coding Duck blog: </strong><a href="https://www.ccstechme.com/coding-duck-blog"><strong>ccstechme.com/coding-duck-blog</strong></a></blockquote><p>So, let’s start this post by talking about something that everybody agrees on: relationships. How many guys out there believe that it is their right to be “in charge” of things in relationships? How many women feel that they are expected to “sit meekly in the corner and agree”? Probably a lot of those of us in heterosexual relationships. Even in relationships where gender is not assigned a role, one of the two probably feels that their job is to lead and make decisions for the family. There are many schools of thought on why this is: the Bible saying women need to be slaves to men, a male-dominated business and political landscape, modern and overt chauvinism, and the like.</p><p>While I feel that arguments can be made to each of those conclusions, there really is no material that says men must take women by the hair, and women must let them. The Bible itself tells men, in slightly more elegant language, that a man who doesn’t listen to what his wife has to say and is a domineering jerk is not only a moron but also <em>does not</em> have God’s favor. So, why is there this idea that men need to be dominant and women not?</p><p>It is for the same reason modern managers and CEOs feel that “their word goes” and “nobody has a right to tell them anything because <em>they</em> are running the show”: perception. The world we live in has a really screwed up method of applying the definition of the word “leadership”. Webster’s Dictionary’s first definition of leadership is:</p><p><strong><em>the office or position of a leader</em></strong></p><p>Which is unsurprising. However, the <em>second</em> definition is:</p><p><strong><em>capacity to lead</em></strong></p><p>and the third is:</p><p><strong><em>the act or an instance of leading</em></strong></p><p>See the pattern? A leader is not simply one with a title or qualifications. Leadership is an active concept that requires capacity and fitness. Why?</p><p>First of all, because a leader is not an authority but rather, an example. Too much power today is given to words spoken in keynotes at conferences or social media grandstanding. The concept of “actions speak louder than words” is largely lost in modern society because people now see <em>giving</em> a keynote or choosing to <em>post</em> on social media as actions when, in reality, both are simply forms of people talking.</p><p>Second, many see the benevolent actions of a <em>company</em> or the ignorance of a <em>country</em> and equate that to its leader or prominent board members. But Satya Nadella does not personally donate to high profile charities and post it on LinkedIn, Microsoft does. Microsoft then writes the money off as a tax deduction. That is a <em>business</em> decision designed, initiated, and marketed by several teams of people with the goal of improving an image. Nothing against Satya, I’m sure he’s a great guy, but the reality is that he gets credit for what his company does.</p><p>Elon Musk is another great example. We always refer to the companies that <em>he</em> built. That <em>he</em> is solely responsible for the success of SpaceX, Tesla, The Boring Co, etc. A tweet of his once read<em> “Oh yeah, I’m also building a Cyborg Dragon.”</em> Really? Are you going to personally descend into SpaceX’s laboratories and invent the biological and chemical, not to mention mechanical components necessary to build a mythical creature that is part animal, part machine? Are you sure you aren’t going to just put your signature on a check for several billion dollars of SpaceX’s money that will pay some experts in those fields to do that for you while you talk to press?</p><p>You see, the god-like status of leaders over centuries, and especially in recent decades, has resulted in leaders actually starting to believe that they <em>are</em> the ones solely responsible for success. They seem to have completely forgotten that, without teams of people that get paid to do stuff, they would be just another person with an idea. Of course, there are exceptions to this rule. Indie Hackers are a great example.</p><p>But leadership, <strong><em>true</em></strong> leadership, is a person who has the humility to acknowledge that, even if they put in the hard work of building something from the ground up, eventually they become a figurehead. The job of a true leader is to receive criticism of the <em>brand</em> they represent and pass credit onto the <em>people</em> they represent. Being a real leader is hard because it means your job is to always take the blame and never take the credit. Many current leaders have this concept totally backward.</p><p>Why is it that Mark Zuckerburg is said to be worth the same amount that Facebook is worth? Are you telling me that the CEO of the company makes as much money as the company itself? Not at all. Or, at least it shouldn’t be valued that way. He is a man who, according to his corporation, is assigned a salary that is assuredly <strong>not</strong> $40B/year. But now that things have hit the fan with what his employees have done, more people blame Sheryl Sandberg than they do him. Sure, he receives plenty of flak, but even in leaked all-hands meetings, he tells his employees “we” need to be better, “we” need to do this-and-that. Okay, but when things were going well, all mention of the company was “Mark Zuckerburg’s Facebook has doubled in revenue blah blah blah”. So if it was his company then, why isn’t Facebook’s epic nosedive <em>his</em> problem and his alone?</p><p>Leadership is not power, is not control. Leadership is responsibility, and responsibility comes with a lot of cons and very few if <em>extremely</em> valuable pros. Responsibility is hard, demanding, and arduous work that requires you to look beyond yourself, even removing yourself from your position if/when your leadership has proven ineffective or damaging. You are not your product or your brand. These things are a result of you. But just like a young child’s actions are typically blamed on the parents, true leaders accept that they will lead a life of making sure that the people they surround themselves with succeed and should hope, not to preserve their own dominance, but train and nurture those they lead to ultimately replace them. It is difficult, humbling, takes courage and patience, and seems to be totally lost in the world we live in.</p><p>But we, as innovators, thinkers, builders, and problem-solvers have the power to change this. We have the power to do what so few desire to. If we wish to lead, we must do so from the front. We will not stand in an operations base miles from battle and radio orders for our subordinates to fall into the line of fire. We will lead by example, training those that report to us to pick up our banner when we inevitably fall in battle. And we will fall. We hope to fall. And we hope to do so mounted on a horse in front of our soldiers on foot, being the first to take whatever comes their way so as to spare those we lead and therefore, support.</p><p><em>Originally published on the Coding Duck blog: </em><a href="https://www.ccstechme.com/coding-duck-blog/leadership-that-concept-that-actually-doesn-t-mean-that-youre-in-charge"><em>www.ccstechme.com/coding-duck-blog</em></a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fb2a0d6%3Fas_embed%3Dtrue&amp;dntp=1&amp;display_name=Upscribe&amp;url=https%3A%2F%2Fupscri.be%2Fb2a0d6%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/0707f5c806284d01a4a13c7b13a91ce3/href">https://medium.com/media/0707f5c806284d01a4a13c7b13a91ce3/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=caea9208d154" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/leadership-that-concept-that-actually-doesnt-mean-that-you-re-in-charge-caea9208d154">Leadership: That Concept that Actually *Doesn’t* Mean That You’re In Charge</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Programming Flavors of the Month: Why I feel We Need Them]]></title>
            <link>https://medium.datadriveninvestor.com/programming-flavors-of-the-month-why-i-feel-we-need-them-431951e72f15?source=rss-cd58749cce23------2</link>
            <guid isPermaLink="false">https://medium.com/p/431951e72f15</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[code]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Jake Cahill]]></dc:creator>
            <pubDate>Mon, 10 Dec 2018 17:45:26 GMT</pubDate>
            <atom:updated>2018-12-14T15:10:14.789Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong>Originally published on the Coding Duck blog: </strong><a href="https://www.ccstechme.com/coding-duck-blog"><strong>ccstechme.com/coding-duck-blog</strong></a></blockquote><p>React, Vue, Angular, Ember, Typescript, TensorFlow, PowerBI, Blockchain, Machine Learning, Neural Networks, NoSQL, and the list goes on…and on…and on. So, other than to prove the long-hypothesized theory that you certainly <em>can</em> unite developers of all areas into a worldwide riot with a simple list of 12 recent technologies, why did I lead with that list?</p><p>The reason is two-fold. 1) I want you to be either grinding your teeth or cynically chuckling to yourself right now, and 2) Because I think you need to take a look at that list again whether you want to or not.</p><p>But why? Well, because that list, and our reactions to it, make a strong point that speaks to one of the greatest strengths and weaknesses of the development and engineering community: bias. It does not matter if you are biased for or against a technology. What matters is that we, as developers, seem to be simultaneously driven by a desire to create change <em>and</em> avoid change.</p><p>It is my honest opinion that we <em>need</em> buzzy, flavor-of-the-month technologies in our daily lives. Allow me to illustrate this point using a product most of us have heard of: ShamWOW. For those of you who <em>don’t</em> know, the ShamWOW was a stupid microfiber shammy like many others that had a really creepy guy demonstrate its use in a commercial, QVC-style. When this commercial was flooding our lives every time we turned on the TV, most of us went through the natural reaction cycle we all do with products like that:</p><p><strong>1. Good commercial, but I wouldn’t get on an elevator with that guy, let alone buy something from him.</strong></p><p><strong>2. Okay, this is getting annoying. I get it: “You followin’ me camera guy?”, just enough with it already!</strong></p><p><strong>3. Do you really think it actually <em>does</em> get half the cola from the rug with no pressure? I totally doubt it.</strong></p><p><strong>4. You know what? I’m thinking of buying that stupid thing just so that I can put a video on YouTube that shows what a load of crap this is.</strong></p><p><strong>5. Hey, my ShamWOW is here.</strong></p><p><strong>6. Huh. It actually <em>is</em> really absorbent.</strong></p><p><strong>7. This cleans fingerprints off my tablet and phone really well too…</strong></p><p><strong>8. [You now own a drawer full of cheap microfiber shammies for screen care that you never would have thought twice about otherwise]</strong></p><p>Weird right? See, while many people get annoyed by things but are capable of letting it go, problem-solvers need closure on an issue before it leaves their mind. As developers, we are lifelong problem solvers who like to get answers for ourselves rather than rely on the word of others. Sometimes, we see the value in something and want to test it to see if it lives up to expectations. Other times, we think something is stupid and simply want to test it out so that we can report to the world just <strong><em>how</em></strong> stupid.</p><p>Regardless of our reasons, the result is the same. What other industries innovate, change, and back away from failures as quickly as modern Software? Even Microsoft, the Methuselah-sloth of admitting a product is a lost cause, is calling it quits on Edge in favor of a Chromium-based browser.</p><p>In contrast, Ford Motor Company <em>still</em> uses live-axle rear ends on their Mustangs. And sure, we all love a car that goes sort of fast in a straight line, then bursts into flame when you turn the wheel to corner, before rolling down a hill and exploding. But maybe there is a better set up? You know, like the kind of set up used by every other performance car, and most non-performance cars, for the past 25 years?</p><p>My point is that I believe we need to have a constant interchange of ideas. While annoying, I think being bombarded with assertions that this new way of doing things will “change the face of software”, encourages us to attempt to either prove or disprove, this assertion. This, in turn, breeds discussion (or argument) which forces those on both sides to take a deeper look at the technology in question. In doing so, the project gets forked, its weaknesses improved, and a new flavor of the month is born.</p><p>In short, love it or hate it, the new hotness is good for Software. Not because it really is better than everything else, as it claims. But rather, because it inspires millions of problem solvers to stand up and test the assertions of its creators, thereby improving the product. It also allows those problem solvers to learn how to solve the same problems in a totally different way, making them better problem solvers for the next time a new JS Framework claims it makes Time Travel possible.</p><p><em>Originally published at </em><a href="https://dev.to/kaelscion/programming-flavors-of-the-month-why-i-feel-we-need-them-3n1m"><em>dev.to</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=431951e72f15" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/programming-flavors-of-the-month-why-i-feel-we-need-them-431951e72f15">Programming Flavors of the Month: Why I feel We Need Them</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Context: Social Media’s Most Tragic Victim]]></title>
            <link>https://medium.com/@kaelscion/context-social-medias-most-tragic-murder-57ce1ae6a807?source=rss-cd58749cce23------2</link>
            <guid isPermaLink="false">https://medium.com/p/57ce1ae6a807</guid>
            <category><![CDATA[social-media]]></category>
            <category><![CDATA[instagram]]></category>
            <category><![CDATA[facebook]]></category>
            <category><![CDATA[twitter]]></category>
            <dc:creator><![CDATA[Jake Cahill]]></dc:creator>
            <pubDate>Fri, 07 Dec 2018 21:29:56 GMT</pubDate>
            <atom:updated>2018-12-14T15:03:05.777Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong>Originally published on the Coding Duck blog: </strong><a href="https://www.ccstechme.com/coding-duck-blog"><strong>ccstechme.com/coding-duck-blog</strong></a></blockquote><p>Not too long ago I was reading a book about a small town. Everybody in this small town knew each other well while not everybody got along, most people respected their differences enough to mind their own business. It was classic small town stuff: the Mayor’s father had also been Mayor, the Sheriff never attended an academy or College, and the richest guy in town would be considered a small-time property management company anywhere else in the world. The town drunk knew that the cot in the police station’s single jail cell was pretty much free for his use, and all the local cops knew that he was more of a nuisance than a threat. When a child was young, everybody let the parents be parents the way that they personally had decided to. When that child was a young adult, every person in the town looked out for them, and they caught hell from all of their neighbors when they did the stupid things that young adults inevitably do. But, once the hell-catching was finished, the matter was dropped and that young adult grew from the experience, and moved on without ever having that incident thrown in their face again. Ultimately, the locals had known that child since they was born, and also knew that kids do stupid stuff. The past was the past, and the man or woman that child became was a better person for having done stupid things and learned from their mistakes.</p><p>While most people probably think the aforementioned book was about some small town in Montana in the 1950’s, it wasn’t. In fact, there was no book at all. What I just described was what it was like for me to grow up as a kid in the early 90’s. And I didn’t live in a small town, I lived in one of the biggest cities on the East Coast of the US. So what happened?</p><p>There are a lot of answers to that question: Facebook, Twitter, Instagram, and SnapChat are some of the more famous. And while many people will get to this point and think something like <em>If you do something dumb, don’t share it on social media</em> or <em>The world is a more connected place now so we’re just going to have to get used to this.</em> And while the first statement is absolutely try, I disagree with the second. I’ve been in technology since I was a teenager. The world was an extremely connected place when I started. But this problem of people being crucified for past mistakes, errant tweets, misguided attempts to be funny, etc, is a problem that is fairly recent. James Gunn, and more recently, Kevin Hart are great examples of this. Why is it that these gentlemen are getting in more trouble now than they did <em>when they first posted those inflammatory tweets?</em> It’s not like Twitter didn’t exist back then. And its not like the issues they said stupid things about, namely Alternate Sexuality and Pedophilia, weren’t <em>extremely</em> sensitive issues. So why now?</p><p>Granted, I am not defending what was said by either of those guys. Honestly, it is none of my business what they say online. That is their choice and its not like me saying anything would have made a difference to them at that moment anyway. But what I find bizarre is the fact that Social Media becoming this titanic soap-box is relatively new. Even politicians use it for grandstanding more than ever, and celebrities are using these platforms more and more to force onto us opinions or products we can largely live without.</p><p>My belief is that Social Media needs limits. I don’t mean regulatory limits (although that is definitely part of it), I mean content limits. Platforms have been trying different versions of this for a while now. Twitter’s 160 character limit, Vine’s 6-second video limit, Instagram’s 12-second video limit, to name a few. But the problem with these limits is that you can post <em>as much content as you want as long as it is within these constraints</em>. There were no limits to how many 6-second videos you could post on Vine. No constraints to the number of tweets allowed in a given time. So what happens? Tweets, videos, photos, etc get cherry-picked tabloid-style from hundreds and hundreds of unoffensive pieces of content to spin somebody in the light you choose. What if you could only tweet 6x per month? Well, then you’d have to be very careful about what you put online. And if somebody used most of their limited tweets to say horrible things or grandstand with a ridiculous amount of bias, you wouldn’t need to cherry-pick, it would be pretty obvious what their true feelings are.</p><p>For example, I follow a lot of CEOs and Tech people on LinkedIn. I also follow a lot of technology pros and celebrities I like on Twitter. From their social media profiles, Richard Branson, Bill Gates, Marissa Mayer and Karen Hunter all seem like brilliant, compassionate people that <em>really </em>care about the world and want it to be a better place. On the other hand, people like Elon Musk, Sheryl Sandberg, Mark Zuckerberg, and pretty much any female politician are portrayed as whiny, irritating, self-absorbed babies from affluent families that don’t know what real problems are. Now, I have never met a single one of these people. I doubt I’ve ever been in the same State as one. And yet, because they use limited space to say something to a huge audience, we judge their entire existence. Think about it, if all you knew about Elon Musk is that he was creating fully electric cars people actually want, and technology to terraform Mars just in case the electric cars are too little too late for Earth, you would wonder how close he <em>actually </em>was to completing a real Iron Man suit and fighting crime. But because he weaponizes the ‘MuskBros’, and apparently makes it <em>quite </em>obvious that he believes women should leave journalism and “go back to the kitchen”, the only Tony Stark reference we make toward him is that he is “pre-cave Tony” and needs to be captured by militants and forced to turn his own technology into a devastating weapon in order to get straightened out. So which person is he? Iron Man Elon, or “Pre-Cave” Elon? The same could be asked of any influencer on any platform anywhere.</p><p>My advice? Rather than trying to guess by a celebrity&#39;s Twitter feed, I would implore all social media users to try something for me: <strong>stop caring</strong>. Why is it that we allow our lives to be shaped, our moods controlled, and outrage to be stoked by somebody we have never met, spoken to, or even <em>exchanged emails with?</em> If you’ve been keeping up on Facebook’s latest scandal, you will see that they were well aware that the content on their platform was not always fully truthful, and was outright fabricated at times. How do you really know a person if all you see is their social media profile? Is a person’s life, values, and character so easily reduced to 160 characters or 12 seconds? Should we really be encouraging this? To me, we are all well aware that a magician making an elephant disappear in front of our very face is an illusion. Despite the fact that we saw every moment leading up to, and away from this amazing feat. We know there is some nonsense going on that we are simply not aware, and yet, we have allowed ourselves to be convinced that the brief look into a life that we are <em>shown</em> with only the details, context, and facts that somebody <em>decides</em> to show us is telling us the entire story. Perhaps, the sainthood or demonizing of a person on Social Media should inspire us to wonder what trick of the eye was used, and where that “elephant” <em>really</em> disappeared to.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57ce1ae6a807" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bye Bye 403: Building a Filter Resistant Web Crawler Part I — What is Web Scraping?]]></title>
            <link>https://medium.datadriveninvestor.com/bye-bye-403-building-a-filter-resistant-web-crawler-part-i-what-is-web-scraping-442b268f64f3?source=rss-cd58749cce23------2</link>
            <guid isPermaLink="false">https://medium.com/p/442b268f64f3</guid>
            <category><![CDATA[learning-to-code]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Jake Cahill]]></dc:creator>
            <pubDate>Wed, 05 Dec 2018 23:10:59 GMT</pubDate>
            <atom:updated>2018-12-14T15:04:11.325Z</atom:updated>
            <content:encoded><![CDATA[<h3>Bye Bye 403: Building a Filter Resistant Web Crawler Part I — What is Web Scraping?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/1*9NoncllyNxzqTsqutTZFmQ.png" /><figcaption>What is Web Scraping?</figcaption></figure><blockquote><strong>Originally published on the Coding Duck blog: </strong><a href="https://www.ccstechme.com/coding-duck-blog"><strong>ccstechme.com/coding-duck-blog</strong></a></blockquote><p>If you program with Python, or are interested in the topic, a great deal of information surrounding the language in 2018 is its usefulness in Machine Learning and Data Analysis with Frameworks like SciKit Learn, Tensorflow, Pandas, NumPy and the like. Thing is, if you have no data to analyze, your kind of stuck. In that regard you have two options: Use a service like Quandl to get your data, which is great, but if your like me, the data you want to use is almost always locked behind a paywall. Your other option, is to go out and collect the data yourself, which is my personal preference.</p><p>Doing so requires skills in another, slightly less buzzy, strength of the language: web scraping/crawling/aggregation, which is the use of automated processes to collect publicly available data from public space on the internet. Now, these terms are used to describe slightly different things, but the difference is moot in the early stages of your journey on the topic. However, regardless of what sub-niche your bot is dedicated to, you will inevitably encounter issues with web server filters and firewalls.</p><p>Now please be aware before beginning, the legal ramifications of web-scraping will vary widely based on the jurisdiction in which you perform your data collection. The ethics of the issue remain the same, respect the website and respect the owner. However, there is an inherent risk with scraping a web page that you do not have permission to scrape. When in doubt, PLEASE reach out to the owner of the website and request permission to scrape their content. I will bear no responsibility if you abuse the idea of “public space” and get into trouble. Again, <em>PLEASE RESPECT THE CONTENT OF THE SITE YOU ARE SCRAPING</em> and realize that these websites have finite limits of requests they can handle in a given amount of time. Those resources are meant to serve customers that can potentially buy from them, which your bots will use with no intent of purchase. While web scraping is a fun hobby that has the prospect of growing into a very large industry as data scientists need more and more data to analyze, right now there are a lot of “gray areas” surrounding the topic that could play out differently depending on who hears about it and gets upset. Enjoy this post series, but be respectful.</p><p>That is where the ethics come into play. Even with the GIL limits of Python, concurrent, asynchronous, and even true parallelism are becoming more and more a part of the language’s culture. Along that line of thinking, spawning 20–30+ daemon threads and tossing them at a web server in the form of Requests Sessions is not that difficult if you machine has the horsepower. While that may not sound like all that much, some cheaper web hosting tiers on eCommerce platforms like Shopify, Woocommerce, or WordPress could easily see a significant reduction in performance, or a complete lock up from those threads hitting their server every few seconds. Some of those servers are only using 1GB of RAM and a single processor core so a great rule of thumb is this: If my livelihood or business interest was tied to a server with that little horsepower, how much of it would I be willing to allow a service to access that provides a 0% change of a sale? The trick to this rule is to put yourself in the site-owner’s shoes. Yes, I agree, if your livelihood were based on your server, give it more resources. But that is NOT your decision, as a data collector, to make. No legal entity is going to care if the person who comes after you should have done a better job with their server. The argument will ultimately be: Was your site working fine until this person sent their bot after you? Yup. Okay, so Mr/Ms Web Scraper, the fine you now owe this person is… Basically, the data you are collecting belongs to somebody else. Respect them, respect their data, and respect their website. Follow those rules, and you will find success, or at the very least, avoid any unwanted attention.</p><p>In this tutorial series, we’re going to discuss, not only how to avoid being blocked by a web filter, but also how to not hammer a web server unnecessarily when collecting data. While this series will cater specifically to Python, the process and principles involved can be translated to any language that has an HTTP requests module and HTML parser.</p><p>So, now that you know what web scraping is and why its important, we’re simply going to set up our environment.</p><p>first thing’s first, open up your favorite editor and create a file called: requirements.txt.</p><p>In that file, type the follwing:</p><pre>requests<br>bs4<br>pandas</pre><p>These libraries all assist in the setup of our environment, retrieving of web pages, and the parsing of their HTML. The final module, pandas, will be used later on to parse, sort, clean, and analyze the data we collect.</p><p>Next step is to install our dependencies. But to do that while avoiding messing with the python environment on our current machine, we are going to create a virtual environment for this particular Python setup to live in. Since we are going to be using Python 3 for this series, make sure you have that installed and in your PATH, even if the default python is 2. Then, open your respective terminal application and type:</p><p><strong>pip3 install virtualenv</strong></p><p>From there, enter the command:</p><p><strong>virtualenv -p python3 venv</strong></p><p>this process is telling your Virtualenv to create a python 3 environment that is totally separate from your computer’s global environment where you can install packages, dependencies, python versions, and other goodies that are only accessible from within that environment, effectively keeping your global machine environment from getting confused. This comes in handy when you want to be able to pick and choose which python version will run with which project, or which versions of certain libraries you want to use based on what other dependencies that project needs that may not play nice with later or earlier versions of that module.</p><p>To use our virtualenv, we need to type the following:</p><p><strong>source venv/bin/activate</strong></p><p>This command activates the environment and gives you access to the dependencies, environment variables, etc of venv (btw, when you create the virtualenv, you can name it whatever you want, you need not user the name “venv”. This is simply a making convention that had been widely adopted by the community).</p><p>Now that we have activated our Python 3 environment, let’s install the dependencies we put into requirements.txt</p><p><strong>pip install -r requirements.txt</strong></p><p>This will install all of the dependencies listed in or text file in one command rather than “pipping” them one at a time.</p><p>Alright, our environment is set up and ready to go! In the next post, we’ll start building out our bot, then proceed to make it filter resistant!</p><p><em>Originally published at </em><a href="https://www.ccstechme.com/coding-duck-blog/bye-bye-403-building-a-filer-resistant-web-crawler-part-one-what-is-web-scraping"><em>www.ccstechme.com</em></a><em>/coding-duck-blog.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=442b268f64f3" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/bye-bye-403-building-a-filter-resistant-web-crawler-part-i-what-is-web-scraping-442b268f64f3">Bye Bye 403: Building a Filter Resistant Web Crawler Part I — What is Web Scraping?</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bye Bye 403: Building a Filter Resistant Web Crawler Part II — Building A Proxy List]]></title>
            <link>https://medium.datadriveninvestor.com/bye-bye-403-building-a-filter-resistant-web-crawler-part-ii-building-a-proxy-list-49ffe437f458?source=rss-cd58749cce23------2</link>
            <guid isPermaLink="false">https://medium.com/p/49ffe437f458</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[learn-to-code]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Jake Cahill]]></dc:creator>
            <pubDate>Wed, 05 Dec 2018 18:42:24 GMT</pubDate>
            <atom:updated>2019-04-08T14:32:44.946Z</atom:updated>
            <content:encoded><![CDATA[<h3>Bye Bye 403: Building a Filter Resistant Web Crawler Part II — Building A Proxy List</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/1*dfs60i23qM6nCgTqW0VxjA.png" /><figcaption>How to Build A Web Scraper Proxy List</figcaption></figure><blockquote><strong>Originally published on the Coding Duck blog: </strong><a href="https://www.ccstechme.com/coding-duck-blog"><strong>ccstechme.com/coding-duck-blog</strong></a></blockquote><p>Woohooo! We’ve got our environment set up and are ready to start building our bot! Seeing as the last post in this series (located <a href="https://www.ccstechme.com/coding-duck-blog/bye-bye-403-building-a-filer-resistant-web-crawler-part-one-what-is-web-scraping">here</a>) was mostly informational, let’s get right into the code for part two.</p><p>So, it is my assumption that if you are reading this series so far, that you know the basics of web scraping in Python, and that you are looking to learn how to not get blocked rather than the basics. So lets get right into the first line of defense we have against web filters: IP Address cycling</p><p>Many web scrapers will purchase Proxies that they are allowed to change once or twice a month for the purpose of not getting filtered by address. But did you know that an IP is usually filtered in under 100 requests and, seeing as most scrapers make at least 100 requests an hour, purchasing enough IP address proxies to last a month can get expensive.</p><p>There are free options out there, but the disadvantage there is that those IP Address lists are usually monitored and only last a couple hours before they’re blacklisted. But there is some good news: Free Proxy providers (good ones anyway) originate new IPs faster than their old ones get pinged. So we’re going to use my personal favorite: <a href="http://free-proxy-list.net/">free-proxy-list.net</a></p><p>To use this effectively and account for the IPs we’ve used and abandoned, we’re going to build a web scraper to retrieve the IPs dynamically! First step in that process, and the topic of this article, will be to build our proxy list. First step is to create a new file called proxy_retriever.py that contains the following:</p><pre>import requests<br>import pandas as pd</pre><pre>from bs4 import BeautifulSoup</pre><pre>class proxyRetriever():</pre><pre>    def __init__(self):<br>        self.s = requests.Session()<br>        self.ip_list = []</pre><p>Pretty simple so far. We’ve imported our http library (requests) and our html parser (BeautifulSoup) as well as the library we’re going to use to build the CSV file containing our proxies (pandas). Next, we create a new class called proxyRetriever to handle all actions involved in building our list. Now, you may want to lay out your project differently as far as the OOP concepts you use. But, in my personal opinion, I feel that an application that is as compartmentalized as possible makes life easier for yourself and everybody who uses your code to come. Basically, every class handles one general task (building a proxy list for example). Each function within that class will handle one specific task, the more specific the better. This way, your code will be very DRY and debugging will be a cinch because if something doesn’t work correctly, you will have a very good idea of where the problem is based solely on what class and function is supposed to handle that one, very specific thing.</p><p>You will also notice that we are defining a requests Session here rather than simply a requests.get(). There are a few reasons for this that I find superior than repeatedly using one-off HTTP requests, but a specific post on that is coming soon so I won’t go into it here. Just play along for now.</p><p>The second class-level property is the ip_list we will be building. Defining our properties in this way just makes it easier for our class functions to share access to that particular variable without having to pass data around to each other which can A)Get messy when multiple threads and/or processes become involved and B)Get messy even in a single-threaded environment. To me, it does not matter one whit how elegant your solution is or how brilliant it makes you look if the person who maintains it when you no longer do has no clue how to work with it. Making yourself look more clever than the next guy is the kind of A-type grandstanding that has no place in an environment as collaborative and diverse as the development and engineering field. Simple, readable, easy to follow solutions are superior to those that might be a pithing faster or more efficient, but are totally untenable.</p><p>Now that our class is defined and has some initialization behavior, lets add our first function:</p><pre>def connect_and_parse(self, website):<br>        r = self.s.get(website)<br>        soup = BeautifulSoup(r.text, &quot;html.parser&quot;)<br>        proxy_table = soup.find(&#39;tbody&#39;)<br>        proxy_list = proxy_table.find_all(&#39;tr&#39;)<br>        elites = [tr for tr in proxy_list if &#39;elite&#39; in tr.text]<br>        tds = []<br>        for tr in elites:<br>            tds.append([td.text for td in tr])   <br>        return tds</pre><p>With the principle of “one function per function” in mind, connect_and_parse takes a web address as an argument (in this case ‘<a href="https://free-proxy-list.net/">https://free-proxy-list.net/</a>&#39;), connects via our requests session, and pulls down the HTML located at that address. For the sake of simplicity, I tend to use the “html.parser” parser included with BeautifulSoup, rather than rely on another dependency like lxml.</p><p><strong><em>Pro Tip: Remember, BeautifulSoup is markup parser. It works with HTML strings and nothing else. I see many folks get into a groove and pass the response object (in this case r) to BeautifulSoup and hit an exception. Our response is an object that contains properties that BeautifulSoup works with (like .text and .content), but is useless to us by itself. Just remember to pass r.text or r.content to avoid that annoying holdup.</em></strong></p><p>Using the browser tools of your preference (I use Chrome Dev Tools), examine the site’s content to find that the table containing our proxy information is the first instance of the ‘tbody’ HTML class. Calling the soup.find() function finds the first occurrence of this class and assigns it to proxy_table.</p><p>To create a list of the rows of this table, assign proxy_table.find_all(‘tr’) to the proxy_list variable.</p><p><strong><em>Pro Tip: Any item returned by the soup.find() function will be a BeautifulSoup object that you can further parse with find(), find_all(), next_sibling(), etc. However, soup.find_all() returns a list filled with BeautifulSoup objects. Even if this list contains only one item, you will still need to access it specifically to parse it further. In this example, if we tried to call something like</em></strong> <strong><em>proxy_list.find(‘some-other-element’), we will get an exception because the proxy_list variable is a list containing soup objects and therefore, only has the properties of a list. To further parse an element of find_all(), iterate through the list or access an item by its index.</em></strong></p><p>Our next line creates a list of “elite” status proxies. These types of proxies are totally anonymous and very difficult for filters to pin down and filter on the fly. They are also HTTPS so your traffic will be both anonymous and encrypted with SSL, both things we want our crawler to have. This process is done using a list comprehension, which you can get a crash course in <a href="https://www.ccstechme.com/coding-duck-blog/python-programming-in-5-minutes-list-comprehensions">here</a>.</p><p>Next line is an example of readability vs cleverness. We are creating a list of cells in each row stored in the elites list. This can be done with a list comprehension, however it would be a nested list comprehension that would be a bit of a brain teaser, difficult to read, and would need to be read multiple times just to understand the logic flow going on. This would make debugging any issues it could later develop take much longer than needed. So instead, we use a traditional for loop to access the rows in elites, then leverage a simpler, more readable list comprehension to process the cells of each row into our desired information on each proxy we are collecting.</p><p>Cool! So we’ve got our proxy information in a neat little list of lists. Now to process it and toss it into a CSV file we can reference later:</p><pre>def clean_and_sort(self, data_set):<br>        columns = [’Address’, &#39;Port’, &#39;Country_Code’, &#39;Country’, &#39;Proxy_Type’]<br>        mapped_lists = {}<br>        i=0<br>        while i &lt; 5:<br>            mapped_lists.update({columns[i]: [tag[i] for tag in data_set]})<br>            i+=1<br>        df = pd.DataFrame(mapped_lists, columns=columns)<br>        df.to_csv(’proxy_ips.csv’)</pre><p>This next function takes the output of our previous function (our collection or lists or proxy details), cleans it, sorts it, and outputs it to a csv. The columns variable is a list that contains the column names we will be using for our csv file. We then create an empty dictionary, and use a while loop to only process the first 5 items in each list we created in our connect_and_parse function into a dictionary mapping data to csv columns. This is necessary because, if we re-examine <a href="http://free-proxy-list.net/">free-proxy-list.net</a>, the table contains 8 columns. We only care about the first 5. Now, we could have written our scraper to only retrieve the first 5. However, connect_and_parse is not responsible for cleaning and sorting our data, only grabbing it and returning it as unceremonious blob. Is it the most efficient solution? Perhaps, perhaps not. But when it comes to debugging a problem with sorting the data correctly once retrieved, you will be glad you went about it this way. Because no matter what you change (and subsequently break) in the course of debugging the clean_and_sort function, connect_and_parse will still work just fine. Can you say the same if both responsibilities were given to connect_and_parse by itself?</p><p>From there, we call on pandas to create a dataframe from the data in our mapped_lists dictionaries, using our columns list as the column headers, and exporting the dataframe to our new file, proxy_ips.csv. Once open, this file will look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*iPpgY72ApR0H1SJDW_lJvw.jpeg" /><figcaption>Proxy List for Web Scraping</figcaption></figure><p>Okay! Our first line of defense is in place! But before we can start going after the data we want, we have one more common filtering vector we need to confuse: User-Agents. Tune in for part three to continue your studies in filter-proofing your web crawler!</p><p>#python #programming #learntocode #webscraping #softwaredevelopment #datascience #softwareengineering #webprogramming</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=49ffe437f458" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/bye-bye-403-building-a-filter-resistant-web-crawler-part-ii-building-a-proxy-list-49ffe437f458">Bye Bye 403: Building a Filter Resistant Web Crawler Part II — Building A Proxy List</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Py in 5: Decorators]]></title>
            <link>https://medium.datadriveninvestor.com/py-in-5-decorators-f787f0ea1a74?source=rss-cd58749cce23------2</link>
            <guid isPermaLink="false">https://medium.com/p/f787f0ea1a74</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[learning-to-code]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Jake Cahill]]></dc:creator>
            <pubDate>Thu, 15 Nov 2018 00:50:20 GMT</pubDate>
            <atom:updated>2018-12-14T15:07:28.689Z</atom:updated>
            <content:encoded><![CDATA[<p>What is a decorator in Python, and why should I care?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*abgYD0hiKo0zzq1o3QO9NA.jpeg" /><figcaption>Python Decorators Explained in 5 Minutes</figcaption></figure><blockquote><strong>Originally published on the Coding Duck blog: </strong><a href="https://www.ccstechme.com/coding-duck-blog"><strong>ccstechme.com/coding-duck-blog</strong></a></blockquote><p>Hello everybody and welcome to another Py in 5 discussion. The article series where I take a Python programming concept and try to explain it in human terms in a 5-minute read or less. Our topic today: Decorators!</p><p>As is our customer, let’s get the “official” definition of decorators:</p><p>pythontips.com:</p><p><strong><em>Functions which modify the functionality of other functions</em></strong></p><p>python.org:</p><p><strong><em>A function returning another function applied as a function transformation using the @wrapper syntax</em></strong></p><p>Okay, so what I’m hearing is….functions. Functions that take a function to return a function so that your app functions functionally. And the programming paradigm usually used? Object-Oriented…obviously.</p><p>Even though these definitions aren’t at all talking in circles or using their own definition to define themselves, I’m going to try to clarify things anyway.</p><p>In a really abstracted sense, think of decorators as a kind of encapsulation, but on the function level rather than the class level. Just like classes in Python allow for encapsulation of their properties, decorators allow this kind of OOP within functions. Kind of like when you import a class into your python file, let’s say the “time” module. And let’s say you wanted to tell your program to wait 10 seconds before continuing. For this, you would write: time.sleep(10). What you are basically doing here is leveraging encapsulation. The &quot;sleep&quot; function is encapsulated (or enclosed) in the time class.</p><p>Think of it as if you had a friend name Sleep you wanted to play with as a kid. You would go to his house, ring the bell, and his dad Time would answer the door. “Hi! can Sleep come out and play?” you would ask Time.</p><p>“Sure!” he would respond, “Let me go get him for you.” Then, Sleep would come out and you would go have a jolly old time for yourselves. A decorator provides a similar functionality, but rather than encapsulating functions, it encapsulates context. Any function that is called with a decorator, gets access to the context (stuff that happens before and after the function runs that affect the function as it runs) that is unique to that decorator. Kind of like your parents (the class) would have a particular way you needed to behave in a general sense. But when your uncle (the decorator) would take you to do stuff, your behavior would be different than it was at home. What was expected of you, and therefore how you acted, was different while you were under the temporary care of a different relative than your parents.</p><p>For example, let’s take a look at a few decorators that change the behavior of an HTML form. The form is written as follows and rendered in the Flask web framework:</p><pre>@app.route(&#39;/&#39;)<br>def styling():<br>    return &#39;&#39;&#39;<br>            &lt;form action=&#39;&#39; method=&#39;post&#39;&gt;<br>                &lt;dl&gt;<br>                  &lt;dt&gt;Username:<br>                  &lt;dd&gt;&lt;input type=text name=username&gt;<br>                  &lt;dt&gt;Password:<br>                  &lt;dd&gt;&lt;input type=text name=password&gt;<br>                  &lt;dd&gt;&lt;input type=submit value=Login&gt;<br>                &lt;/dl&gt;<br>            &lt;/form&gt;&#39;&#39;&#39;</pre><p>and renders like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/0*HN84mG1M_Y8FQzD1.png" /></figure><p>Now, let’s add the following decorators to the mix:</p><pre>def style_with_flex_end(func):<br>    def func_wrapper():<br>        return &#39;&#39;&#39;&lt;div style=&quot;display: flex; flex-direction: row; justify-content: flex-end&quot;&gt;{0}&lt;/div&gt;&#39;&#39;&#39;.format(func())<br>    return func_wrapper</pre><pre>def style_with_flex_center(func):<br>    def func_wrapper():<br>        return &#39;&#39;&#39;&lt;div style=&quot;display: flex; flex-direction: row; justify-content: center&quot;&gt;{0}&lt;/div&gt;&#39;&#39;&#39;.format(func())<br>    return func_wrapper</pre><p>Both decorators do similar things. The only difference between them is the justify-content style tag. However, by simply giving the aforementioned HTML form the <em>context</em> these decorators apply, we get a different result. Let&#39;s attach the style_with_flex_end decorator to this Flask route simply by calling it with an &quot;@&quot; symbol and no parentheses above the definition of the &quot;styling&quot; function.</p><pre>@app.route(&#39;/&#39;)<br>@style_with_flex_end<br>def styling():<br>    return &#39;&#39;&#39;<br>            &lt;form action=&#39;&#39; method=&#39;post&#39;&gt;<br>                &lt;dl&gt;<br>                  &lt;dt&gt;Username:<br>                  &lt;dd&gt;&lt;input type=text name=username&gt;<br>                  &lt;dt&gt;Password:<br>                  &lt;dd&gt;&lt;input type=text name=password&gt;<br>                  &lt;dd&gt;&lt;input type=submit value=Login&gt;<br>                &lt;/dl&gt;<br>            &lt;/form&gt;&#39;&#39;&#39;</pre><p>Adding the decorator renders that same form like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/0*yq-Gf900ceF7lrc1.png" /></figure><p>As you can see, the justify-content: flex-end; context provided by the decorator changed where the form appeared on the page without the need to modify the form code itself. The same thing can be done with our other decorator:</p><pre>@app.route(&#39;/&#39;)<br>@style_with_flex_center<br>def styling():<br>    return &#39;&#39;&#39;<br>            &lt;form action=&#39;&#39; method=&#39;post&#39;&gt;<br>                &lt;dl&gt;<br>                  &lt;dt&gt;Username:<br>                  &lt;dd&gt;&lt;input type=text name=username&gt;<br>                  &lt;dt&gt;Password:<br>                  &lt;dd&gt;&lt;input type=text name=password&gt;<br>                  &lt;dd&gt;&lt;input type=submit value=Login&gt;<br>                &lt;/dl&gt;<br>            &lt;/form&gt;&#39;&#39;&#39;</pre><p>which then renders as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/0*OdaJVeJ19fjzDKbr.png" /></figure><p>So, in summary, a decorator adds context to a function. It, basically, gives your function “permission” to use the added window dressing that the decorator provides. Our example shows that if we ask the @style_with...decorators if the justify-content style they implement can &quot;come out to play&quot;, they will give our HTML a different set of expectations then when they are &quot;at home with mom and dad&quot;, therefore allowing them to behave differently than they normally would.</p><p>I hope this has cleared some things up for those of you who were unsure of what a decorator was or what it did. Please leave a comment below if you have any questions or need clarification! I am always willing to help others understand because, as always, the world needs you, your ideas, and your unique approach to this wonderful field so that we can all build the cool, useful, awesome tech of tomorrow!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f787f0ea1a74" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/py-in-5-decorators-f787f0ea1a74">Py in 5: Decorators</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Jinja2 Template: Web Templates for Python Developers]]></title>
            <link>https://medium.datadriveninvestor.com/the-jinja-template-because-im-a-python-developer-c5f702a109ae?source=rss-cd58749cce23------2</link>
            <guid isPermaLink="false">https://medium.com/p/c5f702a109ae</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[learning-to-code]]></category>
            <category><![CDATA[flask]]></category>
            <dc:creator><![CDATA[Jake Cahill]]></dc:creator>
            <pubDate>Wed, 31 Oct 2018 15:15:12 GMT</pubDate>
            <atom:updated>2018-12-14T15:09:25.685Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Jinja2 Template: Web Templates For Python Developers</h3><blockquote><strong>Originally published on the Coding Duck blog: </strong><a href="https://www.ccstechme.com/coding-duck-blog"><strong>ccstechme.com/coding-duck-blog</strong></a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/355/1*OgttR9d4EsJUvxp35trxCg.png" /><figcaption>Jinja2 Web Template Engine Logo</figcaption></figure><h4>Of all of the web templating systems in the world, Jinja2 is definitely one of them.</h4><p>So I was having a conversation with a friend/colleague of mine not too long ago about a project he was working on. He runs a two-person operation that primarily does digital media and marketing. On occasion, he will build a website for somebody to go along with the digital presence he hopes to create for them. He makes no pretenses about his web development skills. “I go to ThemeForest, download a theme, and change the colors, pictures, and text.” I really admire that kind of honesty. He has a marketing and business background and knows that most people would rather deal with as few vendors and contractors as possible when starting a business, so he offers Wordpress re-skinning to make himself a one-stop-shop. I usually hear from him when his clients have matured a bit and are looking for something a bit more flexible and dynamic than he can provide. While I understand that a true PHP master can make WordPress do absolutely ludicrous things, PHP just never appealed to me, so I will build with Flask when asked.</p><p>One day, he asked me how web developers built websites by hand with Flask, Django, Angular, React, Node, etc. “It’s not all that different from WordPress actually,” I responded, “we just tell the code where to put things and how to make them look, rather than a drag and drop interface.”</p><p>At this, he began to chuckle and shake his head. His reaction is based mostly on what he knows of me. I am the quintessential developer/engineer/code jockey type. The coding aspect to me is the only reason I do it; because I love to solve the puzzle. Sure, I need to pay the bills and, though I <em>technically</em> run my own business, I am <strong>far</strong> from a businessman. To me, it’s the machines. I love to talk to them and listen to them talk back. I have come across many machines that are quite “cheeky”, as I refer to them, and rather difficult to work with. This gives them a personality in my eyes; the result of years of neglect or bad configurations to them, are much like negative past life experiences to us, they are shaped by it, molded by it…</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FAqPRIe2gIxrP2%2Ftwitter%2Fiframe&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2Fangry-batman-AqPRIe2gIxrP2&amp;image=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FAqPRIe2gIxrP2%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="319" frameborder="0" scrolling="no"><a href="https://medium.com/media/ab84944634935903e8cac79a3ea2fdf4/href">https://medium.com/media/ab84944634935903e8cac79a3ea2fdf4/href</a></iframe><p>He, on the other hand, is trying to get along and put food on the table, plus as much extra as he can. So after a moment, he replies, “Sure, for people like you. But even when I do have to Google something on WordPress and put in the code, I know that the top of the page goes into wp-header.php, the bottom goes into wp-footer.php and so on. What you do isn’t that easy.”</p><p><strong><em>Oh no????</em></strong></p><p>Many people are afraid of programming languages and get into absolute panic mode the moment that they are required to read one. This could be akin to somebody who sees the title of a book on a shelf but in another language. They immediately move on because reading a language you do not understand is intimidating. This leads to misconceptions. Misconceptions that perhaps, some of you have had about how certain front-ends go together. Maybe you think that JS is best with JS and web apps should really only use Node, or React, or the Angular Typescript compiler. But to believe such would be to see <em>Les Miserable</em> on a bookshelf and assume you couldn’t read it because you speak English. Yet if you picked it up and opened it, you’d see the title is French because the author is French, the story was originally written in French, and it takes place in France, but the book itself is translated into English. Now, whether or not Victor Hugo is understandable even <em>if</em> his prose is in your language is another post in itself. But the point is, laying out a Flask app using Jinja is not all that different from laying our a Wordpress app. The trick: Templates. The templating engine in question: Jinja. Let’s dispel some myths:</p><p>A typical base template looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/53dbd7eff9d2957ae0f21c09d9682fc4/href">https://medium.com/media/53dbd7eff9d2957ae0f21c09d9682fc4/href</a></iframe><p>So this template lays out a modular HTML document that imports and uses other HTML documents to fill in the spaces of the content. Each of the three HTML documents currently contains an “h1” tag that identifies what part of the document it is in. In its current state, it appears as follows once rendered:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*E21znKtcwBFWAeWKfuMlwg.png" /></figure><p>Now that we have this base template laid out, let&#39;s modify the imported files. header.html now looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/991c515a2a9523522afedc4a693529bd/href">https://medium.com/media/991c515a2a9523522afedc4a693529bd/href</a></iframe><p>the content file:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/62f6f298b27fd66c9b8a1e0a43535e9a/href">https://medium.com/media/62f6f298b27fd66c9b8a1e0a43535e9a/href</a></iframe><p>aaaaaaand the footer:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/953db7e4314f98bbcb90b477c2dd8931/href">https://medium.com/media/953db7e4314f98bbcb90b477c2dd8931/href</a></iframe><p>Once the Jinja template is rendered via Flask, it looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Uc7dZ9abGV8WczICif_9w.png" /></figure><p>So why should we care about this? We can put all of this on one page and not worry about imports and crap. Well, we care for the sake of modularity. Using this kind of templating engine allows us to assemble our web app in pieces that are independent of one another. Let’s say something funky happens on your web server and the “header.html” file does not render. That would be REALLY annoying. But, the web page would still display, just minus the header. This would annoy your users, but they will not get the 404 that will potentially keep them from coming back the next time.</p><p>So, how do we remember the syntax of Jinja?</p><p>The easiest way is as follows: Each Jinja specific tag requires two symbols enclosing the conditional (if, for, while) or variables. The opening and closing symbol is <strong><em>always</em></strong> a bracket “{“ and “}”. If you are asking Jinja to <em>do</em> something (like a conditional, loop, include, or block statement), the statement is framed by percent/modulo signs “%”. If you are asking Jinja to <em>say</em> something (display a variable or an iterator in a loop), the item is wrapped in another set of brackets: “{“, “}”. That general rule of thumb applies to the vast majority of the plumbing of Flask. If you want to learn more specifics, check out the documentation which can be found <a href="http://flask.pocoo.org/docs/1.0/">here</a>.</p><p>Leave a comment below if you have any questions or requests for future articles and I hope this has cleared up a bit of Jinja2 and Python web templates for you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5f702a109ae" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/the-jinja-template-because-im-a-python-developer-c5f702a109ae">The Jinja2 Template: Web Templates for Python Developers</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Py in 5: List Comprehensions]]></title>
            <link>https://medium.datadriveninvestor.com/py-in-5-list-comprehensions-2c87de2c80cf?source=rss-cd58749cce23------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c87de2c80cf</guid>
            <category><![CDATA[code]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Jake Cahill]]></dc:creator>
            <pubDate>Fri, 26 Oct 2018 21:49:49 GMT</pubDate>
            <atom:updated>2018-12-14T15:11:52.864Z</atom:updated>
            <content:encoded><![CDATA[<h4>List comprehensions! The way to handle lists and the things that go in them, Pythonically.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*4B3Ty9KN3klLOQV0KGz4NA.jpeg" /><figcaption>Python List Comprehensions in 5 Minutes</figcaption></figure><blockquote><strong>Originally published on the Coding Duck blog: </strong><a href="https://www.ccstechme.com/coding-duck-blog"><strong>ccstechme.com/coding-duck-blog</strong></a></blockquote><p>Today’s topic? List Comprehensions! The Pythonic way of doing Pythonic things with lists Pythonically.</p><p><em>Okay Jake yeah, Pythonic and such. We get it</em></p><p>But <em>do</em> you? I mean do you really? For instance, did you know that there are not only list comprehensions, but also dict <em>AND </em>set comprehensions?</p><p>You know what? Yeah, you probably did. Per the cheeky overuse of “pythonic” above, most who are familiar with the language understand that it values one-liners very highly. A list/dict/set comprehension is, what I feel, the most common of these one-liners. But what is it that they really do? While this post will cover list comprehensions specifically, these same concepts can be applied to other types of comprehensions with some differences. Leave a comment if you’d like a post for each of the other types of comprehensions as well. But seeing as I only have 5 minutes, allow me to answer the question about list comps for you.</p><p><strong><em>…was I supposed to type something here?…</em></strong></p><p>To be honest, a code sample is not entirely necessary to explain comprehensions. Really because there is not that much mystery to them. Basically, a comprehension is a wrapper for flow control and conditionals that assigns to a variable.</p><p>Now for the sake of posterity, I will provide a code sample that shows you these different methods side by side:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/67a08283d8320ebf407f938602f7c8b6/href">https://medium.com/media/67a08283d8320ebf407f938602f7c8b6/href</a></iframe><p>See the difference? Exactly the same thing is accomplished in both cases: we iterate over a string and append each letter to a list. The difference is we do it in fewer lines of code.</p><p>See the way this works is to, essentially, put the logic of a loop between brackets “[]”, which tells the interpreter “hey, I want to make a list out of whatever is in here”. But the syntax is a little weird right? Not if you break it down.</p><p>Essentially, a for loop’s syntax says the following “for each item (i) in this chunk of other items (this_string), perform this action (a_list.append(i)).”</p><p>A comprehension says this exactly, but more like this: “I want you to perform this action (i) on each of these items (i) in this set of other items (this_string) and store it in a list.”</p><p>Where most people get thrown off or don’t really think about what is going on under the hood, is the ‘[i for i]’ bit. Sometimes I even just type something similar out of reflex. But by saying ‘[i for i]’ you are <em>actually</em> saying “keep i as it is each time you see it”</p><p>Let me give you some clarification:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c7f85e65f2d389cf9d1d5a096abd811a/href">https://medium.com/media/c7f85e65f2d389cf9d1d5a096abd811a/href</a></iframe><p>In this case, b_list prints out exactly what it did in the first example, but we added the strip() function onto the first ‘i’. Had we typed in [i for i in this_list], those newline characters (‘\n’) would have remained in the new list. Why? Because [i for i] means “keep i as it is”, whereas [i.strip() for i] means “remove the newline characters from ‘i’ as you go would you?” If we had specified [i.upper() for i], we would have told python “capitalize i as you go would you?”, etc. Basically, the first time you throw down that iterator, you’re telling Python what to do with it as it builds the list. You could even throw (i) into a function in that place as well:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b46bd000f11aabd3c2bd1497cf787ff6/href">https://medium.com/media/b46bd000f11aabd3c2bd1497cf787ff6/href</a></iframe><p>This list would now contain the sentence <strong>“For that which I say to you sir: t\n!!!”</strong><br>all the way down to <strong>“For that which I say to you sir: g\n!!!”</strong>, inserting each character in this_list into the end of the sentence.</p><p><em>Okay, so we can iterate over things in one line and make a list. What else can list comprehensions do?</em> Pretty much anything you can do in a standard loop actually. We can add in conditionals like “if” statements, use multiple functions, even perform calculations on each item as it goes into the list. I’ll be posting a bit more of a deep dive on list comprehensions and their capabilities, limitations, and when you should start hearing alarm bells that maybe a list comprehension just isn’t the right choice for a given situation.</p><p>Until then, I hope this has cleared up what a list comprehension is and has given you the shallow-end version of what they are useful for. Give them a try. Experiment with them a bit and see if you can replace some for loops in your personal projects. As always, keep coding, keep building, and try not to build another social media site. I mean seriously. I know that they can have an employee count of 4 people and no money coming in whatsoever and <strong><em>still </em></strong>be bought by Facebook for over a billion dollars. But, I don’t even like my own selfies, let alone seeing <strong><em>nothing but</em></strong> selfies of people I don’t like or haven’t spoken to in 15 years. It’s not that I don’t care that it&#39;s your youngest dog’s third birthday, its just…no actually it is that. I haven’t spoken to you since we were 6 and I have never met your dog. I really do not care it&#39;s his third birthday.</p><p>So if you have been shouldered with the title “Junior”, “Newbie”, “Intern”, “Coffee-Getter”, “Probie” (we’ve all had that PM that still likes NCIS and calls people “Probie”. I don’t get it either so you’re not alone), or any other name that makes you feel small, stupid, or inadequate, toss that title off and approach this career you’ve pursued as if you can learn anything. My dad always told myself and my brothers this, and I will pass it on to you: <strong><em>Be humble about what you know, and arrogant about what you can learn</em></strong>. Nothing is beyond you so keep building, keep learning and, as always, the world needs you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c87de2c80cf" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/py-in-5-list-comprehensions-2c87de2c80cf">Py in 5: List Comprehensions</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Py in 5: Lambda Expressions]]></title>
            <link>https://medium.datadriveninvestor.com/py-in-5-lambda-expressions-6a6e98266f04?source=rss-cd58749cce23------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a6e98266f04</guid>
            <category><![CDATA[learning-to-code]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[lambda-function]]></category>
            <dc:creator><![CDATA[Jake Cahill]]></dc:creator>
            <pubDate>Thu, 25 Oct 2018 16:58:34 GMT</pubDate>
            <atom:updated>2018-12-14T15:12:31.421Z</atom:updated>
            <content:encoded><![CDATA[<h4>The series where I attempt to take a Python programming concept and explain it, in mere mortal terms, in a 5-minute read or less!</h4><blockquote><strong>Originally published on the Coding Duck blog: </strong><a href="https://www.ccstechme.com/coding-duck-blog"><strong>ccstechme.com/coding-duck-blog</strong></a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*Q4DVDvzmVrhnFJ4NzV-Awg.jpeg" /></figure><p>Today’s topic? Lambda functions! To start, let’s review the official definition of what the Python core team says a Lambda function is(otherwise called a Lambda expression). Per the <a href="https://docs.python.org/3/reference/expressions.html">docs</a>:</p><p><em>Lambda expressions (sometimes called lambda forms) have the same syntactic position as expressions. They are a shorthand to create anonymous functions; the expression lambda parameters: expression yields a function object</em></p><p>Super clear right? No? Your not alone in thinking so. For many programmers Lambdas are features they know how to use, but don’t really understand what it is they are actually doing. The truth of the matter is that a Lambda function works like a regular function, it’s just shorter and more temporary. Like absurdly temporary. The phrases “single use” and “throw away” come to mind immediately. Let’s take a look at an example. Below is a traditional function that takes a number and cubes it (to the power of 3 or num*num*num)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/778c2ee1804018d2a4edc1873cfc3b76/href">https://medium.com/media/778c2ee1804018d2a4edc1873cfc3b76/href</a></iframe><p>Neat. A function like this is not all that useful on it’s own and really only finds its stride when it is used repeatedly on a dataset like a list full of numbers that we wanted to cube. Using the function above we can go about this a couple of ways.</p><p>The cumbersome but <em>highly </em>readable way:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/51da2b5c03b24209626e42b0cd0314e3/href">https://medium.com/media/51da2b5c03b24209626e42b0cd0314e3/href</a></iframe><p>aaaaand the,<em> very</em> pythonic, one-liner way using a list comprehension:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f184bdfe5e26304baa2124830d4faffd/href">https://medium.com/media/f184bdfe5e26304baa2124830d4faffd/href</a></iframe><p>So now we have a list of numbers and a list of cubed numbers. What does that have to do with lambdas? Can they do fun list things too? Yes, they can, as can be seen here:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6201ee151423c20d82d4867c4bdc3909/href">https://medium.com/media/6201ee151423c20d82d4867c4bdc3909/href</a></iframe><p>But the use of list comprehensions does tend to be faster as the datasets get larger so they are preferred for this exact application. So what good are lambdas and what do they actually <em>do???</em></p><p>Well, the best strength (and weakness) of lambda functions comes in situations where you only need to perform a specific task in a couple places in your code, while a normal function is best used when its functionality will be required over and over again within the same code file, or in different files.</p><p>Think of it this way: Let’s say you’r a carpenter building a house, but like a really <em>weird</em> house where nails just randomly come out of the wood sometimes as if they had never been nailed in. On <em>most</em> occasions, you would walk around the job site, looking for nails that came out. When you saw one, you’d take your trusty hammer from your belt, right where it always is, and hammer the nail back in 5 strikes of the hammer. That is using a normal function.</p><p>A lambda would be if you were walking around the house and saw 50 nails right in a row that had come out. You <em>could</em> hammer them all in with your 5 strike hammer. But there is also a hammer in the trailer that will do each nail in 2 strikes. Problem is, the trailer is 100 yards away and seeing as this is a super special hammer, it must <em>always be kept in the trailer when not in use</em>.</p><p>That make more sense? An ordinary function, when initialized, gets put into a spot where the Python interpreter can keep it close at hand. A lambda however, gets created the <em>moment</em> it is needed and removed the <em>moment</em> it is done. So, while it has the advantage of being able to do 10 lines of code worth of stuff in 20 characters or so, to have to “go get it from the trailer” for any reason but a special use case where it will be necessary to do the same task quickly then be done for the day doesn’t make much sense. In the long run, the lines of code you save are not nearly enough to justify the time it will take to “walk to the trailer” <em>every time</em> you see a loose nail. For that, just use your trusty old “belt hammer” in a regular function.</p><p>So I hope this clarifies lambda functions for you! If you have any questions, please leave a comment so that I can provide a bit more clarity on things I may have glazed over (I do only have 5 minutes as it is). As always, get out there and build, build, build! Never think that you cannot learn how to take a text editor and a bit of time and change the world. With all of the crap out there making the world miserable, and all of the technology built simply to help enough people waste enough time to make the developers rich, the world needs you and anybody else who desires to better themselves and the world by typing these funny phrases into a command line.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a6e98266f04" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/py-in-5-lambda-expressions-6a6e98266f04">Py in 5: Lambda Expressions</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>