<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Cristian Medina on Medium]]></title>
        <description><![CDATA[Stories by Cristian Medina on Medium]]></description>
        <link>https://medium.com/@tryexceptpass?source=rss-138c0eb26be5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*CT5CKhmNylXGTADM0PorTQ.jpeg</url>
            <title>Stories by Cristian Medina on Medium</title>
            <link>https://medium.com/@tryexceptpass?source=rss-138c0eb26be5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 23:01:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tryexceptpass/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Using GitHub as a Flat Data Store and AWS Lambda to Update it]]></title>
            <link>https://medium.com/@tryexceptpass/using-github-as-a-flat-data-store-and-aws-lambda-to-update-it-8cfa2d1bd524?source=rss-138c0eb26be5------2</link>
            <guid isPermaLink="false">https://medium.com/p/8cfa2d1bd524</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Cristian Medina]]></dc:creator>
            <pubDate>Sun, 01 Apr 2018 03:34:17 GMT</pubDate>
            <atom:updated>2018-04-01T03:34:40.317Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JggJLMOPdaUrHEl0NM35HQ.jpeg" /><figcaption>Image from <a href="https://unsplash.com/photos/o6GEPQXnqMY">Fabien Barral</a></figcaption></figure><p>This is re-published from my main <a href="http://tryexceptpass.org/article/using-github-as-flat-data-store/">tryexceptpass blog</a>.</p><p>I spend most of my day, every day, knee deep in code. Optimizing, building, fixing and thinking through workflows can be taxing. This means that the last thing I want to do when I come home is deal with <em>more</em> programming. But I also like learning new things and communicating my experiences so they can help others. I do that through the posts in this website.</p><p>Maintaining a web presence without dealing with code means you get to use as many off-the-shelf components as possible. You consider things like WordPress or static site generators that let you concentrate on content, while handling the user interface for you. Write in markdown, build the website, rinse repeat with updates. It’s all very easy, until you need a little more interactivity, like a comments section, or a newsletter signup.</p><p>There are lots of services out there that help put together those functions. Some paid and some “free”, but selecting them is not so easy. With free solutions, the choice becomes to either host the system yourself — which you wanted to avoid — or use a popular service that does it for you while including advertisements (like disqus).</p><p>My personal opinion is that if anyone advertises in my site, that will be me — or at least through a mechanism where I have final say — otherwise there’s no control over what users see. I don’t want a system that collects all kinds of data from my users. I want one that doesn’t even need user accounts. I’d hate to see some random ad for a hotel in Ibiza while I’m at a tech blog, simply because I googled Mediterranean vacations last night after exhaustion settled in.</p><h4>A “low-maintenance” solution</h4><p>Assuming we want to build a system that requires some data backend like a comments section, we’ll need a place to store records and a way to retrieve them. Databases can’t be part of the solution because we don’t want to manage them and we really don’t need anything fancy, not even indexing. All I’m looking for is a straight-up list of records with 3 or 4 fields per record. A simple file will do, but we do want to track history and have some sort of backup mechanism. We can get the last part from a version control system like git.</p><p>These days there are plenty of ways to store structured data. The most universal one with built-in libraries across almost all languages is JSON. Others like YAML — I actually started down this path — and TOML may simplify appending to a file, but the universality of JSON in the web makes it more relevant in this situation.</p><p>Storing comments in a JSON file isn’t new or different. What’s this proposed “low-maintenance” solution? Stick it in GitHub, use JavaScript to read it, and sprinkle some AWS Lambda to make changes to it. Yes, you still have to write a little code, but it’s not complicated and once you get it going, it needs little to no maintenance.</p><h4>Configuring GitHub</h4><p>First you have to choose whether to keep your information in a private or public repository. I went with a private repo because I have the account for it and wanted the option of hiding any base files from the peering eyes of the interwebs. However, the method described below uses GitHub Pages, which works for both.</p><p>GitHub Pages is a feature that enables a regular repository to host static files on the internet. The files are located in a specific branch, typically gh-pages. This is excellent for static generators because you track the config and content files in the master branch, then run a build that commits the generated HTML into the gh-pages branch. This keeps the auto-generated files separate from the actual content. A few minutes after committing, the updated website is available at the URL assigned by GitHub (unless you use DNS).</p><p>Getting this configured is straight forward: * Identify or create the branch where you’ll host the files (the default is gh-pages). * Access the GitHub repository settings and scroll down to the GitHub Pages section. * Select the branch in the Source section * Configure DNS in the Custom domain section, if needed.</p><p>GitHub is now ready to host your files. Test it out by committing an index.html page with some “hello world” text and load it on your browser with the URL that GitHub gave you.</p><h4>Reading the Comments</h4><p>For now, let’s assume we already have a comments.json file with the following structure added into the gh-pages branch:</p><pre>[<br>    {<br>        &quot;user&quot;: &quot;ABCDEFG&quot;,<br>        &quot;comment&quot;: &quot;12345&quot;<br>    }<br>]</pre><p>It’s on you to edit your site generator so it includes JavaScript code that performs an HTTP GET to the URL that’s hosting this file. There are hundreds of ways of doing that, most of which use 3rd party libraries (like jQuery). If you don’t want to complicate the dependency tree more, here’s some sample JavaScript that does it with the basic XmlHttpRequest object:</p><pre>divcomments = document.getElementById(&quot;#div_for_adding_comments&quot;);<br>xhr.open(&#39;GET&#39;, &quot;https://your_user.github.io/ur_repo/comments.json&quot;)<br>xhr.onreadystatechange = function(data, err) {<br>    if (xhr.readyState === XMLHttpRequest.DONE) {<br>        if (xhr.status &gt; 200) {<br>            // There are no comments<br>            divcomments.innerHTML = &#39;&lt;p&gt;Be the first to comment on this article.&lt;/p&gt;&#39;;<br>        }<br>        else {<br>            // Parse the comments into an object that we can iterate<br>            var data = JSON.parse(xhr.response);</pre><pre>            // Iterate through the comments and add them to the div<br>            for (var i=0; i &lt; data.length; i++) {<br>                divcomments.innerHTML = divcomments.innerHTML + <br>                &#39;&lt;blockquote&gt;&lt;p&gt;&#39; + data[i][&#39;comment&#39;] + <br>                &#39;&lt;/p&gt;&lt;/blockquote&gt;&#39;;<br>            }<br>        }<br>    }<br>}</pre><p>Notice that all I parse the response as JSON. This gives us a full JavaScript object for updating the HTML element already in the page.</p><p>However, our plan has a flaw. The comment field in our data structure is in plain text. This means that if someone uses special characters in their message, then it can break the entire section. Usually you need to escape the string before placing it into the field, but the most flexible solution is to <strong>base64 </strong>encode it. JavaScript has built-in functions for doing so: atob() and btoa().</p><h4>Adding comments with AWS Lambda</h4><p>I’ve been looking for a reason to use AWS Lambda for a while. It’s an Amazon service that provides a way of mapping the execution of a function to an event. Functions can be written in a number of languages, one of which is python <strong>3.6</strong>, and they execute inside docker containers. Events are internal triggers from various AWS systems, including the API Gateway service. This means that you’re essentially mapping a function to a URL provided to you by the service.</p><p>There are no service charges for the first 1 million hits on the API endpoint each month, so the Lambda is basically free for an average site. But I’ll add a word of caution: if you’re running a high traffic site and expecting a lot of submissions, it’s important to look through the pricing page and do the math. Cloud services can come with steep surprises.</p><p>To get things going, it’s easy enough to hop on the <a href="https://aws.amazon.com/lambda">Lambda</a> website and make a new function based on the default python hello world example. * Click Create Function. * Select Blueprints. * Search for hello-world-python3. * Give it a name and type in your code changes. * The handler function receives two parameters: event and context. The event is a dictionary with a body field that contains the HTTP request body. If you’re using an HTML form to submit the comments, this is where you’ll find the form-encoded data. * Parsing event[&#39;body&#39;] is easy enough using urllib to decode the values:</p><pre>from urllib.parse import unquote_plus<br>params = [param.split(&#39;=&#39;) for param in event[&#39;body&#39;].split(&#39;&amp;&#39;)]<br>params = {k: unquote_plus(v) for k, v in params}</pre><p>Lambda functions plug into Amazon’s CloudWatch system, so you’ll be able to view output from any print() statements by going into the CloudWatch service.</p><p>Now let’s call the GitHub REST API to get the current list of comments, but do it without using 3rd party libraries. This keeps the function minimal, otherwise you go through a different process to add those libraries to the container that runs the function. Lambda also charges for GB-hours of usage. The bigger the container, the higher the changes you’ll see a bill. This means we’re stuck with good ol’ urllib.urlopen and urllib.Request.</p><p>The documentation for the API that manipulates files in a GitHub repo is available <a href="https://developer.github.com/v3/repos/contents/">here</a>. You can use HTTP Basic Authentication with a <a href="https://github.com/blog/1509-personal-api-tokens">personal access token</a> obtained from GitHub user settings to login. It’s simple to add the token to the http headers of the request and point it to the correct branch and file:</p><pre>from urllib import urlopen, Request<br>from base64 import b64encode, b64decode</pre><pre>headers = {<br>    &#39;content-type&#39;:&#39;application/json&#39;,<br>    &#39;authorization&#39;: f&quot;Basic {str(b64encode(bytes(<br>        &#39;username:access_token&#39;, &#39;utf-8&#39;<br>    )), &#39;utf-8&#39;)}&quot;<br>}<br>resp = urlopen(Request(f&quot;https://api.github.com/repos/{github_user}/{github_repo}/contents/{filename}.json?ref=gh-pages&quot;))</pre><p>The API response contains the <strong>base64</strong> encoded file in a content field, so you’ll want to b64decode(resp[&#39;content&#39;], &#39;utf-8&#39;) to get your comments and then use json.loads() to stick it in a dictionary that’s easy to change. It also gives us the SHA hash of the latest commit in that branch, which is information required for the next step: modifying the file.</p><p>Committing a change is also simple, except we need to pass in metadata describing the modifications along with the new <strong>base64</strong> encoded file contents (in its entirety).</p><pre>data = {<br>    &#39;message&#39;: &#39;A commit message to figure out what happened&#39;,<br>    &#39;branch&#39;: &#39;gh-pages&#39;,<br>    &#39;sha&#39;: sha_value_from_previous_GET_response,<br>    &#39;content&#39;: str(b64encode(bytes(<br>        json.dumps(updated_comments), &#39;utf-8&#39;<br>    )), &#39;utf-8&#39;)<br>}</pre><p>This encode and decode stuff is a bit ugly. We serialize the updated comments back to JSON. But since we need to base64 encode them to pass them to GitHub, we call b64encode(). This function takes bytes not plain strings, so we have to change the serialized contents into bytes. On top of that, the output of b64encode() is also bytes, which means we need to turn them back into a utf-8 string before sticking them in the dictionary that we’ll send along with our HTTP request. Phew!</p><p>The HTTP request to GitHub is a PUT and the authentication is the same as before, but urllib takes data in byte strings (yes, this again!). The call now looks like this:</p><pre>urlopen(Request(<br>    f&quot;https://api.github.com/repos/{github_user}/{github_repo}/contents/{filename}.json&quot;,<br>    headers=headers,<br>    data=bytes(json.dumps(data), &#39;utf-8&#39;),<br>    method=&#39;PUT&#39;<br>))</pre><p>That’s it. Once the request completes, the gh-pages branch will update with a new commit that contains the message you just passed it, and the new file is available to the public. It’s not the fastest solution, but the user will see his comment added to the list and you’ll have a commit history of all the changes over time. You can also add email integration with GitHub so you’ll get a notification of the commit.</p><p>Triggering with API Gateway</p><p>As mentioned earlier, Lambda can trigger on a number of things within the AWS ecosystem, some of those are internal services, others are external. To make your function accessible over the public internet, you have to plug it into <a href="https://aws.amazon.com/api-gateway/">API Gateway</a>. Doing so can be a little confusing the first time, so here’s some info on that.</p><p>Once you create an API with the Gateway service, add the URL path and method that you want routing to the function. You can use /, I decided to do it on a specific resource like /comments. You do this by clicking the Actions drop-down and selecting Create resource, then type in the name and path you chose.</p><p>I do recommend also selecting the Enable API Gateway CORS checkbox, which will pre-configure a response to an OPTIONS request. If you decide to use pure JavaScript to call the API (not form submissions), you can specify the CORS headers that browsers look for in the pre-flight requests sent before the actual API requests. Perform the edit by clicking on Integration Response after selecting the OPTIONS method it created.</p><p>We have a gateway endpoint and the resource path, now we add the method that calls our Lambda function into the base resource: a POST. Do this by selecting the resource path, clicking on the Actions drop-down and choosing Create Method. Select the POST method and use Lambda Function for the Integration Type field, then chose the AWS region where the function exists and type the function name in the Lambda Function text box.</p><p>The API Gateway Dashboard provides the URL for this Lambda function. Use it as the value for the action field in the HTML form of your website.</p><p>The workflow is now ready. Here’s a short review: * Load a webpage with the new comments section. * JavaScript performs an HTTP GET to list existing comments from a publicly available file hosted in GitHub Pages. * Form submission for adding a new comment performs an HTTP POST to an AWS API Gateway URL that triggers a Lambda function. * The Lambda function uses the GitHub REST API to retrieve the latest list of comments, inserts the new comment and commits an updated list to GitHub.</p><h4>Final Notes</h4><p>A complete solution has to consider that one flat file for an entire website is going to be too large. Unless you’re like me and have little traffic. To break things up, I recommend identifying articles or pages where you want to add separate sections by some form of uuid, hash, or a simple numerical identifier that stored somewhere in the page. Static site generators usually provide functions that do this for you. Use this identifier as the filename that you commit in the gh-pages branch. This yields a list that’s easy to manage both for editing and reading.</p><p>It’s also possible to split it up by using a dictionary structure, instead of a list, in one large file. But it still doesn’t solve the problem of loading the entire file before getting to the section that applies to the current page the user is visiting.</p><p>When a consumer submits the HTML form, the API Gateway pipes the HTTP request to the Lambda function which then returns a response. This means the user now navigated away from the site that originated the request. To stay in the site or move somewhere else that has actual content, you can add a hidden redirect field to the HTML form and ask API Gateway to return that URL in an HTTP redirect back to the page that submitted the request.</p><p>To do this, you need to add a mapping into the Method Response section for the POST method in API Gateway, so that it returns an actual 302 with a Location header when the Lambda function answers with something like this:</p><pre>return {<br>    &quot;statusCode&quot;: 302,<br>    &quot;headers&quot;: {<br>        &quot;Location&quot;: params[&#39;redirect&#39;]<br>    }<br>}</pre><p>One last note if you don’t care about privacy, is that you can bypass Lambda and perform all the changes in JavaScript. Just keep in mind that any credentials used to log into GitHub will be visible in the JavaScript code. Depending on the application though, this may not be an issue.</p><p>While this workflow is a tad complicated, it’s not too hard to create and easy to maintain. I hope that it serves you in the future, if anything, it’s a good exercise on AWS Lambda workflows and GitHub REST APIs.</p><figure><a href="https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fpublish.twitter.com%2F&amp;ref_src=twsrc%5Etfw&amp;region=follow_link&amp;screen_name=tryexceptpass&amp;tw_p=followbutton"><img alt="" src="https://cdn-images-1.medium.com/max/181/1*_04PhhMqHB3qXQnOQNP5ag.png" /></a></figure><p>If you liked this article and want to keep up with what I’m working on, please recommend it, visit <a href="http://tryexceptpass.org/">tryexecptpass.org</a> for more topics and <a href="https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fpublish.twitter.com%2F&amp;ref_src=twsrc%5Etfw&amp;region=follow_link&amp;screen_name=tryexceptpass&amp;tw_p=followbutton">follow me on Twitter</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8cfa2d1bd524" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Controlling Python Async Creep]]></title>
            <link>https://medium.com/hackernoon/controlling-python-async-creep-ec0a0f4b79ba?source=rss-138c0eb26be5------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec0a0f4b79ba</guid>
            <category><![CDATA[python-async]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[multitasking]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[asyncio]]></category>
            <dc:creator><![CDATA[Cristian Medina]]></dc:creator>
            <pubDate>Sun, 30 Jul 2017 23:28:48 GMT</pubDate>
            <atom:updated>2018-04-01T03:35:56.880Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4lpO60MFa0QBfY60ZvF3tA.jpeg" /><figcaption>Photo Credit: <a href="https://unsplash.com/collections/597524/growth?photo=aO_jMXTduUE">Christian Jourdy</a> via <a href="http://unsplash.com/">Unsplash</a></figcaption></figure><p>Python added formal asynchronicity in the base language a while ago. It’s fun to play with <a href="https://docs.python.org/3/library/asyncio.html">asyncio</a> tasks and coroutines, the basic constructs that execute almost in parallel. But as you start to integrate more with a regular codebase, you may find that things can get tricky. Especially if you’re forced to interact with synchronous code.</p><p>The complication arises when invoking <em>awaitable</em> functions. Doing so requires an async defined code block or coroutine. A non-issue except that if your caller has to be async, then you can’t call <em>it</em> either unless its caller is async. Which then forces its caller into an async block as well, and so on. This is “async creep”.</p><p>It can escalate fast, finding its way into all corners of your code. Not a big deal if the codebase is asynchronous, but can impede development when mixing it up.</p><p>Following are the two main mechanisms I use to work around this. Both assume we’re building an application that benefits from asynchronous execution.</p><h4>Waiting for blocks of async code</h4><p>Whether building an asynchronous application or enhancing a linear one, it’s important to determine the sections that will gain the most from async execution. This is usually not hard to answer, but no one else can do it for you. The general guideline is to start with things that wait on I/O, like file or socket access, HTTP requests, etc.</p><p>Once you know which pieces to optimize, start identifying the ones that can run on top of each other. The more you can group together, the better. A great example is code that needs information from several REST APIs that don’t depend on each other. You can use aiohttp and make all the calls in parallel instead of waiting for each one to finish before getting the next one.</p><p>Now it’s a matter of loading up those blocks of code into the main event loop. There’s a few ways of doing that, I like putting them into async functions and using asyncio.ensure_future() to put them in the loop and loop.run_until_complete() to wait for completion:</p><pre>import asyncio<br>import aiohttp</pre><pre>async def fetch(url):<br>    response = await aiohttp.request(&#39;GET&#39;, url)<br>    return await response.text()</pre><pre>loop = asyncio.get_event_loop()<br>loop.run_until_complete(asyncio.gather(<br>    asyncio.ensure_future(fetch(&quot;http://www.google.com&quot;)),<br>    asyncio.ensure_future(fetch(&quot;http://www.github.com&quot;)),<br>    asyncio.ensure_future(fetch(&quot;http://www.reddit.com&quot;))<br>))</pre><p>This is a similar example to one I used in a previous article: <a href="https://hackernoon.com/threaded-asynchronous-magic-and-how-to-wield-it-bba9ed602c32">Threaded Asynchronous Magic and How to Wield it</a>.</p><p>asyncio.ensure_future() turns the functions into coroutines, asyncio.gather() groups them together, while loop.run_until_complete() blocks execution until all calls complete. The output of this is a list with the results from each call.</p><p>Following the points discussed so far will produce code that runs synchronous blocks. But some of those blocks will execute several asynchronous functions together.</p><h4>Use a thread</h4><p>Also discussed in my <a href="https://hackernoon.com/threaded-asynchronous-magic-and-how-to-wield-it-bba9ed602c32">earlier</a> article, it’s not hard to create a separate thread that operates as a worker. It runs its own event loop and you use thread-safe asyncio methods to give it work. The nice part is you can give it synchronous work with call_soon() or async work with run_coroutine_threadsafe().</p><pre>from threading import Thread</pre><pre>...</pre><pre>def start_background_loop(loop):<br>    asyncio.set_event_loop(loop)<br>    loop.run_forever()</pre><pre># Create a new loop<br>new_loop = asyncio.new_event_loop()</pre><pre># Assign the loop to another thread<br>t = Thread(target=start_background_loop, args=(new_loop,))<br>t.start()</pre><pre># Give it some async work<br>future = asyncio.run_coroutine_threadsafe(<br>    fetch(&quot;http://www.google.com&quot;), <br>    new_loop<br>)</pre><pre># Wait for the result<br>print(future.result())</pre><pre># Do it again but with a callback<br>asyncio.run_coroutine_threadsafe(<br>    fetch(&quot;http://www.github.com&quot;),<br>    new_loop<br>).add_done_callback(lambda future: print(future.result()))</pre><p>We get a Future back from run_coroutine_threadsafe which we can wait on using the result(timeout) method, or add a callback with add_done_callback(function). The callback function will receive the future as an argument.</p><h4>Supporting both async and sync calls in the same API methods</h4><p>Let’s look at something more complicated. What if you have a library or module where most functions can run in parallel, but you only want to do so if the caller is async?</p><p>We can take advantage of the threaded model here because the scheduler methods are synchronous. Meaning that your user doesn’t need to declare itself as async and have to deal with the async creep in their code. Asynchronous blocks remain contained to your module.</p><p>It also allows for api interfaces that can choose whether to use asyncio or not. In fact, we can even go one step further and auto-detect when to be async using some inspect magic.</p><p>Without threads you have no control over your event loop. Users could do their own async fiddling that interferes with how your methods execute. The thread will at least guarantee asynchronous execution inside an event loop that you operate. One that you start and stop when needed. This leads to more predictable, repeatable results.</p><p>Let’s look at an example that builds on the earlier ones. Here we make a wrapper method that calls the appropriate sync or async function based on its caller.</p><pre>import inspect<br>import requests</pre><pre>...</pre><pre>def is_async_caller():<br>    &quot;&quot;&quot;Figure out who&#39;s calling.&quot;&quot;&quot;</pre><pre>    # Get the calling frame<br>    caller = inspect.currentframe().f_back.f_back</pre><pre>    # Pull the function name from FrameInfo<br>    func_name = inspect.getframeinfo(caller)[2]</pre><pre>    # Get the function object<br>    f = caller.f_locals.get(<br>        func_name, <br>        caller.f_globals.get(func_name)<br>    )</pre><pre>    # If there&#39;s any indication that the function object is a <br>    # coroutine, return True. inspect.iscoroutinefunction() should<br>    # be all we need, the rest are here to illustrate.</pre><pre>    if any([inspect.iscoroutinefunction(f),<br>            inspect.isgeneratorfunction(f),<br>            inspect.iscoroutine(f), inspect.isawaitable(f),<br>            inspect.isasyncgenfunction(f) , inspect.isasyncgen(f)]):<br>        return True<br>    else:<br>        return False</pre><pre>def fetch(url):<br>    &quot;&quot;&quot;GET the URL, do it asynchronously if the caller is async&quot;&quot;&quot;<br><br>    # Figure out which function is calling us<br>    if is_async_caller():<br>        print(&quot;Calling ASYNC method&quot;)</pre><pre>        # Run the async version of this method and<br>        # print the result with a callback<br>        asyncio.run_coroutine_threadsafe(<br>            _async_fetch(url), <br>            new_loop<br>        ).add_done_callback(lambda f: print(f.result()))</pre><pre>    else:<br>        print(&quot;Calling BLOCKING method&quot;)</pre><pre>        # Run the synchronous version and print the result<br>        print(_sync_fetch(url))<br></pre><pre>def _sync_fetch(url):<br>    &quot;&quot;&quot;Blocking GET&quot;&quot;&quot;</pre><pre>    return requests.get(url).content<br></pre><pre>async def _async_fetch(url):<br>    &quot;&quot;&quot;Async GET&quot;&quot;&quot;</pre><pre>    resp = await aiohttp.request(&#39;GET&#39;, url)<br>    return await resp.text()<br></pre><pre>def call_sync_fetch():<br>    &quot;&quot;&quot;Blocking fetch call&quot;&quot;&quot;</pre><pre>    fetch(&quot;<a href="http://www.github.com/">http://www.github.com</a>&quot;)<br></pre><pre>async def call_async_fetch():<br>    &quot;&quot;&quot;Asynchronous fetch call (no different from sync call <br>       except this function is defined async)&quot;&quot;&quot;</pre><pre>    fetch(&quot;<a href="http://www.github.com/">http://www.github.com</a>&quot;)<br></pre><pre># Perform a blocking GET<br>call_sync_fetch()</pre><pre># Perform an async GET<br>loop = asyncio.get_event_loop()<br>loop.run_until_complete(call_async_fetch())</pre><p>We’re using inspect in is_async_caller() to get the function object that called us and determine whether its a coroutine or not. While this is fancy and illustrates the possibilities, it may not be very performant. We could easily replace the mechanism with an async_execute argument in the fetch wrapper and have the user decide.</p><p>The call_sync_fetch and call_async_fetch functions are there to show how a user could go about calling our wrapper. As you can see, there’s no need to await the fetch call because it’s done automatically by running in a separate thread.</p><p>This could prove useful to any python packages that are wanting to add support for asynchronous execution while still supporting legacy code. I’m sure there are pros and cons, feel free to start the discussion in the comments below.</p><figure><a href="https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fpublish.twitter.com%2F&amp;ref_src=twsrc%5Etfw&amp;region=follow_link&amp;screen_name=tryexceptpass&amp;tw_p=followbutton"><img alt="" src="https://cdn-images-1.medium.com/max/181/1*_04PhhMqHB3qXQnOQNP5ag.png" /></a></figure><p>If you liked this article and want to keep up with what I’m working on, please recommend it, visit <a href="http://tryexceptpass.org/">tryexecptpass.org</a> for more topics and <a href="https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fpublish.twitter.com%2F&amp;ref_src=twsrc%5Etfw&amp;region=follow_link&amp;screen_name=tryexceptpass&amp;tw_p=followbutton">follow me on Twitter</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec0a0f4b79ba" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/controlling-python-async-creep-ec0a0f4b79ba">Controlling Python Async Creep</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making 3D Interfaces for Python with Unity3D]]></title>
            <link>https://medium.com/hackernoon/user-interfaces-with-unity3d-and-python-eb2e7744518a?source=rss-138c0eb26be5------2</link>
            <guid isPermaLink="false">https://medium.com/p/eb2e7744518a</guid>
            <category><![CDATA[unity3d]]></category>
            <category><![CDATA[game-development]]></category>
            <category><![CDATA[user-interface]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[interfaces]]></category>
            <dc:creator><![CDATA[Cristian Medina]]></dc:creator>
            <pubDate>Fri, 19 May 2017 14:20:18 GMT</pubDate>
            <atom:updated>2018-04-01T03:44:30.384Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using Sofi and WebSockets to command game engines</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aVg78EydVeq5albTLfiXEA.jpeg" /></figure><p>A while back I wrote a post on <a href="http://tryexceptpass.org/on-democratizing-the-next-generation-of-user-interfaces">using with game engines</a> as frontend interfaces. I want to enable rich interfaces in the Python ecosystem that are usable in virtual and augmented reality. Since then, I was able to build some basic concepts on top of <a href="https://www.github.com/tryexceptpass/sofi">Sofi</a> and I’m here to share them.</p><h4>The backend</h4><p>Sofi works as a WebSocket server that can issue commands and handle events from a client. It’s written to simplify the use of frontend web technologies as interfaces to a Python backend. You can even <a href="http://tryexceptpass.org/how-to-turn-a-web-app-into-a-desktop-app">package it as a desktop app</a> with a desktop look and feel.</p><p>It functions by sending the user to a webpage with the basics needed to open a WebSocket client. This client then registers handlers that process server commands telling it how to alter the DOM or respond to events. All the logic resides with Python, the webpage is the user interface.</p><p>I wrote it with enough modularity such that the command and event structure is reusable for any technology. So it wasn’t much of a stretch to think that I could replace the webpage portion with a game engine. It was only a matter of finding one with easy access to WebSocket clients.</p><h4>The game engine</h4><p>I looked at Unity3D, Unreal and CryEngine. Of the three, the first two have the most expansive communities, and of those, Unity seems to have the larger ecosystem. While usually a matter of taste, I’m not here to argue which one has better graphics. At this point, I want simplicity for a proof of concept, and this came in the form of <a href="https://github.com/sta/websocket-sharp">WebSocketSharp</a>.</p><p>WebSocketSharp is a C# library available in Unity3D’s asset store. It provides client and server classes usable by any script in your project’s scene. I ran a few tests and it worked well enough for my needs. Mapping back to Sofi, this part performs the function of the JavaScript WebSocket client library that runs on your browser. Now, let’s wrap it so it understands how to talk to the Sofi WebSocket server, and make the equivalent of sofi.js.</p><h4>Making the frontend</h4><p>If you haven’t used Unity3D before, or any game engine for that matter, getting to “hello world” can be confusing. While not required to understand this implementation, I’ll go over some of the main concepts here. Please note that Unity has their own extensive <a href="https://docs.unity3d.com/Manual/index.html">documentation</a> available for more details.</p><p>First thing to do when making a “game” — which is what I’m doing for all intensive purposes — is to create a project. Think of this as a package that contains all the assets you’ll need for rendering a world and its interactions. Almost everything in a project is an asset: textures, shaders, scripts, objects, etc.</p><p>Within a project there are scenes. Think of these like game levels, or scenes in a movie. It’s a collection of objects, scripts and camera views that render into an experience with a common objective.</p><p>Once there’s a scene, I can add the WebSocketSharp library as an asset. This makes it possible to import its WebSocket class into a script in which to do some magic. You can assign scripts to objects in the scene and trigger execution through the engine’s event loop.</p><p>Every scene has a default camera. Cameras, among other things, determine how to view the scene. This includes viewing angles, distance, render settings, lighting, etc. But since they’re objects themselves, they can also hold scripts. Which means I can stick a WebSocketClient script on the camera that loads immediately with the scene.</p><p>Unity scripts can schedule handlers in the main event loop. There are many game events for physical collisions or user inputs detailed in the docs. But I’m interested in those that run at initialization and on a regular basis. In our case, I’m looking for Update and Start. The former runs when rendering every frame, while the latter executes after loading. I also register a handler for OnDestroy to clean things up when exiting.</p><p>To configure the client, instantiate a WebSocket during the engine’s Start event. This requires the address and port of the server, as well as its own set of event handlers. I use OnOpen when a connection completes, OnMessage when receiving new data, and OnClose for disconnections.</p><p>It’s important to think about the event loop when working with Start. Message processing cannot happen here because it can block the scene from loading. So I made a command queue in the form of a List object that lives within the client class. The idea is to review that queue on every Update event — once every frame — and handle the commands then.</p><p>Besides message processing, the Update event is a good point at which to check for a connection. It’s easy to trigger a reconnect, allowing the scene to wait for a server to spin up if needed.</p><p>The engine gathers user inputs and queues them up, much like the earlier socket messages. So I add checks for those here as well, using the sendAsync() method to communicate them back to the server.</p><h4>Communicating with Sofi</h4><p>To talk a little bit about protocol, Sofi’s communication mechanism is very extensible. The socket server listens on a particular port expecting JSON data. Event messages have an &#39;event&#39; key set with the name which the server uses to check for callbacks. If one is present, execution passes to that function, which can in turn use the dispatch() method to send a command. UI commands have a &#39;command&#39; key to identify them and their other parameters. Except for the basics, I don’t hardcode any of this, so it’s very easy to add custom events and commands by making up new names without modifying Sofi itself.</p><p>The WebSocketClient can handle commands for spawning new objects, updating object properties, removing objects and subscribing to events. The types of objects are the very basic shapes: cubes, spheres, cylinders, capsules, planes and text. There are quite a few settings for each, including color, size, position and rotation.</p><h4>Packaging</h4><p>The other neat part is that Unity3D can produce binaries for all major platforms. This includes Windows, Linux, MacOS, Android, iOS, and otherx. So not only does this add 3D capabilities to Python, but it also includes multi-platform support. The build process itself is simple and directed by the game engine itself. Couple it with PyInstaller like in <a href="https://medium.freecodecamp.com/the-python-desktop-application-3a66b4a128d3">my previous article</a>, and now you have only one executable for distribution, Python environment and all.</p><h4>What does it look like?</h4><p>For a demo of what it can do, have a look at my PyCaribbean presentation below. I run it at about 20:09.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FTcx8kwmGWA0%3Fstart%3D1209%26feature%3Doembed%26start%3D1209&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DTcx8kwmGWA0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FTcx8kwmGWA0%2Fhqdefault.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/7d00b8efd626e66112ac05b9def1a93c/href">https://medium.com/media/7d00b8efd626e66112ac05b9def1a93c/href</a></iframe><p>I’ve posted the Python code on GitHub under the <a href="https://github.com/tryexceptpass/sofi-unity3d">sofi-unity3d</a> repo. The WebSocketClient implementation is located in the engine/Assets folder. It also includes the Unity3D project along with it. The full code will run through my slides for PyCaribbean and starts a listener for any #python tweet.</p><p>I also went off and built the binaries for <a href="https://s3.amazonaws.com/tryexceptpass/sofi3d.x86.zip">Linux</a>, <a href="https://s3.amazonaws.com/tryexceptpass/sofi3d.exe">Windows</a> and <a href="https://s3.amazonaws.com/tryexceptpass/sofi3d.app.zip">MacOS</a>, so you can download and play with it yourselves. When you run the engine, it will load an empty sky and grassy plane scene that sits waiting for a WebSocket server. This could be the Sofi instance in server.py or something of your own making that listens on port 9000.</p><h4>What’s next?</h4><p>Today we’ve explored the possibilities around creating game engine based “widget libraries”. Demonstrating that it’s very possible for a Python backend to provide the logic, while leaving the UI heavy lifting to the engine. It’s also portable cross-platform and distributable with little effort.</p><p>The next step on the game engine side is to provide practical widgets and assets. With this framework we can spawn any pre-existing game objects, meshes and scripts. It’s up to us to define what makes for good reusable components and interactions, and package them with the built binaries.</p><p>On the Python side, I’d like to provide an HTTP-based asset server. This allows folks to create custom assets in something like <a href="https://www.blender.org/">Blender</a>, which is also Python-enabled, and load them into any scene. No need to touch the game engine.</p><p>I also want to build a world with that enables VR or AR on hardware like the Rift, Vibe, Hololens, etc. I have a DK2 at home, but haven’t looked into the making any projects with it yet.</p><p>For additional reading on how I made Sofi, have a look at the <a href="https://medium.com/@tryexceptpass/a-python-ate-my-gui-971f2326ce59">A Python Ate My GUI</a> series.</p><figure><a href="https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fpublish.twitter.com%2F&amp;ref_src=twsrc%5Etfw&amp;region=follow_link&amp;screen_name=tryexceptpass&amp;tw_p=followbutton"><img alt="" src="https://cdn-images-1.medium.com/max/181/1*_04PhhMqHB3qXQnOQNP5ag.png" /></a></figure><p>If you liked this article and want to keep up with what I’m working on, please recommend it, visit <a href="http://tryexceptpass.org/">tryexecptpass.org</a> for more topics and <a href="https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fpublish.twitter.com%2F&amp;ref_src=twsrc%5Etfw&amp;region=follow_link&amp;screen_name=tryexceptpass&amp;tw_p=followbutton">follow me on Twitter</a>.</p><figure><a href="http://bit.ly/HackernoonFB"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0hqOaABQ7XGPT-OYNgiUBg.png" /></a></figure><figure><a href="https://goo.gl/k7XYbx"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vgw1jkA6hgnvwzTsfMlnpg.png" /></a></figure><figure><a href="https://goo.gl/4ofytp"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gKBpq1ruUi0FVK2UM_I4tQ.png" /></a></figure><blockquote><a href="http://bit.ly/Hackernoon">Hacker Noon</a> is how hackers start their afternoons. We’re a part of the <a href="http://bit.ly/atAMIatAMI">@AMI</a> family. We are now <a href="http://bit.ly/hackernoonsubmission">accepting submissions</a> and happy to <a href="/service/mailto:partners@amipublications.com">discuss advertising &amp; sponsorship</a> opportunities.</blockquote><blockquote>If you enjoyed this story, we recommend reading our <a href="http://bit.ly/hackernoonlatestt">latest tech stories</a> and <a href="https://hackernoon.com/trending">trending tech stories</a>. Until next time, don’t take the realities of the world for granted!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*35tCjoPcvq6LbB3I6Wegqw.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb2e7744518a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/user-interfaces-with-unity3d-and-python-eb2e7744518a">Making 3D Interfaces for Python with Unity3D</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Turn a Web App Into a Desktop App, Using Chromium and PyInstaller]]></title>
            <link>https://medium.com/free-code-camp/the-python-desktop-application-3a66b4a128d3?source=rss-138c0eb26be5------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a66b4a128d3</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Cristian Medina]]></dc:creator>
            <pubDate>Thu, 27 Apr 2017 23:23:18 GMT</pubDate>
            <atom:updated>2018-04-01T03:45:51.074Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lmD7FIkUL4t5P-swMEPH7Q.jpeg" /></figure><p>Packaging and distributing your app sounds simple in principle. It’s just software. But in practice, it’s quite challenging.</p><p>I’ve been working on a Python module called <a href="https://github.com/tryexceptpass/sofi">Sofi</a> that generates user interfaces. It can deliver a desktop feel while using standard single-page web technologies. For flexibility, I designed it to work through two methods of distribution: in-browser and executable.</p><p>Running in the browser, it functions much like a normal webpage. You can load it by opening a file, or launch it from your shell. I also built an executable that runs as a packaged app, independent and without external requirements.</p><p>Over time, as I hacked at code in Atom — my editor of choice these days — I remembered that Atom is actually a browser. It uses Node.js as a back end, and the Electron framework for its user interface. This inspired me to start poking at Electron’s internals, hoping to find examples and best practices on how they solved desktop packaging.</p><p>It didn’t take long for me to discover that it’s all built on top of free and open sourced technologies: the Chromium browser and the <a href="https://bitbucket.org/chromiumembedded/cef">Chromium Embedded Framework</a>. This featured easy-to-integrate example customizations that were capable of fulfilling my requirements.</p><p>With all this in hand, I got to work.</p><h4>The Chromium Embedded Framework</h4><p>Chromium is the base code that feeds Google’s Chrome browser. It brings together all the elements that render an interface, process user input, and script its functions.</p><p>The Chromium Embedded Framework (CEF) is a group of C functions that that can control that browser. It also provides scripts that help simplify the process of building and compiling it.</p><p>Visual Studio Code, Slack, Mattermost, Curse, Postman, and Kitematic are all examples of desktop apps that use Electron. These systems all qualify as websites that exploit the browser underneath with CEF.</p><p>If you’re thinking that Python can bind with C and take advantage of these features as well, then you’re right. Look no further than the <a href="https://github.com/cztomczak/pycef">pycef</a> project to call the CEF wrapper functions directly. However, it does come with the Chromium binary as an added dependency. So if you’re worried about managing complicated support statements, think before you jump.</p><p>In my particular situation, the Sofi project manages all interactions through a websocket, providing a consistent interface across different types of platforms (web, desktop, mobile, etc.). This means I don’t need to manually commanding or drive the browser. I only wish to interact with the DOM that the browser displays through standard web technologies.</p><p>My goal is to customize the UI elements that make a browser look like a browser. I need to remove the menus, toolbars, and status bars. In doing so, I’ll make it appear that we’re in fullscreen mode — but inside an application window.</p><p>Given my simple requirements, I felt that pycef — or any other lower-level bindings — was too much. Instead I took advantage of a pre-built sample from the CEF project: <em>cefsimple</em>. This browser hides all the visual elements I want, so if I use its CLI to open a webpage, the user has no idea that they’re actually inside a browser. It looks like a regular window from any application.</p><p>Building <em>cefsimple</em> wasn’t too complicated once I went through the documentation. But it takes an enormous amount of time if you also build Chromium along with it. To avoid this, the project itself provides pre-built binaries that you can customize and compile into cefsimple. I found it best to take advantage of these.</p><p>The steps are as follows:</p><ol><li>Have a quick look through <a href="https://bitbucket.org/chromiumembedded/cef/wiki/GeneralUsage#markdown-header-using-a-binary-distribution">how to build</a> with CEF from binaries.</li><li>Grab one of the <a href="http://opensource.spotify.com/cefbuilds/index.html">binary distributions</a> from the repo. Be sure to read the tooltips before selecting one, since not all packages contain the same files. I was specifically looking for one with cefsimple.</li><li>Look through the CMakeLists.txt file and make sure you install the necessary build tools. This is platform specific.</li><li>Perform the build. This is explained in the same file as the previous step and is also platform specific, but it tends to follow the process of: make and cd into build directory, run cmake for your compilation tools and architecture while pointing at the parent directory. Since I used the OSX Ninja tools on a 64-bit platform, the command looked like cmake -G &quot;Ninja&quot; -DPROJECT_ARCH=&quot;x86_64&quot; ..</li><li>The build directory will now contain the output files. The structure can be a little confusing, but it’s described in the main README. As a reference, the previous step resulted in an app bundle under build/tests/cefsimple/Release/cefsimple.app.</li><li>Don’t forget you’ll have to do this to create the binaries you need for every platform and OS architecture that your supporting.</li></ol><p>Now that you have an executable, run it from command line with --url set to the webpage you want to open. This means that incorporating it into a Python script is easily done through the subprocess module.</p><p>While not required, if you’re interested in compiling Chromium itself, have a look at the CEF documentation. It will point you in the right direction. But be warned, it takes a lot of time to download, build and compile. Good old fashioned processing horsepower will definitely help get faster results.</p><h4>Packaging</h4><p>Now that we can deliver a desktop experience, we have to consider how to distribute that to our users. Traditional Python package distribution is accomplished through the Python Package Index (PyPI). However, it requires our users to install the Python interpreter and some form of packaging tool like easy_install or pip.</p><p>While this isn’t particularly hard, you should consider the wider range of users. Managing an install process with separate manual steps gets fairly complicated. Especially with non-technical audiences — some of whom don’t know that Python is anything other than a large snake. While others may at least know the air speed velocity of a European unladen swallow.</p><p>If they do know the language, most already have their own version installed. This is where package dependencies, different operating systems, browsers you’ve never heard of (or thought were dead by now) come into play, along with users’ varying skills in setting up virtual environments. This tends to translate into a large amount of time spent supporting mismatched software.</p><p>To avoid such a large mess, there are tools that can embed all your dependencies into OS-specific executable files. After careful consideration, the one I chose for my endeavors is <a href="https://github.com/pyinstaller/pyinstaller">PyInstaller</a>. It seems to provide the most flexibility in supported platforms and formats.</p><p>A brief excerpt from their GitHub repository sums things up nicely:</p><blockquote>PyInstaller reads a Python script written by you. It analyzes your code to discover every other module and library your script needs in order to execute. Then it collects copies of all those files — including the active Python interpreter! — and puts them with your script in a single folder, or optionally in a single executable file.</blockquote><p>The tool delivered on its promise. I pointed it to the Python file for my sample application and it bundles it in a directory easily enough with: pyinstaller sample.py. When I want an executable instead, just add the --onefile parameter.</p><p>It gets a bit trickier when you need to add non-Python data to your bundle. This is the case with the html and js files that form the basis of Sofi, and the <em>cefsimple</em> browser that presents the application interface from earlier. The PyInstaller utility provides --add-data to do just that, allowing a mapping to the path within your bundle where the data file (or directory) will reside. However, it took me a while to figure out how to properly access those directories from within my code. Luckily the documentation pointed me in the right direction.</p><p>As it turns out, when running a PyInstaller bundled application, you can’t rely on __file__ and similar mechanisms to determine paths. Instead, the PyInstaller bootloader stores the absolute path to the bundle in sys._MEIPASS and adds a frozen attribute to let you know that you’re running inside a bundle. If sys.frozen is True then load your files based on sys._MEIPASS, otherwise use normal path functions to determine where things are.</p><p>I was able to successfully create both an OSX bundled app and an executable Linux binary of the same Python script. I verified I can do the same with a Windows executable, but haven’t had time to put together a Windows version of the <em>cefsimple</em> browser to test the bundle path yet.</p><h4>The Final Product</h4><p>For an example of the browser-based user interface packaged with the system described here, have a look at my presentation at PyCaribbean 2017.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FTcx8kwmGWA0%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DTcx8kwmGWA0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FTcx8kwmGWA0%2Fhqdefault.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/07e1ddbe778930e60dd2e5db30c12002/href">https://medium.com/media/07e1ddbe778930e60dd2e5db30c12002/href</a></iframe><p>The demo relevant to CEF and packaging is of an image gallery and it appears around 18:15.</p><p>For additional reading on how I made Sofi, have a look at the <a href="http://tryexceptpass.org/article/a-python-ate-my-gui/">A Python Ate My GUI</a> series.</p><figure><a href="https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fpublish.twitter.com%2F&amp;ref_src=twsrc%5Etfw&amp;region=follow_link&amp;screen_name=tryexceptpass&amp;tw_p=followbutton"><img alt="" src="https://cdn-images-1.medium.com/max/181/1*_04PhhMqHB3qXQnOQNP5ag.png" /></a></figure><p>If you liked this article and want to keep up with what I’m working on, please recommend it, visit <a href="http://tryexceptpass.org/">tryexecptpass.org</a> for more topics and <a href="https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fpublish.twitter.com%2F&amp;ref_src=twsrc%5Etfw&amp;region=follow_link&amp;screen_name=tryexceptpass&amp;tw_p=followbutton">follow me on Twitter</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a66b4a128d3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/free-code-camp/the-python-desktop-application-3a66b4a128d3">How to Turn a Web App Into a Desktop App, Using Chromium and PyInstaller</a> was originally published in <a href="https://medium.com/free-code-camp">We’ve moved to freeCodeCamp.org/news</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Threaded Asynchronous Magic and How to Wield It]]></title>
            <link>https://medium.com/hackernoon/threaded-asynchronous-magic-and-how-to-wield-it-bba9ed602c32?source=rss-138c0eb26be5------2</link>
            <guid isPermaLink="false">https://medium.com/p/bba9ed602c32</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[multithreading]]></category>
            <category><![CDATA[asyncio]]></category>
            <dc:creator><![CDATA[Cristian Medina]]></dc:creator>
            <pubDate>Tue, 06 Dec 2016 18:06:16 GMT</pubDate>
            <atom:updated>2018-04-01T03:46:20.162Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VG4z1QlZa6J7siTU0zkTjQ.jpeg" /><figcaption>Photo Credit: <a href="https://en.wikipedia.org/wiki/File:CTA_loop_junction.jpg">Daniel Schwen</a> via Wikipedia</figcaption></figure><h4>A dive into Python’s asyncio tasks and event loops</h4><p>Ok let’s face it. Clock speeds no longer govern the pace at which computer processors improve. Instead we see increased transistor density and higher core counts. Translating to software terms, this means that code won’t run faster, but more of it can run in parallel.</p><p>Although making good use of our new-found silicon real estate requires improvements in software, a lot of programming languages have already started down this path by adding features that help with parallel execution. In fact, they’ve been there for years waiting for us to take advantage.</p><p>So why don’t we? A good engineer always has an ear to the ground, listening for the latest trends in his industry, so let’s take a look at what Python is building for us.</p><h4>What do we have so far?</h4><p>Python enables parallelism through both the threading and the multiprocessing libraries. Yet it wasn’t until the 3.4 branch that it gave us the <em>asyncio </em>library to help with single-threaded concurrency. This addition was key in providing a more convincing final push to start swapping over from version 2.</p><p>The asyncio package allows us to define coroutines. These are code blocks that have the ability of yielding execution to other blocks. They run inside an <em>event loop</em> which iterates through the scheduled tasks and executes them one by one. A task switch occurs when it reaches an await statement or when the current task completes.</p><p>Task execution itself happens the same as in a single-threaded system. Meaning, this is not an implementation of parallelism, it’s actually closer to multithreading. We can perceive the concurrency in situations where a block of code depends on external actions.</p><p>This illusion is possible because the block can yield execution while it waits, making anything that depends on external IO, like network or disk storage, a great candidate. When the IO completes, the coroutine receives an interrupt and can proceed with execution. In the meantime, other tasks execute.</p><p>The asyncio event loop can also serve as a task scheduler. Both asynchronous and blocking functions can queue up their execution as needed.</p><h4>Tasks</h4><p>A Task represents callable blocks of code designed for asynchronous execution within event loops. They execute single-threaded, but can run in parallel through loops on different threads.</p><p>Prefixing a function definition with the async keyword turns it into an asynchronous coroutine. Though the task itself will not exist until it’s added to a loop. This is usually implicit when calling most loop methods, but asyncio.ensure_future(your_coroutine) is the more direct mechanism.</p><p>To denote an operation or instruction that can yield execution, we use the await keyword. Although it’s only available within a coroutine block and causes a syntax error if used anywhere else.</p><p>Please note that the async keyword was not implemented until Python version 3.5. So when working with older versions, use the @asyncio.coroutine decorator and yield from keywords instead.</p><h4>Scheduling</h4><p>In order to execute a task, we need a reference to the event loop in which to run it. Using loop = asyncio.get_event_loop() gives us the current loop in our execution thread. Now it’s a matter of calling loop.run_until_complete(<em>your_coroutine</em>) or loop.run_forever() to have it do some work.</p><p>Let’s look at a short example to illustrate a few points. I strongly encourage you to open an interpreter and follow along:</p><pre>import time<br>import asyncio</pre><pre>async def do_some_work(x):<br>    print(&quot;Waiting &quot; + str(x))<br>    await asyncio.sleep(x)</pre><pre>loop = asyncio.get_event_loop()<br>loop.run_until_complete(do_some_work(5))</pre><p>Here we defined do_some_work() as a coroutine that waits on the results of external workload. The workload is simulated through asyncio.sleep.</p><p>Running the code may be surprising. Did you expect run_until_complete to be a <em>blocking</em> call? Remember that we’re using the event loop from the <strong>current</strong> <strong>thread </strong>to execute the task. We’ll discuss alternatives in more detail later. So for now, the important part is to understand that while execution blocks, the await keyword still enables concurrency.</p><p>For a better picture, let’s change our test code a bit and look at executing tasks in batches:</p><pre>tasks = [asyncio.ensure_future(do_some_work(2)), <br>         asyncio.ensure_future(do_some_work(5))]</pre><pre>loop.run_until_complete(asyncio.gather(*tasks))</pre><p>Introducing the asyncio.gather() function enables results aggregation. It waits for several tasks in the same thread to complete and puts the results in a list.</p><p>The main observation here is that both function calls did not execute in sequence. It did not wait 2 seconds, then 5, for a total of 7 seconds. Instead it <em>started </em>to wait 2s, then moved on to the next item which <em>started </em>to wait 5s, returning when the longer task completed, for a total of 5s. Feel free to add more print statements to the base function if it helps visualize.</p><p>This means that we can put long running tasks with <em>awaitable </em>code in an execution batch, then ask Python to run them in parallel and wait until they all complete. If you plan it right, this will be faster than running in sequence.</p><p>Think of it as an alternative to the threading package where after spinning up a number of Threads, we wait for them to complete with .join(). The major difference is that there’s less overhead incurred than creating a new thread for each function.</p><p>Of course, it’s always good to point out that your millage may vary based on the task at hand. If you’re doing compute-heavy work, with little or no time waiting, then the only benefit you get is the grouping of code into logical batches.</p><h4>Running a loop in a different thread</h4><p>What if instead of doing everything in the current thread, we spawn a separate Thread to do the work for us.</p><pre>from threading import Thread<br>import asyncio</pre><pre>def start_loop(loop):<br>    asyncio.set_event_loop(loop)<br>    loop.run_forever()</pre><pre>new_loop = asyncio.new_event_loop()<br>t = Thread(target=start_loop, args=(new_loop,))<br>t.start()</pre><p>Notice that this time we created a new event loop through asyncio.new_event_loop(). The idea is to spawn a new thread, pass it that new loop and then call thread-safe functions (discussed later) to schedule work.</p><p>The advantage of this method is that work executed by the other event loop will not block execution in the current thread. Thereby allowing the main thread to manage the work, and enabling a new category of execution mechanisms.</p><h4>Queuing work in a different thread</h4><p>Using the thread and event loop from the previous code block, we can easily get work done with the call_soon(), call_later() or call_at() methods. They are able to run regular function code blocks (those not defined as coroutines) in an event loop.</p><p>However, it’s best to use their _threadsafe alternatives. Let’s see how that looks:</p><pre>import time</pre><pre>def more_work(x):<br>  print(&quot;More work %s&quot; % x)<br>  time.sleep(x)<br>  print(&quot;Finished more work %s&quot; % x)</pre><pre>new_loop.call_soon_threadsafe(more_work, 6)<br>new_loop.call_soon_threadsafe(more_work, 3)</pre><p>Now we’re talking! Executing this code does not block the main interpreter, allowing us to give it more work. Since the work executes in order, we now essentially have a task queue.</p><p>We just went to multi-threaded execution of single-threaded code, but isn’t concurrency part of what we get with asyncio? Sure it is! That loop on the worker thread is still async, so let’s enable parallelism by giving it awaitable coroutines.</p><p>Doing so is a matter of using asyncio.run_coroutine_threadsafe(), as seen bellow:</p><pre>new_loop.call_soon_threadsafe(more_work, 20)<br>asyncio.run_coroutine_threadsafe(do_some_work(5), new_loop)<br>asyncio.run_coroutine_threadsafe(do_some_work(10), new_loop)</pre><p>These instructions illustrate how python is going about execution. The first call to more_work blocks for 20 seconds, while the calls to do_some_work execute in parallel immediately after more_work finishes.</p><h4>Real World Example #1 — Sending Notifications</h4><p>A common situation these days is to send notifications as a result of a task or event. This is usually simple, but talking to an email server to submit a new message can take time, and so can crafting the email itself.</p><p>There are many scenarios where we don’t have the luxury of waiting around for tasks to complete. Where doing so provides no benefit to the end user. A prime example being a request for a password reset, or a webhook event that triggers repository builds and emails the results.</p><p>The recommended practice so far has been to use a task queuing system like celery, on top of a message queue server like rabbitmq to schedule the work. I’m here to tell you that for small things that can easily execute from another thread of your main application, it’s not a bad idea to just use asyncio. The pattern being fairly similar to the code examples we’ve seen so far:</p><pre>import asyncio<br>import smtplib<br>from threading import Thread</pre><pre>def send_notification(email):<br>    &quot;&quot;&quot;Generate and send the notification email&quot;&quot;&quot;</pre><pre>    # Do some work to get email body<br>    message = ...<br>    <br>    # Connect to the server<br>    server = smtplib.SMTP(&quot;smtp.gmail.com:587&quot;)<br>    server.ehlo()<br>    server.starttls()<br>    server.login(username, password)</pre><pre>    # Send the email<br>    server.sendmail(from_addr, email, message)</pre><pre>    server.quit()</pre><pre>def start_email_worker(loop):<br>    &quot;&quot;&quot;Switch to new event loop and run forever&quot;&quot;&quot;<br> <br>    asyncio.set_event_loop(loop)<br>    loop.run_forever()</pre><pre># Create the new loop and worker thread<br>worker_loop = asyncio.new_event_loop()<br>worker = Thread(target=star_email_worker, args=(worker_loop,))</pre><pre># Start the thread<br>worker.start()</pre><pre># Assume a Flask restful interface endpoint<br>@app.route(&quot;/notify&quot;)<br>def notify(email):<br>    &quot;&quot;&quot;Request notification email&quot;&quot;&quot;</pre><pre>    worker_loop.call_soon_threadsafe(send_notification, email)</pre><p>Here we assume a Flask web API with an endpoint mounted at /notify in which to request a notification email of some sort.</p><p>Notice that send_notification is not a coroutine, so each email will be a blocking call. The worker thread’s event loop will serve as the queue in which to track the outgoing emails.</p><p>Why are the SMTP calls synchronous you wonder? Well, while this is a good example of what <em>should</em> be awaitable IO, I’m not aware of an asynchronous SMTP library at the moment. Feel free to substitute with an async def, await and run_coroutine_threadsafe, if you do find one.</p><h4>Real World Example #2 — Parallel Web Requests</h4><p>Here’s an example of batching HTTP requests that run concurrently to several servers, while waiting for responses before processing. I expect it to be useful for those of you that do a lot of scraping, as well as a quick intro to the <a href="http://aiohttp.readthedocs.io/en/stable/">aiohttp</a> module.</p><pre>import asyncio<br>import aiohttp</pre><pre>async def fetch(url):<br>    &quot;&quot;&quot;Perform an HTTP GET to the URL and print the response&quot;&quot;&quot;</pre><pre>    response = await aiohttp.request(&#39;GET&#39;, url)<br>    return await response.text()</pre><pre># Get a reference to the event loop<br>loop = asyncio.get_event_loop()</pre><pre># Create the batch of requests we wish to execute<br>requests = [asyncio.ensure_future(fetch(&quot;https://github.com&quot;)),<br>            asyncio.ensure_future(fetch(&quot;https://google.com&quot;))]</pre><pre># Run the batch<br>responses = loop.run_until_complete(asyncio.gather(*requests))</pre><pre># Examine responses<br>for resp in responses:<br>  print(resp)</pre><p>Fairly straightforward, it’s a matter of grouping the work in a list of tasks and using run_until_complete to get the responses back. This can easily change to use a separate thread in which to make requests, where it would be simple to add all the URLs through the thread-safe methods described previously.</p><p>I want to note that the <a href="http://docs.python-requests.org/en/v0.10.6/user/advanced/#asynchronous-requests">requests</a> library has asynchronous support through gevent, but I haven’t done the work to figure out how that can tie into asyncio. In contrast, I’m not aware of asyncio plans for the popular scraping framework scrapy, but I assume they’re working on it.</p><h4>Stopping the loop</h4><p>If at any point you find yourself wanting to stop an infinite event loop, or want to cancel tasks that haven’t completed, I tend to use a KeyboardInterrupt exception clause to trigger cancellation as shown below. Although the same can be accomplished by using the signal module and registering a handler for signal.SIGINT.</p><pre>try:<br>    loop.run_forever()</pre><pre>except KeyboardInterrupt:<br>    # Canceling pending tasks and stopping the loop<br>    asyncio.gather(*asyncio.Task.all_tasks()).cancel()</pre><pre>    # Stopping the loop<br>    loop.stop()</pre><pre>    # Received Ctrl+C<br>    loop.close()</pre><p>This time we’re introducing the use of Task.all_tasks() to generate a list of all currently running or scheduled tasks. When coupled with gather() we can send the cancel() command to each one and have them all stop executing or remove them from the queue.</p><p>Please note that due to signaling deficiencies in Windows, if the loop is empty, the keyboard interrupt is never triggered. A workaround for this situation is to queue a task that sleeps for several seconds. This guarantees that if the interrupt arrives while the task sleeps, the loop will notice when it wakes.</p><p>Asynchronous programming can be very confusing. I must confess that I started with some base assumptions that turned out to be wrong. It wasn’t until I dove deeper into it that I realized what’s really going on.</p><p>I hope this served as a good introduction to asyncio event loops and tasks, as well as their possible uses. I know there are plenty of other articles out there, but I wanted to make something that tied things to some real world examples. If you have any questions or comments, feel free to drop them below and I’ll help as best I can.</p><figure><a href="https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fpublish.twitter.com%2F&amp;ref_src=twsrc%5Etfw&amp;region=follow_link&amp;screen_name=tryexceptpass&amp;tw_p=followbutton"><img alt="" src="https://cdn-images-1.medium.com/max/181/1*_04PhhMqHB3qXQnOQNP5ag.png" /></a></figure><p>If you liked this article and want to keep up with what I’m working on, please recommend it, visit <a href="http://tryexceptpass.org/">tryexecptpass.org</a> for more topics and <a href="https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fpublish.twitter.com%2F&amp;ref_src=twsrc%5Etfw&amp;region=follow_link&amp;screen_name=tryexceptpass&amp;tw_p=followbutton">follow me on Twitter</a>.</p><figure><a href="http://bit.ly/HackernoonFB"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0hqOaABQ7XGPT-OYNgiUBg.png" /></a></figure><figure><a href="https://goo.gl/k7XYbx"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vgw1jkA6hgnvwzTsfMlnpg.png" /></a></figure><figure><a href="https://goo.gl/4ofytp"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gKBpq1ruUi0FVK2UM_I4tQ.png" /></a></figure><blockquote><a href="http://bit.ly/Hackernoon">Hacker Noon</a> is how hackers start their afternoons. We’re a part of the <a href="http://bit.ly/atAMIatAMI">@AMI</a> family. We are now <a href="http://bit.ly/hackernoonsubmission">accepting submissions</a> and happy to <a href="/service/mailto:partners@amipublications.com">discuss advertising &amp; sponsorship</a> opportunities.</blockquote><blockquote>If you enjoyed this story, we recommend reading our <a href="http://bit.ly/hackernoonlatestt">latest tech stories</a> and <a href="https://hackernoon.com/trending">trending tech stories</a>. Until next time, don’t take the realities of the world for granted!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*35tCjoPcvq6LbB3I6Wegqw.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bba9ed602c32" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/threaded-asynchronous-magic-and-how-to-wield-it-bba9ed602c32">Threaded Asynchronous Magic and How to Wield It</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Trusted Packaging Index]]></title>
            <link>https://medium.com/python-pandemonium/the-trusted-packaging-index-d16986de73c6?source=rss-138c0eb26be5------2</link>
            <guid isPermaLink="false">https://medium.com/p/d16986de73c6</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[packaging]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Cristian Medina]]></dc:creator>
            <pubDate>Wed, 16 Nov 2016 15:57:19 GMT</pubDate>
            <atom:updated>2018-04-01T03:46:55.150Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QLTvjb2KYPendbXWFDjdkA.jpeg" /></figure><h4>A proposal for funding PyPI infrastructure and development</h4><p>A few days ago, I was listening to the latest episode of <a href="https://talkpython.fm/episodes/show/84/are-we-failing-to-fund-python-s-core-infrastructure">Talk Python To Me: Are we failing to fund Python’s core infrastructure?</a>, which had a panel of guests from the <a href="https://www.python.org/psf/">Python Software Foundation</a>, <a href="https://pypi.org/">PyPI</a> and <a href="https://readthedocs.org/">Read The Docs</a>. As someone that writes open source code, the topic of sustainability is always floating around in my mind. Being able to mostly work on the things that tickle my brain would definitely be awesome, but even if you had a fantastically successful project — which I don’t — it still is extraordinarily difficult to achieve.</p><p>I always wondered how organizations like the PSF made it all work, especially with infrastructure and systems that have the level of traffic we see in PyPI. The closest parallel I can draw is to research projects, where a considerable amount of time is dedicated towards finding the <em>right</em> kind of funding.</p><h4>The problem</h4><p>There’s a lot of open source software, tooling and infrastructure out there providing a considerable number of capabilities that form essential parts of a larger ecosystem — Python in this particular case — without which many other software projects would be unable to function.</p><p>Don’t believe me? Let’s take the NPM craziness from earlier this year as an example, where the owner of a widely used package decided to take it down for a few interesting reasons, in essence breaking a large percentage of all packages, most of which were unaware of their dependency on it.</p><p>There was another problem a few years back with NPM SSL certificates — not trying to pick on these guys, it’s just fresh in my mind — where a simple change caused everyone’s installs and builds to fail. The fix was easy but the ecosystem so vast that it had a large impact.</p><p>If you listen to the podcast episode mentioned earlier, you’ll realize how critical the PyPI and Read The Docs systems really are. The packaging index alone is ferrying data around the vicinity of 40TB a month, with an infrastructure cost of $40k/month, donated by Rackspace. If Rackspace decided to stop funding this project, or PyPI were to go down in any way, what will the rest of us do? Yes there are contingencies but I don’t like resting one of the most basic Python features on the good will of just one donor. No offense to Rackspace, but there are always unforeseen circumstances, and I’m sure they would appreciate some partners.</p><p>These folks have been looking at possible sources of donations and income for a while, but there are no silver bullets. Even more so when you consider that some of the typical solutions for these types of projects (like advertisements) could be detrimental to the main use cases of the systems themselves.</p><p>If a company could pay for having their packages featured or prioritized in some way, or if you went to PyPI looking for an AirBnB module and found a set of ads that follow you around the internet trying to sell you on a Caribbean vacation, you may find it annoying and avoid visiting the site in the future. Any possible solutions will require careful consideration of their impacts on the users.</p><p>Given they are open to suggestions, I wanted to add what I could to the discussion, so below is an idea to help monetize PyPI without the use of advertising.</p><h4>The Idea</h4><p>I’ve worked in large corporations for the majority of my career, becoming acutely aware not only of the infrastructure needs of these businesses, but also of the security and compartmentalization requirements that keep trade secrets and intellectual property private and secure, as well as protect the business from ill-intended intruders that want to wreak havoc wherever they can.</p><p>In my experience over the many years of provisioning, testing and deploying, a lot of work and effort has gone into a number of package management systems — especially within operating systems — that provide varying degrees of functionality, all with the same purpose of centralizing distribution of software (usually in compiled binary form) across large swaths of systems, along with some implementation of dependency management.</p><p>However, it wasn’t until a different episode from Talk Python — one of the earlier ones, I think <a href="https://talkpython.fm/episodes/show/23/3d-printing-with-python-at-authentise">Episode 23</a> — that I heard of someone using pip for internal deployments. Ever since then, I’ve done my best to push that in my organizations, and boy oh boy has it made my life easier.</p><p>Spin up a docker container with a server — I use the <a href="https://pypi.org/project/pypiserver/">pypiserver</a> package — and after a few home directory config files, just python setup.py sdist upload -r your_server to push a package and pip install --extra-index-url your_server to install it wherever you’re deploying.</p><p>Of course, this is fairly simple, insecure and limiting. In big business, it’s usually only accepted for smaller segregated teams and less important / non-critical projects. If you can get away with it at all.</p><p>Enter what I’m calling the Trusted Package Index. Businesses always have a need for an on-premises, secure, encrypted and highly available distribution mechanism of compiled binaries. Together with setuptools providing various install capabilities that can cover non-python code just as well, it seems like we could put together a decent product. Something like a Docker Trusted Registry.</p><p>If you take warehouse, add LDAP and Active Directory support for authenticating both package uploads and downloads, provide relevant monitoring statistics, backend storage encryption, high availability, a replication mechanism, plus the ability to integrate webhooks from GitHub Enterprise, GitLab or any other source control system that can trigger package rebuilds, I think you could put together a fairly decent appliance with enough appeal to a number of businesses willing to pay a modest yearly fee for it, especially if you can find a way to provide a support plan along with it.</p><p>This is just a suggestion, developing it will take time and effort that may be hard to find, but not only would it serve as a source of funding, it may also help develop a few infrastructure advancements to distribute network load, like balancing the real PyPI across multiple volunteer sites running replication.</p><p>Supporting these systems and infrastructure is not an easy task, so I’d like to encourage anyone that has some extra time to pitch in where they can. If you have any ideas to help solve this, don’t be afraid to make them public by adding them in the comments below.</p><figure><a href="https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fpublish.twitter.com%2F&amp;ref_src=twsrc%5Etfw&amp;region=follow_link&amp;screen_name=tryexceptpass&amp;tw_p=followbutton"><img alt="" src="https://cdn-images-1.medium.com/max/181/1*_04PhhMqHB3qXQnOQNP5ag.png" /></a></figure><p>If you liked this article and want to keep up with what I’m working on, please recommend it, visit <a href="http://tryexceptpass.org/">tryexecptpass.org</a> for more topics and <a href="https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fpublish.twitter.com%2F&amp;ref_src=twsrc%5Etfw&amp;region=follow_link&amp;screen_name=tryexceptpass&amp;tw_p=followbutton">follow me on Twitter</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d16986de73c6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/python-pandemonium/the-trusted-packaging-index-d16986de73c6">The Trusted Packaging Index</a> was originally published in <a href="https://medium.com/python-pandemonium">Python Pandemonium</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to write your own Python documentation generator]]></title>
            <link>https://medium.com/python-pandemonium/python-introspection-with-the-inspect-module-2c85d5aa5a48?source=rss-138c0eb26be5------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c85d5aa5a48</guid>
            <category><![CDATA[documentation]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Cristian Medina]]></dc:creator>
            <pubDate>Tue, 25 Oct 2016 02:46:24 GMT</pubDate>
            <atom:updated>2018-04-01T03:47:17.373Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vn5gMzaD4ltSkH4VT4aWaQ.jpeg" /></figure><h4>Introspection with the inspect module</h4><p>In my early days with Python, one of the things that I really liked was using the built-in help function to examine classes and methods while sitting at the interpreter, trying to determine what to type next. This function imports an object and walks through its members, pulling out docstrings and generating manpage-like output to help give you an idea of how to use the object it was examining.</p><p>The beauty about it being built into the standard library is that with output being generated straight from code, it indirectly emphasizes a coding style for lazy people like me, who want to do as little extra work as possible to maintain documentation. Especially if you already choose straight forward names for your variables and functions. This style involves things like adding docstrings to your functions and classes, as well as properly identifying private and protected members by prefixing them with underscores.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fde61b678430fd6b5555dec73651f101/href">https://medium.com/media/fde61b678430fd6b5555dec73651f101/href</a></iframe><p>The help function is actually using the pydoc module to generate its output, which is also runnable from the command line to produce a text or html representation of any importable module in your path.</p><p>A little while ago I needed to write more detailed, formal, design documentation and — being a fan of Markdown — I decided to play around with <a href="http://www.mkdocs.org/">mkdocs</a><em> </em>to see if I could get what I was looking for. This module makes it easy to turn your markdown text into nicely styled web pages and can serve them up as you make changes before publishing to an official location. It comes with a template for <a href="https://readthedocs.org/">readthedocs</a><em> </em>and even provides an easy command line interface to push your changes into <a href="https://pages.github.com/">GitHub Pages</a> if you wish to go down that route.</p><p>Having completed my initial batch of text describing design decisions and considerations, I wanted to add details on the actual interface methods I was developing and exposing to the other modules in the project. Since I already wrote the definitions for most of those functions, I intended to automatically generate the reference pages from my source, and I wanted it in markdown so it could render into html along with the rest of my documentation whenever I ran mkdocs.</p><p>However, it turns out there is no default way of generating markdown from source with mkdocs, but there are plugins. After googling and researching a bit more, I was not content with the projects or plugins I found — lots of things are out of date, not maintained, or the output was just not what I was looking for — so I decided to write my own. I thought it would be another interesting foray into learning a little more about a module I used a little while ago when building a debugger for one of my previous articles (see <a href="https://medium.com/@tryexceptpass/hacking-together-a-simple-graphical-python-debugger-efe7e6b1f9a8#.jqe3no3k9">Hacking together a Simple Graphical Python Debugger</a>): the inspect module.</p><blockquote>“The <em>inspect</em> module provides several useful functions to help get information about live objects… ” — <a href="https://docs.python.org/3.6/library/inspect.html">Python Docs</a></blockquote><h4>Inspect this!</h4><p>Originating from the standard library, <em>inspect</em> not only lets you look at lower level python frame and code objects, it also provides a number of methods for examining modules and classes, helping you find the items that may be of interest. It’s what <em>pydoc</em> uses to generate the help files mentioned previously.</p><p>Browsing through the online documentation, you’ll find a number of methods relevant to our adventure. The most important ones being getmembers(), getdoc() and signature(), as well as the many is... functions that serve as filters to getmembers. With these we can easily iterate through functions, including distinctions between generators and coroutines, and recurse into any classes and their internals as needed.</p><h4>Importing the code</h4><p>If we’re going to inspect an object, no matter what it is, the first thing to do is provide a mechanism with which to import it into our namespace. Why even talk about imports? Depending on what you want to do, there’s a number of items to worry about, including virtual environments, custom code, standard modules and reused names. Things can get confusing, and getting it wrong can lead to some frustrating moments trying to figure things out.</p><p>We do have a few options here, the more complete one being a reuse of <a href="https://github.com/python/cpython/blob/master/Lib/pydoc.py#L318">safeimport</a>() directly from pydoc, which takes care of a number of special cases for us and raises a pretty ErrorDuringImport exception when things break. However, it’s also possible to simply run __import__(modulename) ourselves if we have more control over our environment.</p><p>Another item to keep in mind is the path from which code executes. There may be a need to sys.path.append() a directory in order to access the module we’re looking for. My use case is to execute from the command line and within a directory that’s in the path of the module being examined, so I added the current directory to sys.path, which was enough to take care of the typical import path issues.</p><p>Keeping this in mind, our import function would look something like this:</p><pre>def generatedocs(module):<br>    try:<br>        sys.path.append(os.getcwd())</pre><pre>        # Attempt import<br>        mod = safeimport(module)</pre><pre>        if mod is None:<br>           print(&quot;Module not found&quot;)<br>        <br>        # Module imported correctly, let&#39;s create the docs<br>        return getmarkdown(mod)</pre><pre>    except ErrorDuringImport as e:<br>        print(&quot;Error while trying to import &quot; + module)</pre><h4>Deciding on the output</h4><p>Before continuing on, you’ll want to have a mental picture of how to organize the markdown output being generated. Do you want a shallow reference that does not recurse into custom classes? Which methods do we want to include? What about built-ins? Or _ and __ methods? How should we present function signatures? Should we pull annotations?</p><p>My choices were as follows:</p><ul><li>One .md file per run with information generated from recursing into any child classes of the object being inspecting.</li><li>Only include custom code that I’ve created, nothing from imported modules.</li><li>The output must be identified with 2nd level markdown headers (##) per item.</li><li>All headers must include the full path of the item being described (module.class.subclass.function)</li><li>Include the full function signature as pre-formatted text.</li><li>Provide anchors for each header to easily link into the docs, as well as within the docs themselves.</li><li>Any function that starts with _ or __ is not intended to be documented.</li></ul><h4>Putting it all together</h4><p>Once the object is imported, we can get to work inspecting it. It’s simply a matter of performing repeated calls to getmembers(object, filter) where the filter is one of the helper is functions. You’ll be able to get pretty far with isclass and isfunction, but the other relevant ones will be ismethod, isgenerator and iscoroutine. It all depends on whether you want to write something generic that can handle all the special cases, or something smaller and more specific the source code. I stuck with the first two because I knew I didn’t have anything else to worry about and split things into three methods to create the formatting I wanted for modules, classes and functions.</p><pre>def getmarkdown(module):<br>    output = [ module_header ]<br>    output.extend(getfunctions(module)<br>    output.append(&quot;***\n&quot;)</pre><pre>    output.extend(getclasses(module))</pre><pre>    return &quot;&quot;.join(output)</pre><pre>def getclasses(item):<br>    output = list()<br>    for cl in inspect.getmembers(item, inspect.isclass):<br>        if cl[0] != &quot;__class__&quot; and not cl[0].startswith(&quot;_&quot;):<br>            # Consider anything that starts with _ private<br>            # and don&#39;t document it</pre><pre>            output.append( class_header )<br>            output.append(cl[0])   </pre><pre>            # Get the docstring<br>            output.append(inspect.getdoc(cl[1])</pre><pre>            # Get the functions<br>            output.extend(getfunctions(cl[1]))</pre><pre>            # Recurse into any subclasses<br>            output.extend(getclasses(cl[1])</pre><pre>    return output</pre><pre>def getfunctions(item):<br>    for func in inspect.getmembers(item, inspect.isfunction):<br>        output.append( function_header )</pre><pre>        output.append(func[0])</pre><pre>        # Get the signature<br>        output.append(&quot;\n```python\n)<br>        output.append(func[0])<br>        output.append(str(inspect.signature(func[1]))</pre><pre>        # Get the docstring<br>        output.append(inspect.getdoc(func[1])</pre><pre>    return output</pre><p>When formatting a large body of text and interweaving with some programmatic code, I tend to prefer to stick it all as separate items in a list or tuple and &quot;&quot;.join() the output to put it all together. At the time of this writing, it’s actually faster than .format and % based interpolation. However, the new string formatting coming with python 3.6 will be faster than this and more readable.</p><p>As you can see, getmembers() returns a tuple with the name of the object in the first position and the actual object in the second, which we can then use to recurse through the object hierarchy.</p><p>For each one of the items retrieved, it’s then possible to pull docstrings and comments with getdoc() or getcomments(). For every function we can use signature() to get a Signature object that represents its positional and keyword arguments, their default values and any annotations, providing us with the flexibility of generating fairly descriptive and well styled text that helps our users understand the intent of the code we’ve written.</p><h4>Other considerations and unintended consequences</h4><p>Please note that the code above is just sample code only meant to give you an idea of what the final product should look like. There are quite a few more considerations to keep in mind before finalizing things:</p><ul><li>As is, getfunctions and getclasses will show <strong>ALL</strong> functions and classes <em>imported</em> in the module. This includes builtins, as well as anything from external packages, so you’ll have to filter that for-loop some more. I wound up using the __file__ property of the module that contains whatever item I’m inspecting. In other words, if the item is defined in a module that exists within the path in which I’m executing, then include it (use os.path.commonprefix())</li><li>There are some gotcha’s with file paths, import hierarchy and names. Like when you import moduleX into a package through __init__.py, you’ll be able to access its functions as package.moduleX.function, but the full name — the one returned by moduleX.__name__ — will be package.moduleX.moduleX.function. You may not care, but I did, so it’s something to keep in mind when iterating through things.</li><li>You’ll import classes from builtins and anything from there does not have __file__, so check for that if you put any filtering in place like described above.</li><li>Because this is markdown and because we’re simply importing docstrings, you can include markdown in your docstrings and it will appear in the page all nice and pretty. However, this means that you should take care to properly escape the docstring so it doesn’t break the HTML generated.</li></ul><h4>Sample output</h4><p>I ran my generator over the sofi package — the sofi.app module to be precise — and here’s the markdown it created.</p><pre># sofi&lt;a name=&quot;sofi&quot;&gt;&lt;/a&gt;<br>&lt;a name=&quot;sofi.__init__&quot;&gt;&lt;/a&gt;<br>### [sofi](#sofi).\_\_init\_\_</pre><pre>```python<br>__init__(self)<br>```</pre><pre>&lt;a name=&quot;sofi.addclass&quot;&gt;&lt;/a&gt;<br>### [sofi](#sofi).addclass</pre><pre>```python<br>addclass(self, selector, cl)<br>```<br>Add the given class from all elements matching this selector.</pre><p>Below is a sample of the final product (excluding function annotations) after running it through mkdocs to produce a readthedocs themed page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y1cT7FhQpijK_wVhFuHNsw.png" /></figure><p>As I’m sure you already know, using these mechanisms for automatically generating documentation leads to complete, accurate and up-to-date information on your modules that’s easy to maintain and edit as you write the code, instead of after the fact. I strongly encourage everyone to give it a shot.</p><p>Before closing, I’d like to take a step back and mention that mkdocs is not the only documentation package out there. There are other well known and widely used systems like <em>Sphinx</em> (which mkdocs is based on) and <em>Doxygen</em>, both of which already do what we discussed here. However, as always, I wanted to go through the exercise in order to learn more about Python internals and the tools that come with it.</p><figure><a href="https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fpublish.twitter.com%2F&amp;ref_src=twsrc%5Etfw&amp;region=follow_link&amp;screen_name=tryexceptpass&amp;tw_p=followbutton"><img alt="" src="https://cdn-images-1.medium.com/max/181/1*_04PhhMqHB3qXQnOQNP5ag.png" /></a></figure><p>If you liked this article and want to keep up with what I’m working on, please recommend it, visit <a href="http://tryexceptpass.org/">tryexecptpass.org</a> for more topics and <a href="https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fpublish.twitter.com%2F&amp;ref_src=twsrc%5Etfw&amp;region=follow_link&amp;screen_name=tryexceptpass&amp;tw_p=followbutton">follow me on Twitter</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c85d5aa5a48" width="1" height="1" alt=""><hr><p><a href="https://medium.com/python-pandemonium/python-introspection-with-the-inspect-module-2c85d5aa5a48">How to write your own Python documentation generator</a> was originally published in <a href="https://medium.com/python-pandemonium">Python Pandemonium</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[On Democratizing the Next Generation of User Interfaces]]></title>
            <link>https://medium.com/@tryexceptpass/on-democratizing-the-next-generation-of-user-interfaces-dfc938b85be8?source=rss-138c0eb26be5------2</link>
            <guid isPermaLink="false">https://medium.com/p/dfc938b85be8</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[virtual-reality]]></category>
            <category><![CDATA[augmented-reality]]></category>
            <category><![CDATA[user-interface]]></category>
            <dc:creator><![CDATA[Cristian Medina]]></dc:creator>
            <pubDate>Wed, 21 Sep 2016 04:30:49 GMT</pubDate>
            <atom:updated>2018-04-01T03:47:44.655Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dTqY86ULQxTOLWwBGji21Q.jpeg" /></figure><h4>Musings of a dev trying to plan for the future</h4><p>I remember making a program come alive back in the days of the IBM System 23. It was my first endeavor into programming. I must have been 7 years old and already using terminals for data entry to help my dad out in his business. I wrote code that highlighted items on the screen in different shades. For the younger me, it was an achievement to be proud of, even though color was not yet available in these monitors.</p><p>Fast forward a few decades — yes, only a few — and we’re in a completely different universe. One where 3D gaming with thousands of people at the same time is possible. A space that collides with realistic physics, represented by lifelike graphics and beautiful scenery.</p><p>While a gigantic leap from the olden days of monochrome monitors and ASCII graphics. From a human’s perspective, there’s still a layer of separation between reality and the digital world. But we’re about ready to break through that barrier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E6Vsx36PMwUJuIFnicfH9g.jpeg" /><figcaption>A scene developed with Cryengine</figcaption></figure><p>On the one hand, virtual reality is only a few steps away from fully immersive experiences. Able to fool your senses into thinking you’re inside an alternate universe. On the other, augmented reality is on its way to bringing that alternate universe into our own. Rendering it on top of how we perceive it.</p><p>But you may already know this, so why am I stating the obvious? Why does it matter? I happen to think that VR and AR are it. The final visual, audio and tactile user interface frontier. Regardless of how we render it or what hardware we use to experience it. Whether wearing head gear or plugged into your neural cortex, soon we’ll fully experience these worlds.</p><p>As a developer that produces user interfaces on a regular basis — one that dislikes that process — I should sit back and reflect on how this future is going to look like. What are the implications for those of us that have to write the code? How can we plan for it?</p><h4>The current state of things</h4><p>While the existing toolset has come a long way, its main purpose is building games, not user interfaces. Even with the work that folks at <a href="https://www.unity3d.com/">Unity</a> and <a href="https://www.unrealengine.com/">Epic Games</a> have done to make it easier, most of the effort is still at a low level. This takes time away from designing better user experiences.</p><p>This means we have to start thinking about general purpose interfaces. While there are several asset libraries available, there’s no reusable standard “widget” library. Nothing that’s designed for generic user interactions, like say a button click. As simplistic as that may sound, it’s one of the basic building blocks we’re lacking. If your aim is to build a complex interface, you have to start from scratch every time.</p><p>Let’s look at the famous Iron Man interface as an example. As a software developer, how would you build that? How far are we from being able to do so? The second question is easier to answer than the first: we can do it right now with the tech we have.</p><p>To answer the first, current tools and supporting software can help create a world in which to spawn such interface. But it’s not quite a platform for interaction with the rest of your software. It’s not applying standard intuitive usage metaphors that everyone understands. It also requires skills that go quite a bit beyond writing software.</p><p>Immersive designs need a whole other level of artistry including modeling, animation, story telling, music composition, and more. To complicate things more, providing fantastic interactions means nothing if we can’t invoke or integrate with the outside world of mundane software.</p><p>Imagine a web page enabled to interact with your 3D browser app. What would you spawn in 3D space to better convey the data that you’d like your user to see? You could go to Medium and find posts that show you animations of the stories they are telling. What about video streaming with an extra visual component where the audience can see and interact with each other as if they were sitting in the same room. Even wikipedia could benefit from spawning a representation of the concept it’s describing and allowing you to interact with it.</p><h4>Possible improvements</h4><p>With this focus in mind, let’s step back and look at the environment of engines and assets we have today. The different languages, scripting backends, graphic formats, shading systems, lighting mechanisms and rendering pipelines. To achieve ubiquity, we need either standard interface methods or extensible abstraction layers. Likely both if we want to make a true impact.</p><p>I want a library, usable from any programming language, that lets me communicate with a pre-configured 3D environment. One that allows dynamic positioning of standard assets in 3D space. Able to load custom assets when needed. With hooks into rendering for VR and AR hardware, as well as any available gesture recognition metaphors. Something that forms a basis for building common reusable patterns that everyone else can enjoy.</p><p>The communications backend I implemented with WebSockets while working on <a href="https://medium.com/@tryexceptpass/a-python-ate-my-gui-part-3-implementation-39fc105b6d81">Sofi</a> could be a decent start. We can use it to command game engines (like Unity3D) and receive events, putting logic at the Python layer. Imagine being able to leverage the entire Python ecosystem at the backend with a 3D frontend.</p><p>The pieces exist, so let’s have a conversation about the type of interfaces we’ll need and the common assets we could provide.</p><p>If you have an opinion on the subject or any ideas on where we could take this, feel free to leave it in the comments below, or hop over to gitter and let’s chat about it: <a href="https://gitter.im/try-except-pass/future-interfaces">https://gitter.im/try-except-pass/future-interfaces</a></p><figure><a href="https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fpublish.twitter.com%2F&amp;ref_src=twsrc%5Etfw&amp;region=follow_link&amp;screen_name=tryexceptpass&amp;tw_p=followbutton"><img alt="" src="https://cdn-images-1.medium.com/max/181/1*_04PhhMqHB3qXQnOQNP5ag.png" /></a></figure><p>If you liked this article and want to keep up with what I’m working on, please recommend it, visit <a href="http://tryexceptpass.org/">tryexecptpass.org</a> for more topics and <a href="https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fpublish.twitter.com%2F&amp;ref_src=twsrc%5Etfw&amp;region=follow_link&amp;screen_name=tryexceptpass&amp;tw_p=followbutton">follow me on Twitter</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dfc938b85be8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to hack together a graphical Python debugger]]></title>
            <link>https://medium.com/free-code-camp/hacking-together-a-simple-graphical-python-debugger-efe7e6b1f9a8?source=rss-138c0eb26be5------2</link>
            <guid isPermaLink="false">https://medium.com/p/efe7e6b1f9a8</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[tutorial]]></category>
            <dc:creator><![CDATA[Cristian Medina]]></dc:creator>
            <pubDate>Thu, 01 Sep 2016 01:46:38 GMT</pubDate>
            <atom:updated>2018-04-01T03:49:32.798Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aOUY9bnxHsfHrZgyDz8_XA.jpeg" /></figure><h4>Zero-to-Debugging in 15 mins</h4><p>You don’t realize the value of a debugger until you’re stuck working on a hard-to-visualize problem. But once you fire up a development environment with decent debugging capabilities, you’ll never look back.</p><p>Want to know where you’re at in code execution? What’s taking so long? Just pause it and check.</p><p>Wonder what value is assigned to that variable? Mouse over it.</p><p>Want to skip a bunch of code and continue running from a different section? Go for it.</p><p>Sometimes print(variable_name) is just not enough to give you an idea of what’s going on with your project. This is when a good debugger can help you figuring things out.</p><p>Python already gives you a built-in debugger in the form of <em>pdb </em>(a command line tool). But thanks to Python’s awesome community, there are a more options that feature graphical interfaces. And there are a ton of Integrated Developer Environments (IDEs) that work with Python, such as <a href="https://www.jetbrains.com/pycharm/">JetBrain’s PyCharm</a>, <a href="https://wingware.com/">Wingare’s WingIDE</a>, and even <a href="https://beta.visualstudio.com/vs/community/">Microsoft’s Visual Studio Community</a>.</p><p>But you’re not here to hear how one debugger is better than another, or which one is prettier, or more elegant. You’re here to learn how simple it is to write a python debugger that steps through your code. That gives you a glimpse into Python’s internals.</p><p>I’m going to show you how you can build one, and in doing so scratch an itch I’ve had for a long time.</p><p>Now let’s get to it.</p><h3>A quick primer on how Python code is organized and processed</h3><p>Contrary to popular belief, Python is actually a compiled language. When you execute code, your module is run through a compiler that spits out <em>bytecode </em>which is cached as <em>.pyc</em> or <em>__pycache__</em> files. The bytecode itself is what later is executed line by line.</p><p>In fact, the actual CPython code that runs a program is nothing more than a gigantic switch case statement running in a loop. It’s an if-else statement that looks at an instruction’s bytecode, then dispositions it based on what that operation is intended to do.</p><p>The executable bytecode instructions are internally referenced as<strong><em> </em></strong><em>code objects</em>, and the <em>dis </em>and <em>inspect </em>modules are used to produce or interpret them. These are immutable structures, that although referenced by other objects — like functions — do not contain any references themselves.</p><p>You can easily look at the bytecode that represents any given source through dis.dis(). Just give it a try with a random function or class. It’s a neat little exercise that’ll help you visualize what’s going on. The output will look something like this:</p><pre>&gt;&gt;&gt; def sample(a, b):<br>...     x = a + b<br>...     y = x * 2<br>...     print(&#39;Sample: &#39; + str(y))<br>...<br>&gt;&gt;&gt; import dis<br>&gt;&gt;&gt; dis.dis(sample)<br>2       0 LOAD_FAST                0 (a)<br>        3 LOAD_FAST                1 (b)<br>        6 BINARY_ADD<br>        7 STORE_FAST               2 (x)</pre><pre>3      10 LOAD_FAST                2 (x)<br>       13 LOAD_CONST               1 (2)<br>       16 BINARY_MULTIPLY<br>       17 STORE_FAST               3 (y)</pre><pre>4      20 LOAD_GLOBAL              0 (print)<br>       23 LOAD_CONST               2 (&#39;Sample: &#39;)<br>       26 LOAD_GLOBAL              1 (str)<br>       29 LOAD_FAST                3 (y)<br>       32 CALL_FUNCTION            1 (1 positional, 0 keyword pair)<br>       35 BINARY_ADD<br>       36 CALL_FUNCTION            1 (1 positional, 0 keyword pair)<br>       39 POP_TOP<br>       40 LOAD_CONST               0 (None)<br>       43 RETURN_VALUE</pre><p>Notice that each line in bytecode references its respective position in source code on the left column, and that it’s not a one-to-one relationship. There could be multiple smaller — one could even say atomic — operations that makeup a higher level instruction.</p><p>A <em>frame object</em> in python is what represents an execution frame. It contains a reference to the code object that’s currently executing, the local variables that it’s running with, the global names (variables) that are available and references to any related frames (like the parent that spawned it).</p><p>There are lot more details about these objects to discuss here, but hopefully this is enough to wet your appetite. You won’t need much more for the purposes of our debugger, though you should check out the Diving Deeper section for links on where to look next.</p><h3>Enter the sys module</h3><p>Python provides a number of utilities in its standard library through the <em>sys </em>module. Not only are there things like <em>sys.path </em>to get the python path or <em>sys.platform </em>to help find details about the OS in which you are running, but there’s also sys.settrace() and sys.setprofile() to help write language tools.</p><p>Yes, you read that right. Python already has built-in hooks to help analyze code and interact with program execution. The sys.settrace() function will allow you to run a callback whenever execution advances to a new frame object and gives us a reference to it, which in turn provides the code object you’re working with.</p><p>For a quick example of how this looks, let’s reuse the function from earlier:</p><pre>def sample(a, b):<br>    x = a + b<br>    y = x * 2<br>    print(&#39;Sample: &#39; + str(y))</pre><p>Assuming that every time a new frame is executed, you want a callback that prints the code object and line number its executing, you can define it as:</p><pre>def trace_calls(frame, event, arg):<br>    if frame.f_code.co_name == &quot;sample&quot;:<br>        print(frame.f_code)</pre><p>Now it’s simply a matter of setting it as our trace callback:</p><pre>sys.settrace(trace_calls)</pre><p>And executing <em>sample(3,2)</em> should produce</p><pre>$ python debugger.py<br>&lt;code object sample at 0x0000000000B46C90, file “.\test.py”, line 123&gt;<br>Sample: 10</pre><p>You need the if-statement to filter out function calls. Otherwise you’ll see a whole bunch of things that you don’t care about, especially when printing to the screen. Try it.</p><p>The code and frame objects have quite a few fields to describe what they represent. These include things like the file being executed, the function, variable names, arguments, line numbers, and the list goes on. They are fundamental to the execution of any python code and you can go through the language documentation for more details.</p><h3>What if you want to debug every line?</h3><p>The trace mechanism will set subsequent callbacks depending on the return value of the first callback. Returning <em>None</em> means that you’re finished, while returning another function effectively sets it as the trace function inside that frame.</p><p>Here’s what this looks like:</p><pre>5    def sample(a, b):<br>6        x = a + b<br>7        y = x * 2<br>8        print(&#39;Sample: &#39; + str(y))<br>9<br>10   def trace_calls(frame, event, arg):<br>11       if frame.f_code.co_name == &quot;sample&quot;:<br>12           print(frame.f_code)<br>13           return trace_lines<br>14       return<br>15<br>16   def trace_lines(frame, event, arg):<br>17       print(frame.f_lineno)</pre><p>Now, if you execute the same code as before, you can see it print the line numbers as you progress through it:</p><pre>$ python .\test.py<br>&lt;code object sample at 0x00000000006D4DB0, file &quot;.\test.py&quot;, line 5&gt;<br>6<br>7<br>8<br>Sample: 10<br>8</pre><h3>Putting a user interface in front of it</h3><p>Using the <a href="https://github.com/tryexceptpass/sofi"><em>sofi</em></a><em> </em>python module, you can easily produce a web application that directly interacts with our python code.</p><p>Here’s what you would do:</p><ol><li>Show the file, function name and line number being executed.</li><li>Show the code for the current frame with a pointer identifying the line.</li><li>Show the value of the local variables.</li><li>Provide step-by-step execution, meaning you have to block before executing a line until the user clicks a button.</li><li>Add step-over functionality.</li><li>Add a step-out mechanism.</li><li>Provide a method of stopping execution.</li></ol><p>From the UI perspective, #1, #2 and #3 can all be handled through a Bootstrap <em>Panel</em> where #1 is the title, and #2 and #3 are part of the body wrapped in <em>samp </em>tags to show proper spacing.</p><p>Since the interface will essentially block waiting for user input, and the debugger waits for stop / go commands, it’s a good idea to separate those event loops using our old friend <em>multiprocessing</em>. You can then implement one <em>queue</em> to send debug commands to one process, and a different application queue for UI updates in the other.</p><p>Through multiprocessing queues, it’s easy to block the debugger waiting for user commands at the <em>trace_lines</em> function using the<em> .get()</em> method.</p><p>If the command is given to move to the next line of code (#4), everything stays the same, while stepping out (#6) will change the return value back to the <em>trace_calls </em>function — effectively removing further calls into <em>trace_lines —</em> and stop (#7) will raise a custom exception that will abort execution.</p><pre># Block until you receive a debug command<br>cmd = trace_lines.debugq.get()</pre><pre>if cmd == &#39;step&#39;:<br>    # continue stepping through lines, return this callback<br>    return trace_lines</pre><pre>elif cmd == &#39;stop&#39;:<br>    # Stop execution<br>    raise StopExecution()</pre><pre>elif cmd == &#39;over&#39;:<br>    # step out or over code, so point to trace_calls<br>    return trace_calls</pre><pre>class StopExecution(Exception):<br>    &quot;&quot;&quot;Custom exception used to abort code execution&quot;&quot;&quot;</pre><pre>    pass</pre><p>Step-over functionality (#5) is implemented at the <em>trace_calls </em>level by never returning the trace_lines callback.</p><pre>cmd = trace_lines.debugq.get()</pre><pre>if cmd == &#39;step&#39;:<br>    return trace_lines</pre><pre>elif cmd == &#39;over&#39;:<br>    return</pre><p>Yes, I attached the queue objects as properties of the trace functions to simplify passing things around. Functions being objects is a great idea, though you shouldn’t abuse it either.</p><p>Now it’s just a matter of setting up the widgets for displaying data and the buttons for controlling flow.</p><p>You can pull out the source code from the code object of the executing frame using the inspect module.</p><pre>source = inspect.getsourcelines(frame.f_code)[0]</pre><p>Now it’s a matter of formatting it line by line into <em>div </em>and <em>samp </em>tags, adding an indicator of a different color to the current line (available through f_lineno<em> </em>and co_firstline) and sticking that into a <em>panel</em> widget’s body, along with the string representation of the frame’s locals (which is a simple dictionary anyway):</p><pre>def formatsource(source, firstline, currentline):<br>    for index, item  in enumerate(source):<br>        # Create a div for each line to better control format<br>        div = Div()</pre><pre>        # Extremly simplified tab index check to add blank space<br>        if item[0:1] == &#39;\t&#39; or item[0:1] == &#39; &#39;:<br>            div.style =&#39;margin-left:15px;&#39;</pre><pre>        # If this currently executing this line, add a red mark<br>        if index == lineno - firstlineno:<br>            div.addelement(Bold(&#39;&gt; &#39;, style=&quot;color:red&quot;))</pre><pre>        # Add the formatted code to the div<br>        div.addelement(Sample(item.replace(&quot;\n&quot;, &quot;&quot;)))</pre><pre>        # Output the html that represents that div<br>        source[index] = str(div)</pre><pre>    return &quot;&quot;.join(source)</pre><p>Only thing left to do is register a few event callbacks for button clicks that control execution flow by adding their respective commands to the debug queue. You do this inside a <em>load </em>event handler which triggers after the initial content finishes loading</p><pre>@asyncio.coroutine<br>def load(event):<br>    &quot;&quot;&quot;Called when the initial html finishes loading&quot;&quot;&quot;</pre><pre>    # Start the debug process<br>    debugprocess.start()</pre><pre>    # Register click functions<br>    app.register(&#39;click&#39;, step, selector=&quot;#code-next-button&quot;)<br>    app.register(&#39;click&#39;, stop, selector=&quot;#code-stop-button&quot;)<br>    app.register(&#39;click&#39;, over, selector=&quot;#code-over-button&quot;)</pre><pre>    # Make sure the display updates<br>    yield from display()</pre><pre>@asyncio.coroutine<br>def step(event):<br>    debugq.put(&quot;step&quot;)</pre><pre>    # Make sure the display updates<br>    yield from display()</pre><pre>@asyncio.coroutine<br>def stop(event):<br>    debugq.put(&quot;stop&quot;)</pre><pre>@asyncio.coroutine<br>def over(event):<br>    debugq.put(&quot;over&quot;)</pre><p>How would this look?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/1*oV3Gp3r2-5XjhrnV2nvKcw.png" /></figure><p>For a view of all of the code put together, check out the sofi-debugger project on GitHub:</p><p><a href="https://github.com/tryexceptpass/sofi-debugger">tryexceptpass/sofi-debugger</a></p><h4>Some notes on what you just did</h4><p>The functions from the <em>sys </em>module mentioned here are implemented in CPython and may not be available in other flavors or interpreters. Make sure to keep this in mind when experimenting.</p><p>They are also specifically meant for use with debuggers, profilers or similar trace tools. This means that you should not be messing with them as part of a normal program or you may come across some unintended consequences, especially when interacting with other modules that may specifically target these same interfaces (like actual debuggers).</p><h3>Diving Deeper</h3><p>For a deeper dive into the Python language constructs, frames, code objects and the dis module, I emphatically recommend that you set aside some time and go through Phillip Guo’s (@pgbovine) CPython Internals lectures.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FLhadeL7_EIU%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DLhadeL7_EIU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FLhadeL7_EIU%2Fhqdefault.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/74773760e58d0ee7641e7b8d1c372b4d/href">https://medium.com/media/74773760e58d0ee7641e7b8d1c372b4d/href</a></iframe><figure><a href="https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fpublish.twitter.com%2F&amp;ref_src=twsrc%5Etfw&amp;region=follow_link&amp;screen_name=tryexceptpass&amp;tw_p=followbutton"><img alt="" src="https://cdn-images-1.medium.com/max/181/1*_04PhhMqHB3qXQnOQNP5ag.png" /></a></figure><p>If you liked this article and want to keep up with what I’m working on, please recommend it, visit <a href="http://tryexceptpass.org/">tryexecptpass.org</a> for more topics and <a href="https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fpublish.twitter.com%2F&amp;ref_src=twsrc%5Etfw&amp;region=follow_link&amp;screen_name=tryexceptpass&amp;tw_p=followbutton">follow me on Twitter</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=efe7e6b1f9a8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/free-code-camp/hacking-together-a-simple-graphical-python-debugger-efe7e6b1f9a8">How to hack together a graphical Python debugger</a> was originally published in <a href="https://medium.com/free-code-camp">We’ve moved to freeCodeCamp.org/news</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Python ate my GUI — Part 3: Implementation]]></title>
            <link>https://medium.com/@tryexceptpass/a-python-ate-my-gui-part-3-implementation-39fc105b6d81?source=rss-138c0eb26be5------2</link>
            <guid isPermaLink="false">https://medium.com/p/39fc105b6d81</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[3d]]></category>
            <category><![CDATA[ui]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Cristian Medina]]></dc:creator>
            <pubDate>Tue, 16 Aug 2016 02:58:45 GMT</pubDate>
            <atom:updated>2018-04-01T03:37:48.410Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U9k6CgmtBRRkWTKYoPUieQ.jpeg" /><figcaption><strong>Photo credit: </strong><a href="https://www.flickr.com/photos/42311564@N00/2355590274/"><strong>Sebastian Niedlich (Grabthar)</strong></a><strong> via </strong><a href="http://foter.com/"><strong>Foter.com</strong></a></figcaption></figure><h4>Time to Bootstrap your D3, pick up that Python and hop on the Autobahn</h4><p>If you aren’t aware of my earlier posts, check out <a href="https://medium.com/@tryexceptpass/a-python-ate-my-gui-971f2326ce59#.6rfz6l5kr">Part 1</a> and <a href="https://medium.com/@tryexceptpass/a-python-ate-my-gui-part-2-5851c971d34c#.tbjqpvxfj">Part 2</a> of this series so that you get some context for this ongoing exercise.</p><p>Since I started thinking about and working on these posts, I’ve also been developing the ideas on GitHub as a side project I called sofi.</p><p><a href="https://github.com/tryexceptpass/sofi">tryexceptpass/sofi</a></p><p>Sofi is a Python 3 package that serves as the starter implementation of the design discussed in Part 2: a system that will generate the necessary HTML and JavaScript code typically needed to produce a single-page application and serve it up through WebSockets (not an HTTP server).</p><p>The webpage functions as a dumb user interface layer on top of your python code by exposing a simple command and event system that allows for communications back and forth with the python logic. The UI itself is generated using Bootstrap components, and enabled by D3.js for processing events and DOM changes, as well as for future integration with charting mechanisms.</p><h4>Why did I do this?</h4><p>I specifically wanted to understand and do more things with WebSockets. I like the technology and believe that it can enhance — possibly even replace — some of the RESTful interfaces that I build or maintain on a regular basis.</p><p>At the same time, I’ve been looking for an excuse to use <a href="https://docs.python.org/3/library/asyncio.html">asyncio</a> and some of the more recent python 3 features, and found that the <a href="http://autobahn.ws/python/">autobahn</a> websocket module is built on top of it.</p><p>On another note, as discussed in a previous post, I’m also lazy! and tired of re-implementing web interfaces in languages that I would prefer not to use.</p><p>However, most importantly, I wanted to dive a little deeper into the implications of writing your own command / event protocol to enable GUI interactions, as I intend to use something similar in a different kind of project (which is a subject for another time).</p><h4>How does it work?</h4><p>The basic concept is fairly simple: statically serve an html file that references a minimal javascript library setup so that when the page loads, it automatically connects to the websocket server, updates the page DOM with whatever is needed and registers any relevant event subscriptions.</p><p>These DOM updates or event subscriptions are received as commands through the websocket, with enough information to specify the element selectors that are then passed onto D3 to perform the changes.</p><p>Whenever an event is triggered, it gets relayed back through the same socket. This includes page events (like click) as well as any control events necessary to establish the workflow (like init, load or close).</p><p>This socket protocol gives us the flexibility to host the static html anywhere we want and run the websockets themselves in some other place, while supporting communication with any operating system and any modern web browser (including Android and iOS).</p><p>To ease development of the user interface, common Bootstrap widgets are wrapped into python classes that allow for easy compositing through basic built-ins like __str__. Every widget object provides the basic set of methods necessary for generating the html that describes it, but also exposes the identifier, class, style and attributes fields to allow for further customization.</p><p>While these widgets give you a mechanism for rapidly prototyping an interface, they are not necessary and can be replaced with a simple string representing any HTML that you’d like to send up to the web page.</p><p>In order to allow for threaded processing of incoming events, any callback functions are expected as asyncio coroutines, otherwise the code would be unresponsive as long events are being processed.</p><p>Currently, the application as a whole is executed from a shell, where python automatically opens the default browser and points it to the static html starter file, with no need to run a webserver. However, it’s totally possible to start it and separately navigate to the html file from an already running browser.</p><p>Execution ends when the browser tab is closed and an event is sent back through the websocket to be handled appropriately, though I still need to improve that code to gracefully manage disconnects and coroutines.</p><h4>What can you make?</h4><p>Anything, of course! Kidding aside though, the system is pretty open to tie into any python program. At the moment you’ll be limited by the number of widgets available to represent your interface, and things are more verbose than I’d like, but all of that will eventually settle down.</p><p>The sample.py file in the main directory is a bit of a widget showcase that dynamically updates over time. It also has a click handler configured for the Primary button which will print out a message with the entire event object, making it clear what data is available to the callbacks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/1*euug6f885sjtRPOMt_Vc6g.png" /><figcaption>Output from sample.py</figcaption></figure><p>The timeline.py file is an example of what can be done fairly quickly when tying into the greatness of the python ecosystem. It leverages the twitter api and outputs a bunch of bootstrap panels with each tweet from your main timeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*AmbFclbXWFdIRYbpa0cyBw.png" /><figcaption>Output from timeline.py</figcaption></figure><h4>What’s next?</h4><p>This is far from ready for prime time. There is still much work to be done to document, add more Bootstrap components, composite those into more useful widgets, add more commands to direct the display, adjust for less common events and input systems (i.e.: touch and gestures), add charts, add advanced table functionality, etc. However, before I get into all of that, I thought a little feedback will be helpful.</p><p>While I’m confident I can make something useful for myself, I’d like to make it at least a minimum viable product for everyone else.</p><figure><a href="https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fpublish.twitter.com%2F&amp;ref_src=twsrc%5Etfw&amp;region=follow_link&amp;screen_name=tryexceptpass&amp;tw_p=followbutton"><img alt="" src="https://cdn-images-1.medium.com/max/181/1*_04PhhMqHB3qXQnOQNP5ag.png" /></a></figure><p>If you liked this article and want to keep up with what I’m working on, please recommend it, visit <a href="http://tryexceptpass.org/">tryexecptpass.org</a> for more topics and <a href="https://twitter.com/intent/follow?original_referer=https%3A%2F%2Fpublish.twitter.com%2F&amp;ref_src=twsrc%5Etfw&amp;region=follow_link&amp;screen_name=tryexceptpass&amp;tw_p=followbutton">follow me on Twitter</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39fc105b6d81" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>