<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Bonobo - Medium]]></title>
        <description><![CDATA[Python 3.5+ ETL - Medium]]></description>
        <link>https://medium.com/bonobo-data?source=rss----294bb6371882---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Bonobo - Medium</title>
            <link>https://medium.com/bonobo-data?source=rss----294bb6371882---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 05:49:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/bonobo-data" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Bonobo ETL 0.6 was released!]]></title>
            <link>https://medium.com/bonobo-data/bonobo-etl-0-6-was-released-cbbc88303ec3?source=rss----294bb6371882---4</link>
            <guid isPermaLink="false">https://medium.com/p/cbbc88303ec3</guid>
            <category><![CDATA[release-notes]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[bonobo-etl]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Romain Dorgueil]]></dc:creator>
            <pubDate>Sun, 07 Jan 2018 16:24:38 GMT</pubDate>
            <atom:updated>2018-01-07T16:30:18.668Z</atom:updated>
            <content:encoded><![CDATA[<p>I’m very proud to announce that Bonobo 0.6 was released last week, and extensions were updated to match the major changes that went during this release.</p><p>There was a lot of structural changes in this release, and before you upgrade your projects (and you should), <a href="https://news.bonobo-project.org/migration-guide-for-bonobo-0-6-alpha-c1d36b0a9d35">please read the migration guide</a>.</p><h3>Changes</h3><h4>New features</h4><ul><li>The standard library got a few new configurable transformations.</li><li>The new bonobo.FixedWindow(width) transformation allows to pack a data stream into a stream of fixed length tuples.</li><li>The new bonobo.Format(**) transformation allows to apply renaming and formating to a stream.</li><li>The new bonobo.OrderFields(**) transformation allows to reorder the fields in a stream.</li><li>The new bonobo.Rename(**) transformation allows to apply a fieldset renaming operation to a stream.</li><li>The new bonobo.SetFields(fields) transformation allows to name the fields of a stream of tuples, or just override the current names.</li><li>The new bonobo.UnpackItems() can extract values from dictionaries in tuples and move them at the top level in a stream. For example, if you have a stream containing ({&#39;foo&#39;: &#39;bar&#39;}, anything, {&#39;bar&#39;: &#39;baz&#39;}, applying bonobo.UnpackItems(0, 2) onto it will result in a stream of (&#39;bar&#39;, &#39;baz&#39;), and the field names will be set to [&#39;foo&#39;, &#39;bar&#39;].</li><li>The advertised way to write Bonobo jobs is now to use the regular python’s if __name__ == &#39;__main__&#39;: block, and arguments are parsed using a bonobo provided argparse.ArgumentParser instance. You can, of course, extend this argument parser with your own logic.</li><li>Environment management (as in, system environment) is handled by this argument parser, and the bonobo.parse_args(...) context manager is in charge of patching/restoring the environment according to the given command lines arguments. You can now use “dotenv” files (using python myjob.py --env-file .env) or override some env from command line (using python myjob.py --env FOO=bar).</li><li>The new LdjsonReader(...) and LdjsonWriter(...) transformations are available to work with line-delimited JSON files, which are much more suitable for efficient stream processing than regular JSON files.</li><li>The CsvReader(...) and CsvWriter(...) transformations now supports all configuration options supported by python’s csv.reader and csv.writer.</li><li>The bonobo convert command now supports arbitrary options for reader and writer.</li></ul><h4>Core changes</h4><ul><li>Bonobo now uses tuples, or a variant of namedtuple, to pass data between nodes. For “positional” data (a.k.a data with fields refered to by position, and not by name), a simple tuple does the job, and bonobo will ensure that the shape (length) does not vary from one to another line. For “named” data (a.k.a data refered to by symbolic names), it will use a custom type, based on the python standard namedtuple code (but rewritten to handle any names), and also ensure that the shape does not change. For mixed data, our implementation of namedtuple handle it just right, making data accessible both by position and name.</li><li>The bonobo init generator changed and no longer use cookiecutter to generate projects. The default is now to generate a single file, using jinja2 templates, and it can also be used to generate a package, using medikit (which is already used to manage bonobo and its extensions). By default, bonobo init somejob.py will be very verbose, with comments, etc., but you can use bonobo init -t bare somejob.py to avoid that.</li><li>Simplification of node execution context’s code.</li><li>Logging was refactored into an external new library, called <a href="https://pypi.python.org/pypi/mondrian">Mondrian</a>, and all code related to logging configuration was moved from Bonobo.</li><li>Plugins architecture changed into something event based, using <a href="https://pypi.python.org/pypi/whistle">Whistle</a>. Plugins does not run anymore in their own thread, but in the main thread, which solve the problem of having the plugins work in non-thread based environments.</li></ul><h4>Extensions and integrations</h4><ul><li>Jupyter integration got a revamp, displaying a graph will show it graphically (if graphviz is available), and the execution widget is now much nicer and readable.</li><li>Jupyter integration was moved to the bonobo.contrib.jupyter module.</li><li>OpenDataSoft integration was moved to the bonobo.contrib.opendatasoft module.</li><li>The new bonobo.contrib.django module contains helpers to write data processing tasks within the django framework (this is subject to future changes and should be considered alpha).</li><li>The new bonobo.contrib.google module contains helpers to connect with google’s OAuth APIs, to read/write from google services (this is subject to future changes and should be considered alpha).</li></ul><h4>Minor changes, fixes and enhancements</h4><ul><li>Generalize tuplize decorator into a cast(...) decorator that can cast the result of a generator into anything.</li><li>The Bonobo API management got refactored, both to make it more “solid” and to provide helpers for extensions to manage their APIs the same way. Both docker and sqlalchemy extensions has been updated to use it.</li><li>Execution contexts now check partially configured transformations that are function based (aka transformation factories), and raise an intelligible error if the transformation is not “final”.</li><li>PrettyPrinter’s output is now more discrete and much more readable.</li><li>Services are now passed to the “naive” execution strategy.</li></ul><h3>Thanks</h3><p>That’s all! A very big thanks to all contributors, and to all contributors that helped with this release. It includes code contributors (Ari Bajo, CW Andrews, Kenneth Koski, Michael Penkov, Mouadh Kaabachi, Stefan Zimmermann), documentation contributors (Daniel Jilg, Peter Uebele), the Pycon.DE sprinters (Jassin Roderigo, Kai Krajewski, Maik Röder, Michael Penkov, Peter Uebele, Stefan Zimmermann) and all persons that brought their thoughts and brains in all the amazing discussions we had both on <a href="https://bonobo-slack.herokuapp.com/">Slack</a> and offline.</p><p>Special thanks goes to Maik Röder for his insights on «not turning python into javascript», which is probably the reason for most of the simplifications that happened in this version, and to Tobias Macey for <a href="https://www.podcastinit.com/bonobo-with-romain-dorgueil-episode-143/">his invitation to talk about Bonobo on Podcast __init__</a>.</p><p>I hope I did not forget anybody (although I’m sure I did), and be assured that it is unwilling from me if I did.</p><p>Please share the love on twitter, and don’t forget to clap! <strong>Bonobo loves claps!</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xa21JDFcjr372xwo8DHGGQ.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cbbc88303ec3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bonobo-data/bonobo-etl-0-6-was-released-cbbc88303ec3">Bonobo ETL 0.6 was released!</a> was originally published in <a href="https://medium.com/bonobo-data">Bonobo</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migration Guide for Bonobo 0.6]]></title>
            <link>https://medium.com/bonobo-data/migration-guide-for-bonobo-0-6-alpha-c1d36b0a9d35?source=rss----294bb6371882---4</link>
            <guid isPermaLink="false">https://medium.com/p/c1d36b0a9d35</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[bonobo]]></category>
            <category><![CDATA[release-notes]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[etl]]></category>
            <dc:creator><![CDATA[Romain Dorgueil]]></dc:creator>
            <pubDate>Sun, 03 Dec 2017 07:28:42 GMT</pubDate>
            <atom:updated>2018-01-07T15:25:44.847Z</atom:updated>
            <content:encoded><![CDATA[<p>Bonobo 0.6 was released and a lot of changes happened so the migration process may be a little less smooth than the previous versions. This article should give you a good understanding on all major changes and guide you through migrating your code to the new version.</p><p>To upgrade your installation, use:</p><p>$ pip install --upgrade &quot;bonobo ~=0.6.0&quot;</p><h3>Migration Overview</h3><p>The overall idea with 0.6 was to remove magic, be more pythonic and solidify the exiting foundations. That’s a bit conceptual, but that’s recurring themes you’ll see again and again in the following sections.</p><h3>Packaging, Init and Templates</h3><ul><li>The cookiecutter template was removed. Having “pseudo-python-packages” was not a good idea, and raised more questions than solutions.</li><li>The bonobo run CLI now behaves the exact same way as executing a bonobo job directly with the python interpreter, and your job files just need a small update for this to work, removing the graph instance detection. It’s advised, unless you need something only the CLI can bring, to run your jobs using the python interpreter (either python foo.py or python -m mypkg.pyjob).</li><li>The bonobo init CLI now create files from templates, instead of directories with a bunch of files. Creating just one file that can be executed using the python interpreter makes it much easier to understand what happens, and much easier for pythonistas to jump in Bonobo. Here is the new layout of a Bonobo job:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/afa11758515ad3e0acb189914edd0a7f/href">https://medium.com/media/afa11758515ad3e0acb189914edd0a7f/href</a></iframe><p>The with parse_args(...) block makes sure that the standard arguments (like --env) will work the same using bonobo CLI and standard python interpreter.</p><p><strong><em>[TL;DR]</em></strong> The general idea is now to start working in a single file, and migrate this file in a package when it’s necessary. Also, the direct execution is now defined like in standard python, using __name__ == &quot;__main__&quot; blocks.</p><h4>How to update your code?</h4><ul><li>You must now define your graph in a function, and call it to pass it to bonobo.run(...). There is no “graph instance” detection anymore, and calling this function within the parse_args block makes an eventual custom environment available.</li><li>If you were using a separate _services.py file for your service definitions, you’ll need to import it.</li><li>The import mechanism will work the same as regular python, even if you’re using bonobo run. If you need relative imports (from . import foo) or importing modules that clash with your site-package namespace (like, a csv.py file), then you’ll need to create a real python package. You can do that the way you want and it’s out of the bonobo scope, but bonobo init can help you, using <a href="https://github.com/python-medikit/medikit">medikit</a>. Try to run bonobo init --package myetl.</li></ul><h3>Bags, Inputs and Outputs</h3><p>Both as a simplification and a restriction of what’s possible, it’s not possible anymore to change the shape (length, and field names) of a node’s output during the execution.</p><p>It means that for a given node, you decide the shape of the output once, and that will be enforced for its life length. If you don’t enforce it, then it will be detected from the first output line.</p><p>All outputs are now tuple-shaped. If an output row is not a tuple (or a subclass of tuple), then it will be forced as a one-item tuple.</p><p>yield &quot;foo&quot; is equivalent to yield (&quot;foo&quot;, ).</p><p>It is less flexible (small con for edge cases, that can be worked out differently), but brings a few nice values:</p><ul><li>You now need to be explicit about your data’s shape. If for some reason one row misses a key, there can’t be missing-offset errors anymore, as the framework will raise an error.</li><li>It allowed to delete the whole Bag() implementation which started to be complicated, for a much more standard way to handle python arguments.</li><li>Field ordering work the same in all python 3.5+ versions, as the implementation does not rely anymore on the packed dict feature of python 3.6+ to have sorted keyword arguments.</li><li>It has a (way) better memory efficience. When you’re using key-value like data, the key-space will be stored once, instead of once per row.</li><li>You are now able to define your own type of data to use as the input/output vehicle (although I’d be careful about whether that’s really something you need before going there).</li><li>The new Bag implementation is similar to the collections.namedtuple implementation from standard python library, but allows more things, including “numeric fields”, so you can both have positional and named arguments to your type.</li></ul><p><strong><em>[TL;DR]</em></strong> <strong>Bags were removed.</strong> There are still types named <strong>Bag</strong>, but they are generated dynamically (instead of defined), and just acts like named tuples.</p><h4>How to update your code?</h4><ul><li><strong>If your data is name-indexed</strong> (as in key-value / dict-like collections), then you should use context.set_output_fields([...]) on the first call.</li><li><strong>If your data is position-indexed</strong> (as in tuple / list-like collections), then you should use</li><li><strong>If you’re using empty bags</strong>, you can just replace them by empty tuples (or EMPTY, that can be found in bonobo.constants, for readability).</li></ul><h3>Configurables and Transformations</h3><ul><li>In <strong>Configurables</strong>, the __call__ dunder shortcut “call” was removed. If you wanna reimplement it, just add __call__ = myhandler (or __call__ = call) at the end of your class definition.</li><li><strong>Services</strong> instances are now passed as kwargs by default, to avoid confusion on parameter order. That’s a big BC break, but the update is simple and it makes the code less error-prone. Use the star, Luke!</li></ul><h3>New dependencies</h3><p><em>This is still subject to change (as in, some deps removed) until release if we have evidence that some dep is a bad idea.</em></p><ul><li><a href="https://github.com/un33k/python-slugify"><strong>Slugify</strong></a></li><li><a href="http://jinja.pocoo.org/"><strong>Jinja2</strong></a></li><li><a href="https://pypi.python.org/pypi/graphviz"><strong>Graphviz</strong></a><em> (python wrapper only)</em></li></ul><h3>Standard Library</h3><ul><li><strong>CsvWriter</strong> “headers” has been renamed to “fields”, for consistency with name choices in other places.</li><li><strong>CsvReader</strong> and <strong>CsvWriter</strong> now can use all options that standard csv.reader and csv.writer module know about: delimiter, quotechar, escapechar, doublequote, skipinitialspace, lineterminator and quoting. Be careful, that can mean that the default format will be slightly different, as it now uses the “excel” dialect from standard csv module.</li><li>If you need to override the default csv.reader or csv.writer instance that <strong>CsvReader</strong> and <strong>CsvWriter</strong> use, you can provide your own factory using respectively the reader_factory and writer_factory method options.</li></ul><p><strong>That’s about it!</strong></p><p>This article will be updated as new alphas are released, please make sure to <strong>send your feedback</strong>, using <a href="https://github.com/python-bonobo/bonobo/issues"><strong>Issues</strong></a> and/or <a href="https://bonobo-slack.herokuapp.com/"><strong>Slack</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3iGZYVIPpstdQc5wsaT9gw.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1d36b0a9d35" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bonobo-data/migration-guide-for-bonobo-0-6-alpha-c1d36b0a9d35">Migration Guide for Bonobo 0.6</a> was originally published in <a href="https://medium.com/bonobo-data">Bonobo</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Discover Bonobo’s logging facilities]]></title>
            <link>https://medium.com/bonobo-data/discover-bonobos-logging-facilities-befd24a7f97a?source=rss----294bb6371882---4</link>
            <guid isPermaLink="false">https://medium.com/p/befd24a7f97a</guid>
            <category><![CDATA[mondrian]]></category>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[console]]></category>
            <dc:creator><![CDATA[Romain Dorgueil]]></dc:creator>
            <pubDate>Tue, 31 Oct 2017 12:46:00 GMT</pubDate>
            <atom:updated>2017-11-06T14:50:27.261Z</atom:updated>
            <content:encoded><![CDATA[<p>From the beginning, we used Python’s standard logging module to provide logs within Bonobo. This is a logic choice, as logging is mature, flexible and very powerful.</p><p>We also wanted the logs to look nicer as a default than plain white on black, and thus we used logging’s filters and formatters to both add colors, and better summarize informations for the end user.</p><h4>Few caveats…</h4><ul><li>Logging code was deeply tied into Bonobo, although it’s only some painting.</li><li><a href="https://github.com/python-medikit/medikit#-medikit-">Medikit</a> was using tornado’s logger, which is plain wrong as it required to install the tornado web framework just to get pretty logging.</li><li>I like to manage all my projects using the same, strict interface, and I felt I needed to have a simple and lightweight package for setting up similar logging everywhere.</li></ul><h4>Meet Mondrian!</h4><p>I just repackaged all this logic in a small, dependency-less python package to be able to use the same nice-looking logs from any other library.</p><p>Let’s have a warm welcome for <a href="https://github.com/hartym/mondrian#mondrian">Mondrian</a>, named after <a href="https://en.wikipedia.org/wiki/Piet_Mondrian">the great painter</a>. I let you guess why I did choose him…</p><h4>Head start</h4><p><a href="https://github.com/hartym/mondrian#mondrian">Mondrian</a> still uses python standard logging facilities. It’s just a small wrapper that configures the display for you.</p><p>First, run pip install mondrian, then make sure your code go through the following bootstrapping code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/73065f3648b0ab049999c3c4cb2aa483/href">https://medium.com/media/73065f3648b0ab049999c3c4cb2aa483/href</a></iframe><p>Once this is done, you can continue to use standard python logging facilities as usual.</p><h4>Result</h4><p>Ok, it’s a bit rainbowish there, but keep in mind you’d mostly see INFO/WARN levels in a real world app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mK7fbpuxLrAtk4rSmJv_3Q.png" /></figure><p>Mondrian will be used in <a href="https://www.bonobo-project.org/">Bonobo ETL</a> starting from the incoming 0.6 release and the matching code will be removed from <a href="https://www.bonobo-project.org/">Bonobo</a>. It is already used in <a href="https://github.com/python-medikit/medikit">Medikit</a>.</p><p>Do you like it? Please share it on social networks, and don’t forget to clap!</p><ul><li><a href="https://github.com/hartym/mondrian">Mondrian on GitHub</a></li><li><a href="https://pypi.python.org/pypi/mondrian">Mondrian on PyPI</a></li><li><a href="https://docs.python.org/3/library/logging.html">Logging module</a></li></ul><p>Happy logging!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=befd24a7f97a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bonobo-data/discover-bonobos-logging-facilities-befd24a7f97a">Discover Bonobo’s logging facilities</a> was originally published in <a href="https://medium.com/bonobo-data">Bonobo</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>