<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Data Version Control - Medium]]></title>
        <description><![CDATA[DVC.org - organize your machine learning projects - Medium]]></description>
        <link>https://medium.com/dataversioncontrol?source=rss----4e21975b393---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Data Version Control - Medium</title>
            <link>https://medium.com/dataversioncontrol?source=rss----4e21975b393---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 01:08:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/dataversioncontrol" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[November ’19 DVC❤️Heartbeat]]></title>
            <link>https://medium.com/dataversioncontrol/november-19-dvc-%EF%B8%8Fheartbeat-110404842192?source=rss----4e21975b393---4</link>
            <guid isPermaLink="false">https://medium.com/p/110404842192</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[version-control]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Svetlana Grinchenko]]></dc:creator>
            <pubDate>Sat, 14 Dec 2019 00:59:48 GMT</pubDate>
            <atom:updated>2019-12-14T00:59:48.852Z</atom:updated>
            <content:encoded><![CDATA[<p>Every month (or two) we are sharing here our news, findings, interesting reads, community takeaways, and everything along the way.</p><p>Some of those are related to our brainchild <a href="https://dvc.org/">DVC </a>and its journey. The others are a collection of exciting stories and ideas centered around ML best practices and workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CMarxc4ZdzjUS_80gWCYHw.jpeg" /><figcaption>How cool is this handmade swag from our community? We were in tears!</figcaption></figure><p>The past few months have been so busy and full of great events! We love how involved our community is and can’t wait to share more with you:</p><ul><li>We have organized our very first <a href="https://www.meetup.com/San-Francisco-Machine-Learning-Meetup/events/264846847/">meetup</a>! So many great conversations, new use cases and insights! Many thanks to <a href="https://www.linkedin.com/in/daniel-fischetti-4a6592bb/">Dan Fischetti</a> from <a href="https://standard.ai/">Standard Cognition</a>, who joined our Dmitry Petrov on stage. Watch the recording <a href="https://youtu.be/RHQXK7EC0jI">HERE</a>.</li><li><a href="https://blog.dataversioncontrol.com/dvc-org-for-hacktoberfest-2019-ce5320151a0c">Hacktoberfest</a> was a great exercise for DVC team on many levels and we really enjoyed supporting new contributors. Kudos to <a href="https://twitter.com/explorer_07">Nabanita Dash</a> for organizing a cool <a href="https://twitter.com/psociiit/status/1185150096792535040?s=20">DVC-themed hackathon</a>!</li><li>We’ve crossed 4k stars mark on <a href="https://github.com/iterative/dvc">Github</a>!</li><li>DVC was participating in the <a href="https://twitter.com/FossMec/status/1192866498324254720">Devsprints</a> (Thank you <a href="https://twitter.com/kurianbenoy2">Kyrian Benoy</a> for the intro!) and we were happy to jump in and help with some mentoring.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GxwHe5-zTaVlh2uxI0Sf9A.png" /><figcaption>Devsprints participants on our <a href="http://dvc.org/chat">Discord</a> channel</figcaption></figure><ul><li>DVC became part of the default <a href="https://formulae.brew.sh/formula/dvc">Homebrew formulae</a>! So now you can install it as easy as 𝚋𝚛𝚎𝚠 𝚒𝚗𝚜𝚝𝚊𝚕𝚕 𝚍𝚟𝚌!</li><li>We helped 2 aspiring speakers deliver their very first conference talks. <a href="https://twitter.com/kurianbenoy2/status/1183427495342694401?s=20">Kurian Benoy</a> was speaking at <a href="https://in.pycon.org/2019/">PyconIndia</a> and <a href="https://www.linkedin.com/in/aman-sharma606/">Aman Sharma</a> was speaking at <a href="https://scipy.in/2019#speakers">SciPyIndia</a>. <strong>Supporting speakers is something we are passionate about and if you ever wanted to give a talk on a DVC-related topic — we are here to help, just </strong><a href="https://dvc.org/support"><strong>let us know</strong></a><strong>!</strong></li><li>Our own <a href="https://medium.com/u/2b1c8f1b5d10">Dmitry Petrov</a> went to Europe to speak at the <a href="https://osseu19.sched.com/speaker/dmitry35">Open Source Summit Europe</a> in Lyon, <a href="https://www.highload.ru/moscow/2019/abstracts/6032">Highload++</a> in Moscow and made a stop in in Berlin to co-host a <a href="https://www.meetup.com/codecentric-Berlin/events/265555810/">meetup</a> with our favourite AI folks from <a href="https://www.codecentric.de/">Codecentric</a>!</li></ul><p>Here are some of the great pieces of content around DVC and ML ops that we discovered in October and November:</p><p><a href="https://www.deploymachinelearning.com/"><strong>Deploy Machine Learning Models with Django</strong></a><strong> by </strong>Piotr Płoński.</p><blockquote>…building your ML system has a great advantage — it is tailored to your needs. It has all features that are needed in your ML system and can be as complex as you wish. <br>This tutorial is for readers who are familiar with ML and would like to learn how to build ML web services.</blockquote><p><a href="https://www.deploymachinelearning.com/">Deploy Machine Learning Models with Django</a></p><p><a href="https://towardsdatascience.com/how-to-manage-your-machine-learning-workflow-with-dvc-weights-biases-and-docker-5529ea4e59e0"><strong>How to Manage Your Machine Learning Workflow with DVC, Weights &amp; Biases, and Docker</strong></a><strong> by </strong><a href="https://towardsdatascience.com/@james_aka_yale?source=post_page-----5529ea4e59e0----------------------"><strong>James Le</strong></a><strong>.</strong></p><blockquote>In this article, I want to show 3 powerful tools to simplify and scale up machine learning development within an organization by making it easy to track, reproduce, manage, and deploy models.</blockquote><p><a href="https://towardsdatascience.com/how-to-manage-your-machine-learning-workflow-with-dvc-weights-biases-and-docker-5529ea4e59e0">How to Manage Your Machine Learning Workflow with DVC, Weights &amp; Biases, and Docker</a></p><p><a href="https://towardsdatascience.com/creating-a-solid-data-science-development-environment-60df14ce3a34"><strong>Creating a solid Data Science development environment</strong></a><strong> by </strong><a href="https://towardsdatascience.com/@gabrielsgoncalves?source=post_page-----60df14ce3a34----------------------"><strong>Gabriel dos Santos Goncalves</strong></a></p><blockquote>We do believe that Data Science is a field that can become even more mature by using best practices in project development and that Conda, Git, DVC, and JupyterLab are key components of this new approach</blockquote><p><a href="https://towardsdatascience.com/creating-a-solid-data-science-development-environment-60df14ce3a34">Creating a solid Data Science development environment</a></p><p><a href="https://medium.com/y-data-stories/creating-reproducible-data-science-workflows-with-dvc-3bf058e9797b"><strong>Creating reproducible data science workflows with DVC</strong></a><strong> by </strong><a href="https://medium.com/@glib.ivashkevych?source=post_page-----3bf058e9797b----------------------"><strong>Gleb Ivashkevich</strong></a><strong>.</strong></p><blockquote>DVC is a powerful tool and we covered only the fundamentals of it.</blockquote><p><a href="https://medium.com/y-data-stories/creating-reproducible-data-science-workflows-with-dvc-3bf058e9797b">Creating reproducible data science workflows with DVC</a></p><h3>Discord gems</h3><p>There are lots of hidden gems in our Discord community discussions. Sometimes they are scattered all over the channels and hard to track down.</p><p>We are sifting through the issues and discussions and share with you the most interesting takeaways.</p><p><strong>Q: When you do a </strong><strong>dvc import you get the state of the data in the original repo at that moment in time from that repo, right? </strong><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/618744949277458462"><strong>The overall state of that repo (e.g. Git commit id/hash) is not preserved upon import, right?</strong></a></p><p><strong>A: </strong>On the contrary, DVC relies on Git commit id/hash to determine the state of the data as well as code. Git commit id (hash) is saved in DVC file upon import, data itself is copied/downloaded into DVC repo cache but would not be pushed to the remote — DVC does not create duplicates. There is a command to advance/update it when it’s needed — dvc update. Git commit hash saved to provide reproducibility. Even if the source repo HEAD has changed your import stays the same until you run dvc update or redo dvc import.</p><p><strong>Q: I’m trying to understand if DVC is an appropriate solution for storing data under GDPR requirements. </strong><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/621057268145848340"><strong>That means that permanent deletion of files with sensitive data needs to be fully supported.</strong></a></p><p>Yes, in this sense DVC is not very different from using bare S3, SSH or any other storage where you can go and just delete data. DVC can give a bit of overhead to locate a specific file to delete, but otherwise it’s all the same you will be able to delete any file you want. Read more details in <a href="https://discordapp.com/channels/485586884165107732/485596304961962003/621062105524862987">this discussion</a>.</p><p><strong>Q: </strong><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/621591769766821888"><strong>Is there anyway to get the remote url for specific DVC-files?</strong></a><strong> Say, I have a DVC-file </strong><strong>foo.png.dvc — is there a command that will show the remote url, something like </strong><strong>dvc get-remote-url foo.png.dvc which will return e.g. the Azure url to download.</strong></p><p>There is no special command for that, but if you are using Python, you could use our API specifically designed for that:</p><pre>from dvc.api import get_url</pre><pre>url = get_url(https://medium.com/path,%20repo=&quot;https://github.com/user/proj&quot;,%20%20rev=&quot;mybranch&quot;)</pre><p>so, you could as well use this from CLI as a wrapper command.</p><p><strong>Q: </strong><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/619244714071425035"><strong>Can DVC be integrated with MS Active Directory (AD) authentication for controlling access?</strong></a><strong> The GDPR requirements would force me to use such a system to manage access.</strong></p><p>Short answer: no (as of the date of publishing this Heartbeat issue) Good news — it should be very easy to add, so we would welcome a contribution :) Azure has a connection argument for AD — quick googling shows this <a href="https://github.com/AzureAD/azure-activedirectory-library-for-python">library</a>, which is what probably needed.</p><p><strong>Q: </strong><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/625124341201502209"><strong>How do I uninstall DVC from Mac installed as a package?</strong></a></p><p>When installing using plain.pkg it is a bit tricky to uninstall, so we usually recommend using things like brew cask instead if you really need the binary package. Try to run these commands:</p><pre>$ sudo rm -rf /usr/local/bin/dvc</pre><pre>$ sudo rm -rf /usr/local/lib/dvc</pre><pre>$ sudo pkgutil --forget com.iterative.dvc</pre><p>to uninstall the package.</p><p><strong>Q: We are using SSH remote to store data, but the problem is that everyone within the project has different username on the remote machine and thus we cannot set it in the config file (that is committed to Git). </strong><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/619420070111608848"><strong>Is there a way to add just host and path, without the username?</strong></a></p><p>Yes, you should use --local or--global config options to set user per project or per use machine without sharing (committing) them to Git:</p><p>$ dvc remote modify myremote — local (or — global) user myuser</p><p><strong>Q: </strong><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/628227197592797191"><strong>I still get the </strong></a><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/628227197592797191"><strong>SSL ERROR when I try to perform a </strong></a><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/628227197592797191"><strong>dvc push with or without </strong></a><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/628227197592797191"><strong>use_ssl = false</strong></a><strong>?</strong></p><p>A simple environment variable like this:</p><p>$ export AWS_CA_BUNDLE=/path/to/cert/cert.crt dvc push</p><p>should do the trick for now, we plan to fix the ca_bundle option soon.</p><p><strong>Q: I have just finished a lengthy </strong><strong>dvc repro and I’m happy with the result. However, I realized that I didn’t specify a dependency which I needed (and obviously is used in the computation). Can I somehow fix it? </strong><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/620572187841265675"><strong>Add the dependency to the stage file (including the right MD5) without rerunning/reproducing the stage (this is not needed as this additional dependency hasn’t changed).</strong></a></p><p>You need to edit the DVC-file. In the deps section add:</p><p>-path: not/included/file/path</p><p>and run dvc commit file.dvc to save changes w/o running the pipeline again. See an example <a href="https://discordapp.com/channels/485586884165107732/563406153334128681/620641530075414570">here</a>.</p><p><strong>Q: For some reason </strong><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/629704961868955648"><strong>we need to always specify the remote name when doing a </strong></a><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/629704961868955648"><strong>dvc push</strong></a><strong> e.g., </strong><strong>dvc push -r upstream as opposed to </strong><strong>dvc push (mind no additional arguments).</strong></p><p>You can specify “default” remote:</p><p>$ dvc remote add -d remote /path/to/my/main/remote</p><p>then, dvc push (and other commands like dvc pull) will know to push to the default</p><p><strong>Q: </strong><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/620715145374466048"><strong>If I want stage B to run after stage A, but the stage A has no output, can I specify A’s DVC-file as B’s dependency?</strong></a></p><p>No, at least at the time of publishing this. You could use a phony output though. E.g. make the stage A output some dummy file and make B depend on it. Please, consider creating or upvoting a relevant issue on our Github if you’d this to be implemented.</p><p><strong>Q: I’m just getting started with DVC, but I’d like to use it for multiple developers to access the data and share models and code. </strong><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/598867829785362452"><strong>I do own the server, but I’m not sure how to use DVC with SSH remote?</strong></a></p><p>Please, refer to <a href="https://discuss.dvc.org/t/how-do-i-use-dvc-with-ssh-remote/279/2">this answer</a> on the DVC forum and check the documentation for the <a href="https://dvc.org/doc/command-reference/remote/add">dvc remote add</a> and <a href="https://dvc.org/doc/command-reference/remote/modify">dvc remote modify</a> commands to see more options and details.</p><p>If you have any questions, concerns or ideas, let us know in the comments below or connect with DVC team <a href="https://dvc.org/support">here</a>. Our <a href="https://twitter.com/DVCorg">DMs on Twitter</a> are always open, too.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=110404842192" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataversioncontrol/november-19-dvc-%EF%B8%8Fheartbeat-110404842192">November ’19 DVC❤️Heartbeat</a> was originally published in <a href="https://medium.com/dataversioncontrol">Data Version Control</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[October ’19 DVC❤️Heartbeat]]></title>
            <link>https://medium.com/dataversioncontrol/october-19-dvc-%EF%B8%8Fheartbeat-4f494fda4626?source=rss----4e21975b393---4</link>
            <guid isPermaLink="false">https://medium.com/p/4f494fda4626</guid>
            <category><![CDATA[version-control]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[tutorial]]></category>
            <dc:creator><![CDATA[Svetlana Grinchenko]]></dc:creator>
            <pubDate>Mon, 04 Nov 2019 23:07:50 GMT</pubDate>
            <atom:updated>2019-11-04T23:09:58.275Z</atom:updated>
            <content:encoded><![CDATA[<p>Every month we are sharing here our news, findings, interesting reads, community takeaways, and everything along the way.</p><p>Some of those are related to our brainchild <a href="https://dvc.org/">DVC </a>and its journey. The others are a collection of exciting stories and ideas centered around ML best practices and workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jTie-pj6KJa9sQmUvpc1cg.png" /></figure><h3>News and links</h3><p>Autumn is a great season for new beginnings and there is so much we love about it this year. Here are some of the highlights:</p><ul><li>Co-hosting our <a href="https://www.meetup.com/San-Francisco-Machine-Learning-Meetup/events/264846847/">first ever meetup</a>! Our <a href="https://medium.com/u/2b1c8f1b5d10">Dmitry Petrov</a> partnering with <a href="https://www.linkedin.com/in/daniel-fischetti-4a6592bb/">Dan Fischetti</a> from <a href="https://twitter.com/standardAI">Standard Cognition</a> to discuss Open-source tools to version control Machine Learning models and experiments. The recording is available now <a href="https://youtu.be/RHQXK7EC0jI">here.</a></li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FRHQXK7EC0jI%3Ffeature%3Doembed&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DRHQXK7EC0jI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FRHQXK7EC0jI%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/95032fdfefecdb1a7612c697bea8f1c1/href">https://medium.com/media/95032fdfefecdb1a7612c697bea8f1c1/href</a></iframe><ul><li><a href="https://blog.dataversioncontrol.com/dvc-org-for-hacktoberfest-2019-ce5320151a0c">Getting ready for the Hacktoberfest</a> and having the whole team get together to pick up and label nice issues and be ready to support the contributors.</li><li>Discovering some really cool blogposts, talks and tutorials from our users all over the world: check <a href="https://blog.octo.com/mise-en-application-de-dvc-sur-un-projet-de-machine-learning/">this blogpost in French</a> or <a href="https://jupyter-tutorial.readthedocs.io/de/latest/reproduce/dvc/init.html">this tutorial in German</a>!</li><li>Having great time working with a <a href="https://github.com/dashohoxha">tech writer</a> brought to us by the <a href="https://developers.google.com/season-of-docs">Google Season of Docs</a> program. Check out these <a href="https://dvc.org/doc/tutorials/interactive">interactive tutorials</a> we’ve created together.</li><li>Having hot internal discussion about Discord vs Slack support/community channels. If you are on the edge like us, have a look at <a href="https://internals.rust-lang.org/t/exploring-new-communication-channels/7859">this discussion</a> in the Rust community, so helpful.</li><li>Seeing <a href="https://medium.com/u/2b1c8f1b5d10">Dmitry Petrov</a> being really happy one day:</li></ul><h3>Dmitry Petrov on Twitter</h3><p>@martinfowler&#39;s books and his website were always the source of programming wisdom 💎 His Refactoring book is the first book I recommend to developers. Now they write about ML lifecycle and automation. I&#39;m especially excited because they use @DVCorg that we&#39;ve created. https://t.co/HwswZqjOsb</p><p>We at DVC.org are so happy every time we discover an article featuring DVC or addressing one of the burning ML issues we are trying to solve. Here are some of the links that caught our eye past month:</p><p><strong>Continuous Delivery for Machine Learning by </strong><a href="https://twitter.com/dtsato"><strong>Danilo Sato</strong></a><strong>, </strong><a href="https://twitter.com/arifwider"><strong>Arif Wider</strong></a><strong>, </strong><a href="https://twitter.com/intellification"><strong>Christoph Windheuser</strong></a><strong> and curated by </strong><a href="https://martinfowler.com/"><strong>Martin Fowler</strong></a><strong>.</strong></p><blockquote>As Machine Learning techniques continue to evolve and perform more complex tasks, so is evolving our knowledge of how to manage and deliver such applications to production. By bringing and extending the principles and practices from Continuous Delivery, we can better manage the risks of releasing changes to Machine Learning applications in a safe and reliable way.</blockquote><p><a href="https://martinfowler.com/articles/cd4ml.html">Continuous Delivery for Machine Learning</a></p><p><a href="https://medium.com/signaturit-tech-blog/the-path-to-identity-validation-2-3-4f698b2ffae9"><strong>The Path to Identity Validation</strong></a><strong> by </strong><a href="https://medium.com/u/dd78e1cf7c18"><strong>Víctor Segura</strong></a><strong>.</strong></p><blockquote>So, the first question is clear: how to choose the optimal hardware for neural networks? Secondly, assuming that we have the appropriate infrastructure, how to build the machine learning ecosystem to train our models efficiently and not die trying? At <em>Signaturit</em>, we have the solution ;)</blockquote><p><a href="https://medium.com/signaturit-tech-blog/the-path-to-identity-validation-2-3-4f698b2ffae9">The Path to Identity Validation (2/3)</a></p><p><strong>Talk: </strong><a href="https://pretalx.com/pyconuk-2019/talk/GCLBFH/"><strong>Managing Big Data in Machine Learning projects</strong></a><strong> by </strong><a href="https://twitter.com/vvasworld"><strong>V Vishnu Anirudh</strong></a><strong> at the </strong><a href="https://2019.pyconuk.org/"><strong>Pycon UK 2019.</strong></a></p><blockquote>My talk will focus on Version Control Systems (VCS) for big-data projects. With the advent of Machine Learning (ML) , the development teams find it increasingly difficult to manage and collaborate on projects that deal with huge amounts of data and ML models apart from just source code.</blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F4XpHk85_x0E%3Ffeature%3Doembed&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D4XpHk85_x0E&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F4XpHk85_x0E%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/62c3506d7c47cea03fba8e85983a5f1d/href">https://medium.com/media/62c3506d7c47cea03fba8e85983a5f1d/href</a></iframe><p><strong>Podcast: TWIML Talk #295 </strong><a href="https://twimlai.com/twiml-talk-295-managing-deep-learning-experiments-with-lukas-biewald/"><strong>Managing Deep Learning Experiments</strong></a><strong> with </strong><a href="https://twitter.com/l2k"><strong>Lukas Biewald</strong></a></p><blockquote>Seeing a need for reproducibility in deep learning experiments, Lukas founded Weights &amp; Biases. In this episode we discuss his experiment tracking tool, how it works, the components that make it unique in the ML marketplace and the open, collaborative culture that Lukas promotes. Listen to Lukas delve into how he got his start in deep learning experiments, what his experiment tracking used to look like, the current Weights &amp; Biases business success strategy, and what his team is working on today.</blockquote><p><a href="https://twimlai.com/twiml-talk-295-managing-deep-learning-experiments-with-lukas-biewald/">Managing Deep Learning Experiments with Lukas Biewald - Talk #295</a></p><h3>Discord gems</h3><p>There are lots of hidden gems in our Discord community discussions. Sometimes they are scattered all over the channels and hard to track down.</p><p>We are sifting through the issues and discussions and share with you the most interesting takeaways.</p><p><strong>Q: I’ve just run a </strong><strong>dvc run step, and realised I forgot to declare an output file. </strong><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/593743448020877323"><strong>Is there a way to add an output file without rerunning the (computationally expensive) step/stage?</strong></a></p><p>If you’ve already ran it, you could just open created DVC-file with an editor and add an entry to the outs field. After that, just run dvc commit my.dvc and it will save the checksums and data without re-running your command. dvc run --no-exec would also work with commit instead of modifying the DVC-file by hand.</p><p><strong>Q: </strong><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/593869598651318282"><strong>For metric files do I have to use </strong></a><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/593869598651318282"><strong>dvc run to set a metric or can I do it some other way?</strong></a><strong> Can I use metrics functionality without the need to setup and manage DVC cache and remote storage?</strong></p><p>Any file that is under DVC control (e.g. added with dvc add or an output in dvc run -o)<strong> </strong>can be made a metric file with dvc metrics add file. Alternatively a command dvc run -M file makes file a metric without caching it. It means dvc metrics show can be used while file is still versioned by Git.</p><p><strong>Q:</strong> <a href="https://discordapp.com/channels/485586884165107732/485596304961962003/595586670498283520"><strong>Is there a way not to add the full (Azure) connection string to the</strong></a><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/595586670498283520"><strong> .dvc/config file that is being checked into Git for using dvc remotes</strong></a><strong>? I think it’s quite unhealthy to have secrets checked in SCM.</strong></p><p>There are two options — use AZURE_STORAGE_CONNECTION_STRING environment variable or use --local flag that will put it into the .dvc/config.local that is added to the gitignore, so you don’t track it with it and so won’t expose secrets.</p><p><strong>Q:</strong> <a href="https://discordapp.com/channels/485586884165107732/485596304961962003/601068667131920385"><strong>I would like to know if it is possible to manage files under DVC whilst keeping them in their original locations (e.g. on a network drive in a given folder structure)</strong></a><strong>? </strong><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/615278138896941101"><strong>If I want to add a large file to be tracked by DVC, and it is in a bucket on S3 or GCS, can I do that without downloading it locally?</strong></a></p><p>Yes, you are probably looking for external dependencies and outputs. This is the <a href="https://dvc.org/doc/user-guide/managing-external-data">link</a> to the documentation to start.</p><p><strong>Q:</strong> <a href="https://discordapp.com/channels/485586884165107732/485596304961962003/606388040377565215"><strong>How do I setup DVC so that NAS (e.g. Synology) acts as a shared DVC cache?</strong></a></p><p>Using NAS (e.g. NFS) is a very common scenario for DVC. In short you use dvc cache dir to setup a cache externally. Set cache type to use symlinks and enable protected mode. We are preparing a <a href="https://github.com/iterative/dvc.org/blob/31c5d424c6530bb793af69c2af578d2b8a374d02/static/docs/use-cases/shared-storage-on-nfs.md">document</a> how to setup the NFS as a shared cache, but I think it can be applied to any NAS.</p><p><strong>Q: So I have some data that is in the hundreds of gigs. </strong><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/608013531010301952"><strong>If I enable symlink, hardlink strategy and cache protecting, will DVC automatically choose this strategy over copying when trying to use </strong></a><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/608013531010301952"><strong>dvc add</strong></a><strong>?</strong></p><p>Yes, it will! Here is some clarification. So when you set those settings like that, dvc add data will move data to your cache and then will create a hardlink from your cache to your workspace.</p><p>Unless your cache directory and your workspace are on different filesystems, move should be instant. Please, find more information <a href="https://dvc.org/doc/user-guide/large-dataset-optimization">here</a>.</p><p><strong>Q: My repo’s DVC is “busy and locked” and I’m not sure how it got that way and how to remove/diagnose the lock. </strong><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/608392956679815168"><strong>Any suggestions?</strong></a></p><p>DVC uses a lock file to prevent running two commands at the same time. The lock <a href="https://dvc.org/doc/user-guide/dvc-files-and-directories#dvc-files-and-directories">file</a> is under the .dvc directory. If no DVC commands running and you are still getting this error it’s safe to remove this file manually to resolve the issue.</p><p><strong>Q: </strong><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/611209851757920266"><strong>I’m trying to understand how does DVC remote add work in case of a local folder and what is the best workflow when data is outside of your project root?</strong></a></p><p>When using DVC, in most cases we assume that your data will be somewhere under project root. There is an option to use so called <a href="https://dvc.org/doc/user-guide/managing-external-data">external dependencies</a>, which is data that is usually too big to be stored under your project root, but if you operate on data that is of some reasonable size, I would recommend starting with putting data somewhere under project root. Remotes are usually places where you store your data, but it is DVC task to move your data around. But if you want to keep your current setup where you will have data in different place than your project, you will need to refer to data with full paths. So, for example:</p><p>1. You are in /home/gabriel/myproject and you have initialized dvc and git repository</p><p>2. You have featurize.py in your project dir, and want to use data to produce some features and than train.py to train a model.</p><p>3. dvc run -d /research_data/myproject/videos -o /research_data/myproject/features python featurize.py</p><p>to tell DVC, that you use /research_data/myproject/videos to featurize, and produce output to your features dir. Note that your code should be aware of those paths, they can be hardcoded inside featurize.py, but point of dvc run is just to tell DVC what artifacts belong to currently defined step of ML pipeline.</p><p><strong>Q: When I run </strong><strong>du command to check how much space DVC project consumes I see that it duplicates/copies data. </strong><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/613935477896249364"><strong>It’s very space and time consuming to copy large data files, is there a way to avoid that?</strong></a><strong> It takes too long to add large files to DVC.</strong></p><p>Yes! You don’t have to copy files with DVC. First of all, there are two reasons when du can show that it takes double the space to store data under DVC control. du can be inaccurate when the underlying file system supports reflinks (XFS on Linux, APFS on Mac, etc). This is actually the best scenario since no copying is happening and no changes are required to any DVC settings. Second, case means that copy semantics is used by default. It can be turned off by providing cache type symlinks,hardlinks. Please, read more on this <a href="https://dvc.org/doc/user-guide/large-dataset-optimization#file-link-types-for-the-dvc-cache">here</a>.</p><p><strong>Q: </strong><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/615479227189559323"><strong>How can I detach a file from DVC control?</strong></a></p><p>Just removing the corresponding DVC-file and running dvc gc after that should be enough. It’ll stop tracking the data file and clean the local cache that might still contain it. Note! Don’t forget to run dvc unprotect if you use advanced<a href="https://dvc.org/doc/user-guide/large-dataset-optimization"> DVC setup with symlinks and hardlinks</a> (cache.type config option is not default). If dvc gc behavior is not granular enough you can manually find the by its cache from the DVC-file in .dvc/cache and remote storage. Learn <a href="https://dvc.org/doc/user-guide/dvc-files-and-directories#structure-of-cache-directory">here</a> how they are organized.</p><p><strong>Q: </strong><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/621057268145848340"><strong>I’m trying to understand if DVC is an appropriate solution for storing data under GDPR requirements.</strong></a><strong> That means that permanent deletion of files with sensitive data needs to be fully supported.</strong></p><p>Yes, in this sense DVC is not very different from using bare S3, SSH or any other storage where you can go and just delete data. DVC can give a bit of overhead to locate a specific file to delete, but otherwise it’s all the same you will be able to delete any file you want. See more details on how you retrospectively can edit directories under DVC control <a href="https://discordapp.com/channels/485586884165107732/485596304961962003/621062105524862987">here</a>.</p><p>If you have any questions, concerns or ideas, let us know in the comments below or connect with DVC team <a href="https://dvc.org/support">here</a>. Our <a href="https://twitter.com/DVCorg">DMs on Twitter</a> are always open, too.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f494fda4626" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataversioncontrol/october-19-dvc-%EF%B8%8Fheartbeat-4f494fda4626">October ’19 DVC❤️Heartbeat</a> was originally published in <a href="https://medium.com/dataversioncontrol">Data Version Control</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DVC.org for Hacktoberfest 2019]]></title>
            <link>https://medium.com/dataversioncontrol/dvc-org-for-hacktoberfest-2019-ce5320151a0c?source=rss----4e21975b393---4</link>
            <guid isPermaLink="false">https://medium.com/p/ce5320151a0c</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[hacktoberfest]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[dev]]></category>
            <dc:creator><![CDATA[Svetlana Grinchenko]]></dc:creator>
            <pubDate>Tue, 08 Oct 2019 20:28:10 GMT</pubDate>
            <atom:updated>2019-10-10T15:29:18.688Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dfjcUFYm-T-jOtyk-Dn16A.png" /></figure><p>Our favourite month of the year <a href="https://hacktoberfest.digitalocean.com/">Hacktoberfest</a> is already in full swing and we at DVC.org are so excited to be a part of it!</p><p><strong>What is Hacktoberfest?</strong></p><p><a href="https://hacktoberfest.digitalocean.com/">Hacktoberfest</a> is a monthly-long program that celebrates open source and encourages you to contribute to open source projects (and rewards you with stickers and a cool T-shirt!). Whether you’re a seasoned contributor or looking for projects to contribute to for the first time, you’re welcome to participate!</p><p>It is the 6th season of Hacktoberfest and the 2d year of participating for DVC.org team. We really enjoyed it in 2018 and this year we are upping the game with our own cool stickers, special edition T-shirts and a <a href="https://github.com/iterative/dvc/labels/hacktoberfest">collection of carefully picked tickets</a>.</p><p><strong>How to participate?</strong></p><p>If you haven’t started your Hacktoberfest challenge yet, it is just the right time, you have 3 weeks left to submit PRs and get your swag! Here are some important details:</p><ul><li>Hacktoberfest is open to everyone in the global community.</li><li>You can sign up anytime between October 1 and October 31. Make sure to sign up on the <a href="https://hacktoberfest.digitalocean.com/">official Hacktoberfest website</a> for your PRs to count.</li><li>To get a shirt, you must make 4 legit pull requests (PRs) between October 1–31 in any time zone.</li><li>Pull requests can be made in any public GitHub-hosted repositories/projects, not just the ones highlighted.</li></ul><p>And the special addition from DVC.org team:</p><ul><li>Look through the list of <a href="https://github.com/iterative/dvc/labels/hacktoberfest">DVC Hacktoberfest tickets</a> or the list of <a href="https://github.com/iterative/dvc/labels/good%20first%20issue">good DVC first issues</a>.</li><li>Make a PR to DVC and get our stickers.</li><li>Close three issues for DVC and get a special DVC T-shirt.</li></ul><p><strong>Why contribute to DVC?</strong></p><p><a href="http://dvc.org/">DVC</a> (Data Version Control) is a relatively young open source project. It was started in late 2017 by a data scientist and an engineer to fill in the gaps in the ML processes tooling. Nowadays DVC is growing pretty fast and though our in-house team is quite small, we have to thank our contributors (more than 100 in both code and docs) for developing DVC with us.</p><p>DVC is participating in Hacktoberfest for 2 years in a row to bring more people into open source, to learn from them and to give back by sharing our own experience. This year we decided to focus on a single important topic for us — improving UI/UX.</p><p>As our contributors and maintainers were sifting through the feature requests, bugs, and improvements to create a good <a href="https://github.com/iterative/dvc/labels/hacktoberfest">list of Hacktoberfest tickets</a>, we noticed that UI/UX label on Github is popping up again and again. DVC is a command line tool, and improving UI/UX in our case means making decisions on how to name command options, where and when to use <a href="https://github.com/iterative/dvc/issues/2498">confirmation prompts</a> and/or where abort execution, what exactly user would expect to see in the output, how to test it later, etc.</p><p>Why improving UI/UX appears to be so important for DVC at this stage? Perhaps because the project is more mature now and we are ready to spend more time on polishing it. Or maybe because it is still too-engineering focused and we used to disregard/de-prioritize all this ‘fancy’ stuff. Or it is because we just lack experience in creating good CLI UI/UX!</p><p>One or another, those are great reasons to focus on improving UI (in a broader sense than just GUI), improving docs, creating powerful consistent experience for our users and increasing accessibility of DVC.</p><p>That’s how <a href="https://devcenter.heroku.com/articles/cli-style-guide">Heroku’s CLI style guide</a> starts:</p><blockquote>Heroku CLI plugins should provide a clear user experience, targeted primarily for human readability and usability, which delights the user, while at the same time supporting advanced users and output formats. This article provides a clear direction for designing delightful CLI plugins.</blockquote><p>At DVC we are building user experience in line with these principles too, but we also have our own challenges. And here we turn for help to the global open source community and all the contributors out there.</p><p>For all of us who have a heart for open source — let’s discuss, contribute, learn, take the technologies forward and build something great together!</p><p>Happy hacking!</p><p>We are happy to hear from you <a href="https://dvc.org/support">here</a>. Our <a href="https://twitter.com/DVCorg">DMs on Twitter</a> are always open, too!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce5320151a0c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataversioncontrol/dvc-org-for-hacktoberfest-2019-ce5320151a0c">DVC.org for Hacktoberfest 2019</a> was originally published in <a href="https://medium.com/dataversioncontrol">Data Version Control</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[September ’19 DVC❤️Heartbeat]]></title>
            <link>https://medium.com/dataversioncontrol/september-19-dvc-%EF%B8%8Fheartbeat-89df508fde42?source=rss----4e21975b393---4</link>
            <guid isPermaLink="false">https://medium.com/p/89df508fde42</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[version-control]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Svetlana Grinchenko]]></dc:creator>
            <pubDate>Wed, 25 Sep 2019 22:49:52 GMT</pubDate>
            <atom:updated>2019-10-01T04:46:20.799Z</atom:updated>
            <content:encoded><![CDATA[<p>Every month we are sharing here our news, findings, interesting reads, community takeaways, and everything along the way.</p><p>Some of those are related to our brainchild <a href="https://dvc.org/">DVC </a>and its journey. The others are a collection of exciting stories and ideas centered around ML best practices and workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_fre2CrgYnzNXIkUvrLGig.jpeg" /></figure><h3>News and links</h3><p>We are super excited to co-host our very first <a href="https://www.meetup.com/San-Francisco-Machine-Learning-Meetup/events/264846847/"><strong>meetup in San Francisco on October 10</strong></a>! We will gather at the brand new Dropbox HQ office at 6:30 pm to discuss open-source tools to version control ML models and experiments. <a href="https://medium.com/u/2b1c8f1b5d10">Dmitry Petrov</a> is teaming up with <a href="https://www.linkedin.com/in/daniel-fischetti-4a6592bb/">Daniel Fischetti</a> from <a href="https://standard.ai/">Standard Cognition</a> to discuss best ML practices. Join us and save your spot now:</p><p><a href="https://www.meetup.com/San-Francisco-Machine-Learning-Meetup/events/264846847/">Open-source tools to version control Machine Learning models and experiments</a></p><p>If you are not in SF on this date and happen to be in Europe — don’t miss the PyCon DE &amp; PyData Berlin 2019 joint event on October 9–11. We cannot make it to Berlin this year, but we were thrilled to discover 2 independent talks featuring DVC by <a href="https://de.pycon.org/program/pydata-ppgwxl-version-control-for-data-science-alessia-marcolini/">Alessia Marcolini</a> and <a href="https://de.pycon.org/program/pydata-cwmae7-tools-that-help-you-get-your-experiments-under-control-katharina-rasch/">Katharina Rasch</a>.</p><p>Some other highlights of the end of summer:</p><ul><li>Our users and contributors keep creating fantastic pieces of content around DVC (sharing some links below, but it’s only a fraction of what we have in stock — can’t be more happy and humbled about it!).</li><li>We’ve reached 79 contributors to <a href="https://github.com/iterative/dvc">DVC core project</a> and 74 contributors to <a href="https://github.com/iterative/dvc.org">DVC documentation</a> (and have something special in mind to celebrate our 100th contributors).</li><li>we enjoyed working with all the talented <a href="https://developers.google.com/season-of-docs/">Google Season of docs</a> applicants and now moving to the next stage with our chosen tech writer <a href="http://dashohoxha.fs.al/">Dashamir Hoxha</a>.</li><li>We’ve <a href="https://twitter.com/DVCorg/status/1147220439472545793?s=20">crossed the 3,000 stars mark</a> on Github (<a href="https://github.com/iterative/dvc">over 3,500 now</a>). Thank you for your support!</li><li>We’ve <a href="https://twitter.com/a142hr/status/1164256520235675648?s=20">had great time</a> at the <a href="https://events.linuxfoundation.org/events/open-source-summit-north-america-2019/program/">Open Source Summit</a> by Linux foundation in San Diego — speaking on stage, running a booth and chatting with all the amazing open-source crowd out there.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*a7YDzfsGIPFPLcVb" /></figure><p>Here are some of the great pieces of content around DVC and ML ops that we discovered in July and August:</p><p><a href="https://twitter.com/NathanBenaich/status/1151815916512010242"><strong>Great insightful discussion</strong></a><strong> on Twitter about versioning ML projects started by </strong><a href="https://medium.com/u/7c25066d2234"><strong>Nathan Benaich</strong></a><strong>.</strong></p><h3>Nathan Benaich on Twitter</h3><p>🙏Question to ML friends: How do you go about version control for your ML projects (data, models, and intermediate steps in your data pipelines)? Have you built your own tools? Are using something open source? Or a SaaS? Or does this come bundled with your ML infra products? Thx!</p><p><a href="https://medium.com/ixorthink/our-machine-learning-workflow-dvc-mlflow-and-training-in-docker-containers-5b9c80cdf804"><strong>Our Machine Learning Workflow: DVC, MLFlow and Training in Docker Containers</strong></a><strong> by </strong><a href="https://medium.com/@ward.vanlaer?source=post_page-----5b9c80cdf804----------------------"><strong>Ward Van Laer</strong></a><strong>.</strong></p><blockquote>It is possible to manage your work flow using open-source and free tools.</blockquote><p><a href="https://medium.com/ixorthink/our-machine-learning-workflow-dvc-mlflow-and-training-in-docker-containers-5b9c80cdf804">Our Machine Learning Workflow: DVC, MLFlow and Training in Docker Containers</a></p><p><a href="https://medium.com/qonto-engineering/using-dvc-to-create-an-efficient-version-control-system-for-data-projects-96efd94355fe"><strong>Using DVC to create an efficient version control system for data projects</strong></a><strong> by </strong><a href="https://medium.com/@basile_16101?source=post_page-----96efd94355fe----------------------"><strong>Basile Guerrapin</strong></a><strong>.</strong></p><blockquote>DVC brought versioning for inputs, intermediate files and algorithm models to the VAT auto-detection project and this drastically increased our <strong>productivity</strong>.</blockquote><p><a href="https://medium.com/qonto-engineering/using-dvc-to-create-an-efficient-version-control-system-for-data-projects-96efd94355fe">Using DVC to create an efficient version control system for data projects</a></p><p><a href="https://techsparx.com/software-development/ai/dvc/versioning-example.html"><strong>Managing versioned machine learning datasets in DVC, and easily share ML projects with colleagues</strong></a><strong> by </strong><a href="https://medium.com/u/d517e29ab56b"><strong>David Herron</strong></a><strong>.</strong></p><blockquote>In this tutorial we will go over a simple image classifier. We will learn how DVC works in a machine learning project, how it optimizes reproducing results when the project is changed, and how to share the project with colleagues.</blockquote><p><a href="https://techsparx.com/software-development/ai/dvc/versioning-example.html">Managing versioned machine learning datasets in DVC, and easily share ML projects with colleagues</a></p><p><a href="https://towardsdatascience.com/how-to-use-data-version-control-dvc-in-a-machine-learning-project-a78245c0185"><strong>How to use data version control (dvc) in a machine learning project</strong></a><strong> by </strong><a href="https://towardsdatascience.com/@matthiasbitzer94?source=post_page-----a78245c0185----------------------"><strong>Matthias Bitzer</strong></a><strong>.</strong></p><blockquote>To illustrate the use of dvc in a machine learning context, we assume that our data is divided into train, test and validation folders by default, with the amount of data increasing over time either through an active learning cycle or by manually adding new data.</blockquote><p><a href="https://towardsdatascience.com/how-to-use-data-version-control-dvc-in-a-machine-learning-project-a78245c0185">How to use data version control (dvc) in a machine learning project</a></p><p><a href="https://towardsdatascience.com/version-control-ml-model-4adb2db5f87c"><strong>Version Control ML Model</strong></a><strong> by </strong><a href="https://towardsdatascience.com/@TianchenW?source=post_page-----4adb2db5f87c----------------------"><strong>Tianchen Wu</strong></a></p><blockquote>This post presents a solution to version control machine learning models with git and dvc (<a href="https://dvc.org/doc/tutorial">Data Version Control</a>).</blockquote><p><a href="https://towardsdatascience.com/version-control-ml-model-4adb2db5f87c">Version Control ML Model</a></p><p><a href="https://dev.to/robogeek/reflinks-vs-symlinks-vs-hard-links-and-how-they-can-help-machine-learning-projects-1cj4"><strong>Reflinks vs symlinks vs hard links, and how they can help machine learning projects</strong></a><strong> by </strong><a href="https://medium.com/u/d517e29ab56b"><strong>David Herron</strong></a></p><blockquote>In this blog post we’ll go over the details of using links, some cool new stuff in modern file systems (reflinks), and an example of how DVC (Data Version Control, <a href="https://dvc.org/">https://dvc.org/</a>) leverages this.</blockquote><p><a href="https://dev.to/robogeek/reflinks-vs-symlinks-vs-hard-links-and-how-they-can-help-machine-learning-projects-1cj4">Reflinks vs symlinks vs hard links, and how they can help machine learning projects</a></p><p><a href="https://blog.codecentric.de/en/2019/08/dvc-dependency-management/"><strong>DVC dependency management — a guide</strong></a><strong> by </strong><a href="https://blog.codecentric.de/en/author/bert-besser/"><strong>Bert Besser</strong></a><strong> and </strong><a href="https://blog.codecentric.de/en/author/veronika-schindler/"><strong>Veronika Schwan</strong></a><strong>.</strong></p><blockquote>This post is a follow-up to <a href="https://blog.codecentric.de/en/2019/03/walkthrough-dvc/">A walkthrough of DVC</a> that deals with managing dependencies between DVC projects. In particular, this follow-up is about importing specific versions of an artifact (e.g. a trained model or a dataset) from one DVC project into another.</blockquote><p><a href="https://blog.codecentric.de/en/2019/08/dvc-dependency-management/">DVC dependency management - a guide - codecentric AG Blog</a></p><p><a href="https://medium.com/@czeslaw.szubert/effective-ml-teams-lessons-learned-6a6e761bc283"><strong>Effective ML Teams — Lessons Learne</strong></a><strong>d by </strong><a href="https://medium.com/@czeslaw.szubert?source=post_page-----6a6e761bc283----------------------"><strong>Czeslaw Szubert</strong></a></p><blockquote>In this post I’ll present lessons learned on how to setup successful ML teams and what you need to devise an effective enterprise ML strategy.</blockquote><p><a href="https://medium.com/@czeslaw.szubert/effective-ml-teams-lessons-learned-6a6e761bc283">Effective ML Teams — Lessons Learned</a></p><p><a href="https://www.esentri.com/lessons-learned-from-training-a-german-speech-recognition-model/">Lessons learned from training a German Speech Recognition model</a> by <a href="https://www.linkedin.com/in/dschoenleber/">David Schönleber</a>.</p><blockquote>Setting up a documentation-by-design workflow and using appropriate tools where needed, e.g. <em>mlflow</em> and <em>dvc,</em> can be a real deal-breaker.</blockquote><p><a href="https://www.esentri.com/lessons-learned-from-training-a-german-speech-recognition-model/">Lessons Learned from Training a German Speech Recognition Model - esentri AG</a></p><h3>Discord gems</h3><p>There are lots of hidden gems in our Discord community discussions. Sometimes they are scattered all over the channels and hard to track down.</p><p>We are sifting through the issues and discussions and share with you the most interesting takeaways.</p><p><strong>Q: I’m getting an error message while trying to use AWS S3 storage:</strong><strong> “ERROR: failed to push data to the cloud — Unable to locate credentials.” </strong><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/587792932061577218"><strong>Any ideas what’s happening?</strong></a></p><p>Most likely you haven’t configured your S3 credentials/AWS account yet. Please, read the full documentation on the AWS website. The short version of what should be done is the following:</p><ul><li><a href="https://portal.aws.amazon.com/gp/aws/developer/registration/index.html">Create your AWS account.</a></li><li>Log in to your AWS Management Console.</li><li>Click on your user name at the top right of the page.</li><li>Click on the Security Credentials link from the drop-down menu.</li><li>Find the Access Credentials section, and copy the latest Access Key ID.</li><li>Click on the Show link in the same row, and copy the Secret Access Key.</li></ul><p>Follow <a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html">this link</a> to setup your environment.</p><p><strong>Q: I added data with </strong><strong>dvc add or </strong><strong>dvc run and see that it takes twice what it was before (with </strong><strong>du command). </strong><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/595402051203235861"><strong>Does it mean that DVC copies data that is added under its control? How do I prevent this from happening?</strong></a></p><p>To give a short summary — by default, DVC copies the files from your working directory to the cache (this is for safety reasons, it is better to duplicate the data.). If you have reflinks (copy-on-write) enabled on your file system, DVC will use that method — which is as safe as copying. You can also configure DVC to use hardlinks/symlinks to save some space and time, but it will require enabling the protected mode (making data files in workspace read-only). Read more details <a href="https://dvc.org/doc/user-guide/large-dataset-optimization">here</a>.</p><p><strong>Q: </strong><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/599345778703597568"><strong>How concurrent-friendly is the cache? And different remotes? Is it safe to have several containers/nodes fill the same cache at the same time?</strong></a></p><p>It is safe and a very common use case for DVC to have a shared cache. Please, check <a href="https://discuss.dvc.org/t/share-nas-data-in-server/180/12">this thread</a>, for example.</p><p><strong>Q:</strong><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/603890677176336394"><strong>What is the proper way to exit the ASCII visualization?</strong></a><strong> (when you run </strong><strong>dvc pipeline show command).</strong></p><p>See this <a href="https://dvc.org/doc/commands-reference/pipeline/show#options">document</a>. To navigate, use arrows or W, A, S, D keys. To exit, press Q.</p><p><strong>Q: </strong><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/606197026488844338"><strong>Is there an issue if I set my </strong></a><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/606197026488844338"><strong>cache.s3 external cache to my default remote?</strong></a><strong> I don’t quite understand what an external cache is for other than I have to have it for external outputs.</strong></p><p>Short answer is that we would suggest keeping them separately to avoid possible checksum overlaps. Cheksum on S3 might theoretically overlap with our checksums (with the content of the file being different), so it could be dangerous. The chances of losing data are pretty slim, but we would not risk it. Right now, we are working on making sure there are no possible overlapping.</p><p><strong>Q: </strong><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/606425815139221504"><strong>What’s the right procedure to move a step .dvc file around the project?</strong></a></p><p>Assuming the file was created with dvc run. There are few possible ways. Obvious one is to delete the file and create a new one with dvc run --no-exec -f file/path/and/name.dvc. Another possibility is to rename/move and then edit manually. See <a href="https://dvc.org/doc/user-guide/dvc-file-format">this document</a> that describes how DVC-files are organized. No matter what method you use, you can run dvc commit file.dvc to save changes without running the command again.</p><p><strong>Q: </strong><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/606917839688957952"><strong>dvc status doesn’t seem to report things that need to be </strong></a><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/606917839688957952"><strong>dvc pushed, is that by design?</strong></a></p><p>You should try with dvc status --cloudor dvc status --remote &lt;your-remote&gt; to compare your local cache with a remote one, by default it only compares the “working directory” with your local cache (to check whether something should be reproduced and saved or not).</p><p><strong>Q: </strong><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/608701494035873792"><strong>What kind of files can you put into </strong></a><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/608701494035873792"><strong>dvc metrics?</strong></a></p><p>The file could be in any format, dvc metric show will try to interpret the format and output it in the best possible way. Also, if you are using csv or json, you can use the--xpath flag to query specific measurements.<strong> </strong>In general, you can make any file a metric file and put any content into it, DVC is not opinionated about it.<strong> </strong>Usually though these are files that measures the performance/accuracy of your model and captures configuration of experiments. The idea is to use dvc metrics show to display all your metrics across experiments so you can make decisions of which combination (of features, parameters, algorithms, architecture, etc.) works the best.</p><p><strong>Q: </strong><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/613639458000207902"><strong>Does DVC take into account the timestamp of a file or is the MD5 only depends on the files actual/bits content?</strong></a></p><p>DVC takes into account only content (bits) of a file to calculate hashes that are saved into DVC-files.</p><p><strong>Q: </strong><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/616421757808541721"><strong>Similar to </strong></a><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/616421757808541721"><strong>dvc gc is there a command to garbage collect from the remote?</strong></a></p><p>dvc gc --remote NAME is doing this, but you should be extra careful, because it will remove everything that is not currently “in use” (by the working directory). Also, please check this <a href="https://github.com/iterative/dvc/issues/2325">issue</a> — semantics of this command might have changed by the time you read this.</p><p><strong>Q: </strong><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/591237578209099786"><strong>How do I use and configure remote storage on IBM Cloud Object Storage?</strong></a></p><p>Since it’s S3 compatible, specifying endpointurl (exact URL depends on the <a href="https://cloud.ibm.com/docs/services/cloud-object-storage?topic=cloud-object-storage-endpoints">region</a>) is the way to go:</p><pre>dvc remote add -d mybucket s3://path/to/dir<br>dvc remote modify mybucket endpointurl<br><a href="https://s3.eu.cloud-object-storage.appdomain.cloud/">https://s3.eu.cloud-object-storage.appdomain.cloud</a></pre><p><strong>Q: </strong><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/592958360903483403"><strong>How can I push data from client to google cloud bucket using DVC?</strong></a><strong> Just want to know how can i set the credentials.</strong></p><p><strong>A: </strong>You can do it by setting environment variable pointing to yours credentials path, like:</p><pre>export GOOGLE_APPLICATION_CREDENTIALS=path/to/credentials</pre><p>It is also possible to set this variable via dvc config:</p><p>dvc remote modify myremote credentialpath /path/to/my/creds</p><p>where myremote is your remote name.</p><p>If you have any questions, concerns or ideas, let us know in the comments below or connect with DVC team <a href="https://dvc.org/support">here</a>. Our <a href="https://twitter.com/DVCorg">DMs on Twitter</a> are always open, too.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=89df508fde42" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataversioncontrol/september-19-dvc-%EF%B8%8Fheartbeat-89df508fde42">September ’19 DVC❤️Heartbeat</a> was originally published in <a href="https://medium.com/dataversioncontrol">Data Version Control</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[July ’19 DVC❤️Heartbeat]]></title>
            <link>https://medium.com/dataversioncontrol/july-19-dvc-%EF%B8%8Fheartbeat-b178bf8454e7?source=rss----4e21975b393---4</link>
            <guid isPermaLink="false">https://medium.com/p/b178bf8454e7</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[version-control]]></category>
            <dc:creator><![CDATA[Svetlana Grinchenko]]></dc:creator>
            <pubDate>Thu, 01 Aug 2019 20:31:17 GMT</pubDate>
            <atom:updated>2019-08-01T20:31:16.821Z</atom:updated>
            <content:encoded><![CDATA[<p>Every month we are sharing here our news, findings, interesting reads, community takeaways, and everything along the way.</p><p>Some of those are related to our brainchild <a href="https://dvc.org/?source=post_page---------------------------">DVC </a>and its journey. The others are a collection of exciting stories and ideas centered around ML best practices and workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*adOL5Bjwvb23JBmKAKZy3A.png" /><figcaption><a href="https://twitter.com/rkuprieiev/status/1144298339200098306?s=20">Special edition DVC shirt.</a> We made this one for <a href="https://github.com/efiop">Ruslan</a> — DVC maintainer and the best tech lead.</figcaption></figure><h3>News and links</h3><p>As we continue to grow DVC together with our fantastic contributors, we enjoy more and more insights, discussions, and articles either created or brought to us by our community. We feel it is the right time to start sharing more of your news, your stories and your discoveries. New Heartbeat is coming soon!</p><p>Speaking of our own news — next month DVC team is going to the <a href="https://events.linuxfoundation.org/events/open-source-summit-north-america-2019/">Open Source North America Summit</a>. It is taking place in San Diego on August 21–23. <a href="https://ossna19.sched.com/speaker/dmitry35">Dmitry</a> and <a href="https://ossna19.sched.com/speaker/svetlanagrinchenko">Sveta</a> will be giving talks and we will run a booth. So looking forward to it! Stop by for a chat and some cool swag. And if you are in San Diego on those days and want to catch up — please let us know <a href="http://dvc.org/support">here</a> or on Twitter!</p><ul><li><a href="https://ossna19.sched.com/event/PUVv/open-source-tools-for-ml-experiments-management-dmitry-petrov-ruslan-kuprieiev-iterative-ai">Open Source Summit + ELC North America 2019: Open Source Tools for ML Experiments Man...</a></li><li><a href="https://ossna19.sched.com/event/PWNk/speaker-preparation-simple-steps-with-a-tremendous-impact-svetlana-grinchenko-dvcorg">Open Source Summit + ELC North America 2019: Speaker Preparation: Simple Steps with a...</a></li></ul><p>Every month our team is excited to discover new great pieces of content addressing some of the burning ML issues. Here are some of the links that caught our eye in June:</p><p><a href="https://dev.to/robogeek/principled-machine-learning-4eho"><strong>Principled Machine Learning: Practices and Tools for Efficient Collaboration</strong></a><strong> by </strong><a href="https://medium.com/u/d517e29ab56b"><strong>David Herron</strong></a></p><p><a href="https://dev.to/robogeek/principled-machine-learning-4eho">Principled Machine Learning: Practices and Tools for Efficient Collaboration</a></p><blockquote>As we’ve seen in this article some tools and practices can be borrowed from regular software engineering. However, the needs of machine learning projects dictate tools that better fit the purpose.</blockquote><p><strong>First </strong><a href="http://ml-repa.ru/"><strong>ML-REPA</strong></a><strong> </strong><a href="http://ml-repa.ru/page6697700.html"><strong>Meetup: Reproducible ML experiments</strong></a><strong> hosted by </strong><a href="https://www.raiffeisen-digital.ru/?utm_referrer="><strong>Raiffeisen DGTL</strong></a><strong> — check out the video and slide decks.</strong></p><p><a href="http://ml-repa.ru/">Machine Learning REPA</a></p><p><a href="http://ml-repa.ru/">ML-REPA</a> is an a new fantastic resource for Russian-speaking folks interested in Reproducibility, Experiments and Pipelines Automation. Curated by <a href="https://twitter.com/mnrozhkov">Mikhail Rozhkov</a> and highly recommended by our team.</p><p><a href="https://www.reddit.com/r/MachineLearning/comments/bx0apm/d_how_do_you_manage_your_machine_learning/"><strong>How do you manage your machine learning experiments?</strong></a><strong> discussion on Reddit is full of insights.</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=reddit&amp;url=https%3A//www.reddit.com/r/MachineLearning/comments/bx0apm/d_how_do_you_manage_your_machine_learning/&amp;image=" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/b898b23fca101162303e8b9b13618fe2/href">https://medium.com/media/b898b23fca101162303e8b9b13618fe2/href</a></iframe><h3>Discord gems</h3><p>There are lots of hidden gems in our Discord community discussions. Sometimes they are scattered all over the channels and hard to track down.</p><p>We are sifting through the issues and discussions and share with you the most interesting takeaways.</p><ul><li><strong>I have within one git repository different folders with very different content (basically different projects, or content I want to have different permissions to), and I thought about using different buckets in AWS as remotes. </strong><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/575718048330416158"><strong>I’m not sure if it’s possible with DVC to store some files in some remote, and some other files in some other remote, is it?</strong></a></li></ul><p>You can definitely add more than one remote (see <a href="https://dvc.org/doc/commands-reference/remote-add">dvc remote add</a>) and then <a href="https://dvc.org/doc/commands-reference/push">dvc push</a> has a -R option to pick which one to send the cached data files (deps, outs, etc) to. We would not recommend doing this though. It complicates the commands you have to run — you will need to remember to specify a remote name for every command that deals with data — push, pull, gc, fetch, status, etc. Please, leave a comment in the relevant issue <a href="https://github.com/iterative/dvc/issues/2095">here</a> if this case is important for you.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/578532350221352987"><strong>Is that possible with DVC to have multiple (few) metric files and compare them all at once?</strong></a><strong> For example, we’d like to consider as metrics the loss of a neural network training process (loss as a </strong><strong>-M output of a training stage), and also apart knowing the accuracy of the NN on a test set (another </strong><strong>-M output of eval stage).</strong></li></ul><p>Yes, it is totally fine to use -M in different stages. dvc metrics show will just show both metrics.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/577362750443880449"><strong>I have a scenario where an artifacts (data) folder is created by the </strong></a><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/577362750443880449"><strong>dvc run command via the </strong></a><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/577362750443880449"><strong>-o flag. I have manually added another file into or modified the artifacts folder but when I do </strong></a><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/577362750443880449"><strong>dvc push nothing happens, is there anyway around this?</strong></a></li></ul><p>Let’s first do a quick recap on how DVC handles data files (you can definitely find more information on the <a href="http://dvc.org/docs">DVC documentation site</a>).</p><ul><li>When you do dvc add, dvc run or dvc import DVC puts artifacts (in case of dvc run artifacts == outputs produced by the command) into .dvc/cache directory (default cache location). You don’t see this happening because <a href="https://dvc.org/doc/user-guide/large-dataset-optimization">DVC keeps links</a> (or in certain cases creates a copy) to these files/directories.</li><li>dvc push does not move files from the workspace (that what you see) to the remote storage, it always moves files/directories that are already in cache (default is .dvc/cache).</li><li>So, now you’ve added a file manually, or made some other modifications. But these files are not in cache yet. The analogy would be git commit. You change the file, you do git commit, only after that you can push something to Git server (Github/Gitlab, etc). The difference is that DVC is doing commit (moves files to cache) automatically in certain cases — dvc add, dvc run, etc.</li></ul><p>There is an explicit command — dvc commit-that you should run if you want to enforce the change to the output produced by dvc run. This command will update the corresponding DVC- files (.dvc extension) and will move data to cache. After that you should be able to run dvc push to save your data on the external storage.</p><p>Note, when you do an explicit commit like this you are potentially “breaking” the reproducibility. In a sense that there is no guarantee now that your directory can be produced by dvc run/dvc repro — since you changed it manually.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/578898899469729796"><strong>I’d like to transform my dataset in-place to avoid copying it, but I can’t use </strong></a><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/578898899469729796"><strong>dvc run to do this because it doesn’t allow the same directory as an output and a dependency.</strong></a></li></ul><p>You could do this in one step (one stage). So that getting your data and modifying it, is one stage. So you don’t depend on the data folder. You just could depend on your download + modifying script.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/579283950778712076"><strong>Can anyone tell me what this error message is about?</strong></a><strong> </strong><strong>“To avoid unpredictable behaviour, rerun command with non overlapping outs paths.”</strong></li></ul><p>Most likely it means that there is a DVC-file that have the same output twice. Or there two DVC-files that share the same output file.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/580176327701823498"><strong>I’m getting “No such file or directory” error when I do </strong></a><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/580176327701823498"><strong>dvc run or </strong></a><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/580176327701823498"><strong>dvc repro</strong></a><strong>. The command runs find if I don’t use DVC.</strong></li></ul><p>That happens because dvc run is trying to ensure that your command is the one creating your output and removes existing outputs before executing the command. So that when you run dvc repro later, it will be able to fully reproduce the output. So you need to make the script create the directory or file.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/581256265234251776"><strong>I’m implementing a CI/CD and I would like to simplify my CI/CD or even my training code (keeping them cloud agnostic) by using </strong></a><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/581256265234251776"><strong>dvc pull inside my Docker container when initializing a training job. </strong></a><strong>Can DVC be used in this way?</strong></li></ul><p>Yes, it’s definitely a valid case for DVC. There are different ways of organizing the storage that training machines are using to access data. From the very simple — using local storage volume and pulling data from the remote storage everytime — to using NAS or EFS to store a shared DVC cache.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/598866528984891403">I was abl<strong>e to follow the getting started examples, however now I am trying to push my data to Github, I keep getting the following error: </strong></a><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/598866528984891403"><strong>“ERROR: failed to push data to the cloud — upload is not supported by https remote”.</strong></a></li></ul><p>HTTP remotes do not support upload yet. Example Get Started repository is using HTTP to keep it read-only and abstract the actual storage provider we are using internally. If you actually check the remote URL, you should see that it is an S3 bucket and AWS provides an HTTP end-point to read data from it.</p><ul><li><strong>I’m looking to configure AWS S3 as a storage for DVC. I’ve set up the remotes and initialized dvc in the git repository. I tried testing it by pushing a dataset in the form of an excel file. The command completed without any issues but this is what I’m seeing in S3. </strong><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/585967551708921856"><strong>DVC seems to have created a subdirectory in the intended directory called “35” where it placed this file with a strange name.</strong></a></li></ul><p>This is not an issue, it is an implementation detail. There’s no current way to upload the files with the original filename (In this case, the S3 bucket will have the file data.csv but with another name 20/893143…). The reason behind this decision is because we want to store a file only once no matter how many dataset versions it’s used in. Also, it’s a reliable way to uniquely identify the file. You don’t have to be afraid that someone decided to create a file with the same name (path) but a different content.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/587730054893666326"><strong>Is it possible to only have a shared ‘local’ cache and no remote?</strong></a><strong> I’m trying to figure out how to use this in a 40 node cluster which already has very fast NFS storage across all the nodes. Not storing everything twice seems desirable. Esp. for the multi-TB input data</strong></li></ul><p>Yes and it’s one of the very common use case, actually. All you need to do is to use dvc cache dir command to setup an external cache. There are few caveats though. Please, read <a href="https://discuss.dvc.org/t/share-nas-data-in-server/180/4?u=shcheklein">this link</a> for an example of the workflow.</p><p>If you have any questions, concerns or ideas, let us know in the comments below or connect with DVC team <a href="https://dvc.org/support">here</a>. Our <a href="https://twitter.com/DVCorg">DMs on Twitter</a> are always open, too.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b178bf8454e7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataversioncontrol/july-19-dvc-%EF%B8%8Fheartbeat-b178bf8454e7">July ’19 DVC❤️Heartbeat</a> was originally published in <a href="https://medium.com/dataversioncontrol">Data Version Control</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[June ’19 DVC❤️Heartbeat]]></title>
            <link>https://medium.com/dataversioncontrol/june-19-dvc-%EF%B8%8Fheartbeat-505900be9aac?source=rss----4e21975b393---4</link>
            <guid isPermaLink="false">https://medium.com/p/505900be9aac</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[versioning]]></category>
            <category><![CDATA[pycon]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Svetlana Grinchenko]]></dc:creator>
            <pubDate>Tue, 25 Jun 2019 22:17:36 GMT</pubDate>
            <atom:updated>2019-06-25T22:29:03.460Z</atom:updated>
            <content:encoded><![CDATA[<p>Every month we are sharing here our news, findings, interesting reads, community takeaways, and everything along the way.</p><p>Some of those are related to our brainchild <a href="https://dvc.org/">DVC </a>and its journey. The others are a collection of exciting stories and ideas centered around ML best practices and workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xkHR-kERe16cVk-YvUYjbw.png" /><figcaption>Thanks to the amazing <a href="https://twitter.com/SignaturitTech">Signaturit Tech</a> team for <a href="https://twitter.com/SignaturitTech/status/1127927520140120065?s=20">this photo</a>!</figcaption></figure><h3>News and links</h3><p>We want to start by saying to our users, contributors, and community members how grateful we are for the fantastic work you are doing contributing to DVC, giving talks about DVC, sharing your feedback, use cases and your concerns. A huge thank you to each of you from the DVC team!</p><p>We would love to give back and support any positive initiative around DVC — just let us know <a href="https://dvc.org/support">here</a> and we will send you a bunch of cool swag, connect to a tech expert or find another way to support your project. Our <a href="https://twitter.com/DVCorg">DMs on Twitter</a> are open, too.</p><p><strong>And if you have 4 minutes to spare, we are conducting out first </strong><a href="https://docs.google.com/forms/d/1tmn8YHLUkeSi5AIq4DGJi28iZy9HTazl6DWKe3Hxpnc/edit?ts=5cfc47c2"><strong>DVC user survey</strong></a><strong> and would love to hear from you!</strong></p><p>Aside from admiring great DVC-related content from our users we have one more reason to particularly enjoy the past month — DVC team went to Cleveland to attend <a href="https://us.pycon.org/2019/about/">Pycon 2019</a> and it was a blast!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bbj7FkUcfcEqf6tyE9Y8Fw.jpeg" /><figcaption>Amazing <a href="https://github.com/sureL">Jennifer</a> and her artwork for our <a href="https://twitter.com/hashtag/SupportOpenSource?src=hashtag_click">#SupportOpenSource</a> contest</figcaption></figure><p>We had it all. Running our first ever conference booth, leading an impromptu unconference discussion and arranging some cool <a href="https://twitter.com/hashtag/SupportOpenSource?src=hashtag_click">#SupportOpenSource</a> activities was great! Last-minute accommodation cancellations, booth equipment delivery issues, and being late for our very own talk was not so great. Will be sharing more about it in a separate blogpost soon.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fjkfh2PM5Sz8%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Djkfh2PM5Sz8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fjkfh2PM5Sz8%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f86f17e8cc4cc2ce1f889478835a0e27/href">https://medium.com/media/f86f17e8cc4cc2ce1f889478835a0e27/href</a></iframe><p>Here is <a href="https://medium.com/u/2b1c8f1b5d10">Dmitry Petrov</a>’s Pycon <a href="https://www.youtube.com/watch?v=jkfh2PM5Sz8">talk</a> and <a href="https://docs.google.com/presentation/d/1CYt0w8WoZAXiQEtVDVDsTnQumzdZx91v32MwEK20R-E/edit">slides</a> on Machine learning model and dataset versioning practices.</p><p>We absolutely loved being at Pycon and can’t wait for our next conference!</p><p>Our team is so happy every time we discover an article featuring DVC or addressing one of the burning ML issues we are trying to solve. Here are some of the links that caught our eye past month:</p><p><a href="https://towardsdatascience.com/the-rise-of-dataops-from-the-ashes-of-data-governance-da3e0c3ac2c4"><strong>The Rise of DataOps (from the ashes of Data Governance)</strong></a><strong> by </strong><a href="https://towardsdatascience.com/@ryanwgross"><strong>Ryan Gross</strong></a><strong>.</strong></p><p>A brilliant comprehensive read on the current data management issues. It might be the best article we have ever read on this subject. Every word strongly resonates with our vision and ideas behind DVC. Highly recommended by DVC team!</p><p><a href="https://towardsdatascience.com/the-rise-of-dataops-from-the-ashes-of-data-governance-da3e0c3ac2c4">The Rise of DataOps (from the ashes of Data Governance)</a></p><blockquote>Legacy Data Governance is broken in the ML era. Let’s rebuild it as an engineering discipline…</blockquote><blockquote>…At the end of the transformation, data governance will look a lot more like DevOps, with data stewards, scientists, and engineers working closely together to codify the governance policies.</blockquote><p><a href="https://medium.com/@christopher.samiullah/first-impressions-of-data-science-version-control-dvc-fe96ab29cdda"><strong>First Impressions of Data Science Version Control (DVC)</strong></a><strong> by </strong><a href="https://christophergs.github.io/"><strong>Christopher Samiullah</strong></a></p><p><a href="https://medium.com/@christopher.samiullah/first-impressions-of-data-science-version-control-dvc-fe96ab29cdda">First Impressions of Data Science Version Control (DVC)</a></p><blockquote>In 2019, we tend to find organizations using a mix of git, Makefiles, ad hoc scripts and reference files to try and achieve reproducibility. DVC enters this mix offering a cleaner solution, specifically targeting Data Science challenges.</blockquote><p><a href="https://github.com/peopledoc/mlv-tools-tutorial"><strong>Versioning and Reproducibility with MLV-tools and DVC</strong></a><strong>: </strong><a href="https://peopledoc.github.io/mlv-tools-tutorial/talks/pyData/presentation.html#/"><strong>Talk</strong></a><strong> and </strong><a href="https://peopledoc.github.io/mlv-tools-tutorial/talks/workshop/presentation.html#/"><strong>Tutorial</strong></a><strong> by </strong><a href="https://github.com/sbracaloni"><strong>Stéphanie Bracaloni</strong></a><strong> and </strong><a href="https://github.com/SdgJlbl"><strong>Sarah Diot-Girard</strong></a><strong>.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wJ9OtAjZ2pv1wpKWwhogPg.png" /></figure><p><a href="https://www.oreilly.com/ideas/becoming-a-machine-learning-company-means-investing-in-foundational-technologies"><strong>Becoming a machine learning company means investing in foundational technologies</strong></a><strong> by </strong><a href="https://www.oreilly.com/people/4e7ad-ben-lorica"><strong>Ben Lorica</strong></a></p><p><a href="https://www.oreilly.com/ideas/becoming-a-machine-learning-company-means-investing-in-foundational-technologies">Becoming a machine learning company means investing in foundational technologies</a></p><blockquote>With an eye toward the growing importance of machine learning, we recently completed <a href="https://www.oreilly.com/data/free/evolving-data-infrastructure.csp">a data infrastructure survey</a> that drew more than 3,200 respondents.</blockquote><h4>Discord gems</h4><p>There are lots of hidden gems in our Discord community discussions. Sometimes they are scattered all over the channels and hard to track down.</p><p>We are sifting through the issues and discussions and share with you the most interesting takeaways.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/575655655629651968"><strong>Does DVC support Azure Data Lake Gen1?</strong></a></li></ul><p>Azure data lake is HDFS compatible. And DVC supports HDFS remotes. Give it a try and let us know if you hit any problems <a href="https://dvc.org/chat">here</a>.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/575681811401801748"><strong>An excellent discussion on versioning tabular (SQL) data.</strong></a><strong> Do you know of any tools that deal better with SQL-specific versioning?</strong></li></ul><p>It’s a wide topic. The actual solution might depend on a specific scenario and what exactly needs to be versioned. DVC does not provide any special functionality on top of databases to version their content.</p><p>Depending on your use case, our recommendation would be to run SQL and pull the result file (CSV/TSV file?) that then can be used to do analysis. This file can be taken under DVC control. Alternatively, in certain cases source files (that are used to populate the databases) can be taken under control and we can keep versions of them, or track incoming updates.</p><p>Read the <a href="https://discordapp.com/channels/485586884165107732/563406153334128681/575681811401801748">discussion</a> to learn more.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/575686711821205504"><strong>How does DVC do the versioning between binary files?</strong></a><strong> Is there a binary diff, similar to git? Or is every version stored distinctly in full?</strong></li></ul><p>DVC is just saving every file as is, we don’t use binary diffs right now. There won’t be a full directory (if you added just a few files to a 10M files directory) duplication, though, since we treat every file inside as a separate entity.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/576160840701575169"><strong>Is there a way to pass parameters from e.g. </strong></a><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/576160840701575169"><strong>dvc repro to stages?</strong></a></li></ul><p>The simplest option is to create a config file — json or whatnot — that your scripts would read and your stages depend on.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/577852740034625576"><strong>What is the best way to get cached output files from different branches simultaneously?</strong></a><strong> For example, cached tensorboard files from different branches to compare experiments.</strong></li></ul><p>There is a way to do that through our (still not officially released) API pretty easily. Here is an <a href="https://cdn.discordapp.com/attachments/563406153334128681/577894682722304030/dvc_get_output_files.py">example script</a> how it could be done.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/563406153334128681/583949033685516299"><strong>Docker and DVC.</strong></a><strong> To being able to push/pull data we need to run a git clone to get DVC-files and remote definitions — but we worry that would make the container quite heavy (since it contains our entire project history).</strong></li></ul><p>You can do git clone — depth 1, which will not download any history except the latest commits.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/574133734136086559"><strong>After DVC pushing the same file, it creates multiple copies of the same file. Is that how it’s supposed to work?</strong></a></li></ul><p>If you are pushing the same file, there are no copies pushed or saved in the cache. DVC is using checksums to identify files, so if you add the same file once again, it will detect that cache for it is already in the local cache and wont copy it again to cache. Same with dvc push, if it sees that you already have cache file with that checksum on your remote, it won’t upload it again.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/574941227624169492"><strong>How do I uninstall DVC on Mac (installed via pkg installer)?</strong></a></li></ul><p>Something like this should work:</p><pre>$ which dvc<br>/usr/local/bin/dvc -&gt; /usr/local/lib/dvc/dvc</pre><pre>$ ls -la /usr/local/bin/dvc<br>/usr/local/bin/dvc -&gt; /usr/local/lib/dvc/dvc</pre><pre>sudo rm -f /usr/local/bin/dvc<br>sudo rm -rf /usr/local/lib/dvc<br>sudo pkgutil --forget com.iterative.dvc</pre><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/575236576309674024"><strong>How do I pull from a public S3 bucket (that contains DVC remote)?</strong></a></li></ul><p>Just add public URL of the bucket as an HTTP endpoint. See <a href="https://github.com/iterative/example-get-started/blob/master/.dvc/config">here</a> for an example. <a href="https://remote.dvc.org/get-started">https://remote.dvc.org/get-started</a> is made to redirect to the S3 bucket anyone can read from.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/575535709490905101"><strong>I’m getting the same error over and over about locking:</strong></a><strong> </strong><strong>ERROR: failed to lock before running a command — cannot perform the cmd since DVC is busy and locked. Please retry the command later.</strong></li></ul><p>Most likely it happens due to an attempt to run DVC on NFS that has some configuration problems. There is a <a href="https://github.com/iterative/dvc/issues/1918">well known problem with DVC on NFS</a> — sometimes it hangs on trying to lock a file. The usual workaround for this problem is to allocate DVC cache on NFS, but run the project (git clone, DVC metafiles, etc) on the local file system. Read <a href="https://discuss.dvc.org/t/share-nas-data-in-server/180/4?u=shcheklein">this answer</a> to see how it can be setup.</p><p>If you have any questions, concerns or ideas, let us know in the comments below or connect with DVC team <a href="https://dvc.org/support">here</a>. Our <a href="https://twitter.com/DVCorg">DMs on Twitter</a> are open, too.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=505900be9aac" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataversioncontrol/june-19-dvc-%EF%B8%8Fheartbeat-505900be9aac">June ’19 DVC❤️Heartbeat</a> was originally published in <a href="https://medium.com/dataversioncontrol">Data Version Control</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[May ’19 DVC❤️Heartbeat]]></title>
            <link>https://medium.com/dataversioncontrol/may-19-dvc-%EF%B8%8Fheartbeat-cf0542905dea?source=rss----4e21975b393---4</link>
            <guid isPermaLink="false">https://medium.com/p/cf0542905dea</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[version-control]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[reproducibility]]></category>
            <dc:creator><![CDATA[Svetlana Grinchenko]]></dc:creator>
            <pubDate>Tue, 21 May 2019 00:00:20 GMT</pubDate>
            <atom:updated>2019-05-21T00:13:33.038Z</atom:updated>
            <content:encoded><![CDATA[<p>Every month we are sharing here our news, findings, interesting reads, community takeaways, and everything along the way.</p><p>Some of those are related to our brainchild <a href="https://dvc.org/">DVC </a>and its journey. The others are a collection of exciting stories and ideas centered around ML best practices and workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Gu_QCq8U9raem53RK4s7g.jpeg" /><figcaption>Kudos to <a href="https://www.stickermule.com/">StickerMule.com</a> for our amazing stickers (and great customer service)!</figcaption></figure><h3>News and links</h3><p>This section of DVC Heartbeat is growing with every new Issue and this is already quite a good piece of news!</p><p>One of the most exciting things we want to share this month is acceptance of DVC into the <a href="https://developers.google.com/season-of-docs/">Google Season of Docs</a>. It is a new and unique program sponsored by Google that pairs technical writers with open source projects to collaborate and improve the open source project documentation. You can find the outline of DVC vision and project ideas in <a href="https://blog.dataversioncontrol.com/dvc-project-ideas-for-google-summer-of-docs-2019-defe3a73b248">this dedicated blogpost</a> and check the <a href="https://developers.google.com/season-of-docs/docs/participants/">full list of participating open source organisations</a>. Technically the <a href="https://developers.google.com/season-of-docs/docs/timeline">program is starting in a few months</a>, but there is already a fantastic increase in the amount of commits and contributors, and we absolutely love it!</p><p>The other important milestone for us was the first offline meeting with our distributed remote team. Working side by side and having non-Zoom meetings with the team was amazing. Joining our forces to prepare for the upcoming conferences turned out to be the most valuable, educating and uniting experience for the whole team.</p><p>It’s a shame that our techlead was unable to join us it due to another visa denial. We do hope he will finally make it to the USA for the next big conference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0-eBHbQp_flE53tAxWSAXw.png" /></figure><p>While we were busy finalizing all the Pycon 2019 prep, our own <a href="https://medium.com/u/2b1c8f1b5d10">Dmitry Petrov</a> flew to New York to speak at the <a href="https://conferences.oreilly.com/artificial-intelligence/ai-ny">O’Reilly AI Conference</a> about the <a href="https://www.oreilly.com/library/view/artificial-intelligence-conference/9781492050544/video324691.html">Open Source tools for Machine Learning Models and Datasets versioning</a>. Unfortunately the video is available for the registered users only (with a free trial option) but you can have a look at Dmitry’s slides <a href="https://www.slideshare.net/DmitryPetrov15/dvc-oreilly-artificial-intelligence-conference-2019-new-york">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/1*DyyuYU7ZcCEkjFNUN26YLQ.png" /></figure><p>We renamed our Twitter! Our old handle was a bit misleading and we moved from @Iterativeai to @DVCorg (yet keep the old one for future projects).</p><p>Our team is so happy every time we discover an article featuring DVC or addressing one of the burning ML issues we are trying to solve. Here are some of our favorite links from the past month:</p><p><a href="https://www.pythonpodcast.com/data-version-control-episode-206/"><strong>Version Control For Your Machine Learning Projects — Episode 206</strong></a><strong> by </strong><a href="https://www.linkedin.com/in/tmacey/"><strong>Tobias Macey</strong></a></p><p><a href="https://www.pythonpodcast.com/data-version-control-episode-206/">Version Control For Machine Learning Projects</a></p><blockquote>Version control has become table stakes for any software team, but for machine learning projects there has been no good answer for tracking all of the data that goes into building and training models, and the output of the models themselves. To address that need Dmitry Petrov built the Data Version Control project known as DVC. In this episode he explains how it simplifies communication between data scientists, reduces duplicated effort, and simplifies concerns around reproducing and rebuilding models at different stages of the projects lifecycle.</blockquote><p>Here is an <a href="https://towardsdatascience.com/data-version-control-with-dvc-what-do-the-authors-have-to-say-3c3b10f27ee">article</a> by <a href="https://medium.com/u/e8ec6fa4d7d4">Favio Vázquez</a> with a transcript of this podcast episode.</p><p><a href="https://towardsdatascience.com/data-version-control-with-dvc-what-do-the-authors-have-to-say-3c3b10f27ee">Data version control with DVC. What do the authors have to say?</a></p><p><a href="https://towardsdatascience.com/why-git-and-git-lfs-is-not-enough-to-solve-the-machine-learning-reproducibility-crisis-f733b49e96e8"><strong>Why Git and Git-LFS is not enough to solve the Machine Learning Reproducibility crisis</strong></a> followed up by a comprehensive <a href="https://www.reddit.com/r/MachineLearning/comments/bjszua/r_why_git_and_gitlfs_is_not_enough_to_solve_the/"><strong>discussion on Reddit</strong></a>.</p><p><a href="https://towardsdatascience.com/why-git-and-git-lfs-is-not-enough-to-solve-the-machine-learning-reproducibility-crisis-f733b49e96e8">Why Git and Git-LFS is not enough to solve the Machine Learning Reproducibility crisis</a></p><blockquote>With Git-LFS your team has better control over the data, because it is now version controlled. Does that mean the problem is solved? Earlier we said the “<em>key issue is the training data</em>”, but that was a lie. Sort of. Yes keeping the data under version control is a big improvement. But is the lack of version control of the data files the entire problem? No.</blockquote><h4>Discord gems</h4><p>There are lots of hidden gems in our Discord community discussions. Sometimes they are scattered all over the channels and hard to track down.</p><p>We are sifting through the issues and discussions and share with you the most interesting takeaways.</p><ul><li><strong>This might be </strong><a href="https://discordapp.com/channels/485586884165107732/485598848111083531/572960640122224640"><strong>a favourite gem of ours </strong></a><strong>— our engineers are so fast that someone assumed they were bots.</strong></li></ul><p>We feared that too until we met them in person. They appeared to be real (unless bots also love Ramen now)!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eWXlHdHpYncESpQWSVa54A.png" /></figure><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/572974117351849997"><strong>Is this the best way to track data with DVC when code and data are separate?</strong></a><strong> Having being burned by this a couple of times, i.e accidentally pushing large files to GitHub, I now keep my code and data separate</strong></li></ul><p>Every time you run dvc add to start tracking some data artifact, its path is automatically added to the .gitignore file, as a result it is hard to commit it to git by mistake — you would need to explicitly modify the .gitignore first. The feature to track some external data is called <a href="https://dvc.org/doc/user-guide/external-outputs">external outputs</a> (if all you need is to track some data artifacts). Usually it is used when you have some data on S3 or SSH and don’t want to pull it into your working space, but it’s working even when your data is located on the same machine outside of the repository.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/571342592508428289"><strong>How do I wrap a step that downloads a file/directory into a DVC stage?</strong></a><strong> I want to ensure that it runs only if file has no been downloaded yet</strong></li></ul><p>Use dvc import to track and download the remote data first time and next time when you do dvc repro if data has changed remotely. If you don’t want to track remote changes (lock the data after it was downloaded), use dvc run with a dummy dependency (any text file will do you do not touch) that runs an actual wget/curl to get the data.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/570943786151313408"><strong>How do I show a pipeline that does not have a default </strong></a><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/570943786151313408"><strong>Dvcfile?</strong></a><strong> (e.g. I assigned all files names manually with </strong><strong>-f in the </strong><strong>dvc run command and I just don’t have </strong><strong>Dvcfile anymore)</strong></li></ul><p>Almost any command in DVC that deals with pipelines (set of DVC-files) accepts a single stage as a target, for example dvc pipeline show — ascii model.dvc.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/570843482218823682"><strong>DVC hangs or I’m getting “database is locked” issue</strong></a></li></ul><p>It’s a well known problem with NFS, CIFS (Azure) — they do not support file locks properly which is required by the SQLLite engine to operate. The easiest workaround — don’t create a DVC project on network attached partition. In certain cases a fix can be made by changing mounting options, check <a href="https://discordapp.com/channels/485586884165107732/485596304961962003/570276668694855690">this discussion</a> for the Azure ML Service.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/570091809594671126">H<strong>ow do I use DVC if I use a separate drive to store the data</strong></a><strong> (and a small/fast SSD to run computations)? I don’t have enough space to bring data to my working space</strong></li></ul><p>An excellent question! The short answer is:</p><p>dvc cache dir --local — to move your data to a big partition;</p><p>dvc config cache.type reflink, hardlink, symlink, copy — to enable symlinks to avoid actual copying;</p><p>dvc config cache.protected true — it’s highly recommended to make links in your working space read-only to avoid corrupting the cache;</p><p>To add your data first time to the DVC cache, do a clone of the repository on a big partition and run dvc add to add your data. Then you can do git pull, dvc pull on a small partition and DVC will create all the necessary links.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/571335064374345749"><strong>Why I’m getting `Paths for outs overlap` error when I run dvc add or dvc run?</strong></a></li></ul><p>Usually it means that a parent directory of one of the arguments for dvc add / dvc run is already tracked. For example, you’ve added the whole datasets directory already. And now you are trying to add a subdirectory, which is already tracked as a part of the datasets one. No need to do that. You could dvc add datasets or dvc repro datasets.dvc to save changes.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/567310354766495747"><strong>I’m getting `ascii codec can’t encode character` error on DVC commands when I deal with unicode file names</strong></a></li></ul><p><a href="https://perlgeek.de/en/article/set-up-a-clean-utf8-environment">Check the locale settings you have</a> (locale command in Linux). Python expects a locale that can handle unicode printing. Usually it’s solved with these commands: export LC_ALL=en_US.UTF-8 and export LANG=en_US.UTF-8. You can place those exports into .bashrc or other file that defines your environment.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/563149775340568576"><strong>Does DVC use the same logins aws-cli has when using an S3 bucket as its repo/remote storage</strong></a><strong>?</strong></li></ul><p>In short — yes, but it can be also configured. DVC is going to use either your default profile (from ~/.aws/*) or your env vars by default. If you need more flexibility (e.g. you need to use different credentials for different projects, etc) check out <a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html">this guide</a> to configure custom aws profiles and then you could use them with DVC using these <a href="https://dvc.org/doc/commands-reference/remote-add#options">remote options</a>.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/566000729505136661"><strong>How can I output multiple metrics from a single file?</strong></a><strong> Let’s say I have the following in a file: </strong><strong>{“AUC_RATIO”: {“train”: 0.8922748258797667, “valid”: 0.8561602726251776, “xval”: 0.8843431199314923}}. How can I show both </strong><strong>train and </strong><strong>valid without </strong><strong>xval?</strong></li></ul><p>You can use metrics show command XPath option and provide multiple attribute names to it:</p><pre>$ dvc metrics add model.metrics --type json --xpath AUC_RATIO[train,valid]<br>metrics.json:<br>             0.89227482588<br>             0.856160272625</pre><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/566314479499870211"><strong>What is the quickest way to add a new dependency to a DVC-file?</strong></a></li></ul><p>There are a few options to add a new dependency:</p><ul><li>simply opening a file with your favorite editor and adding a dependency there without md5. DVC will understand that that stage is changed and will re-run and re-calculate md5 checksums during the next dvc repro;</li><li>use dvc run --no-exec is another option. It will rewrite the existing file for you with new parameters.</li><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/566315265646788628"><strong>Is there a way to add a dependency to a python package, so it runs a stage again if it imported the updated library?</strong></a></li></ul><p>The only recommended way so far would be to somehow make DVC know about your package’s version. One way to do that would be to create a separate stage that would be dynamically printing version of that specific package into a file, that your stage would depend on:</p><pre>dvc run -o mypkgver &#39;pip show mypkg &gt; mypkgver’<br>dvc run -d mypkgver -d ... -o .. mycmd</pre><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/564807276146458624"><strong>Is there anyway to forcibly recompute the hashes of dependencies in a pipeline DVC-file?</strong></a><strong> E.g. I made some whitespace/comment changes in my code and I want to tell dvc “it’s ok, you don’t have to recompute everything”.</strong></li></ul><p>Yes, you could dvc commit -f. It will save all current checksum without re-running your commands.</p><ul><li><strong>I</strong><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/563352000281182218"><strong> have projects that use data that’s stored in S3. I never have data locally to use dvc push, but I would like to have this data version controlled.</strong></a><strong> Is there a way to use the features of DVC in this use case?</strong></li></ul><p>Yes! This DVC features is called <a href="https://dvc.org/doc/user-guide/external-outputs">external outputs</a> and <a href="https://dvc.org/doc/user-guide/external-dependencies">external dependencies</a>. You can use one of them or both to track, process, and version your data on a cloud storage without downloading it locally.</p><p>If you have any questions, concerns or ideas, let us know <a href="https://dvc.org/support">here</a> and our stellar team will get back to you in no time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf0542905dea" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataversioncontrol/may-19-dvc-%EF%B8%8Fheartbeat-cf0542905dea">May ’19 DVC❤️Heartbeat</a> was originally published in <a href="https://medium.com/dataversioncontrol">Data Version Control</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DVC project ideas for Google Summer of Docs 2019]]></title>
            <link>https://medium.com/dataversioncontrol/dvc-project-ideas-for-google-summer-of-docs-2019-defe3a73b248?source=rss----4e21975b393---4</link>
            <guid isPermaLink="false">https://medium.com/p/defe3a73b248</guid>
            <category><![CDATA[gsod]]></category>
            <category><![CDATA[documentation]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Svetlana Grinchenko]]></dc:creator>
            <pubDate>Tue, 23 Apr 2019 19:45:39 GMT</pubDate>
            <atom:updated>2019-04-24T21:02:47.932Z</atom:updated>
            <content:encoded><![CDATA[<h3>DVC project ideas for Google Season of Docs 2019</h3><p><a href="http://www.dvc.org/">DVC.org</a> is applying for <a href="https://developers.google.com/season-of-docs/">Google Season of Docs</a> — a new and unique program sponsored by Google that pairs technical writers with open source projects to collaborate on the open source project documentation.</p><p>It’s happening for the first time in 2019 and we are excited about the opportunity to be a part of it!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JnvVcFtA133QguX1lUDHuw.png" /></figure><p>We strongly believe that well-shaped documentation is key for making the product truly open. We have been investing lots of time and energy in improving our docs lately. Being a team of 90% engineers we are eager to welcome the writers into our team and our community. We are happy to share our experience, introduce them to the world of open source and machine learning best practices, guide through the OS contribution process and work together on improving our documentation.</p><p>DVC was started in late 2017 by a data scientist and an engineer. It is now growing pretty fast and though our in-house team is quite small, we have to thank our contributors (more than 80 in both code and docs) for developing DVC with us. When working with DVC the technical writer will not only get lots of hands-on experience in writing technical docs, but will also immerse into DVC community — a warm and welcoming gathering of ML and DS enthusiasts and an invaluable source of inspiration and expertise in ML engineering.</p><h4><strong>About DVC</strong></h4><p>DVC is a brainchild of a data scientist and an engineer, that was created to fill in the gaps in the ML processes tooling and evolved into a successful open source project.</p><p>ML brings changes in development and research processes. These ML processes require new tools for data versioning, ML pipeline versioning, resource management for model training and others that haven’t been formalized. The traditional software development tools do not fully cover ML team’s needs but there are no good alternatives. It makes engineers to custom develop a new toolset to manage data files, keep track of ML experiments and connect data and source code together. The ML process becomes very fragile and requires tons of tribal knowledge.</p><p>We have been working on <a href="http://dvc.org/">DVC</a> by adopting best ML practices and turning them into Git-like command line tool. DVC versions multi-gigabyte datasets and ML models, make them shareable and reproducible. The tool helps to organize a more rigorous process around datasets and the data derivatives. Your favorite cloud storage (S3, GCS, or bare metal SSH server) could be used with DVC as a data file backend.</p><p>If you are interested in learning a little bit more about DVC and its journey, here is a great interview with DVC creator in the Episode 206 of Podcast.__init__. Listen to it <a href="https://www.pythonpodcast.com/data-version-control-episode-206/">HERE </a>or read the transcript <a href="https://towardsdatascience.com/data-version-control-with-dvc-what-do-the-authors-have-to-say-3c3b10f27ee">HERE.</a></p><h4><strong>The state of DVC documentation</strong></h4><p>DVC is a pretty young project, developed and maintained solely by engineers. As many OS projects we started from the bottom and for a long time our <a href="https://dvc.org/doc">documentation</a> was a bunch of bits and pieces. Nowadays improving documentation is one of our top priorities. We moved to the new in-house built documentation engine and started working with several technical writers. Certain parts have been tremendously improved recently, e.g. <a href="https://dvc.org/doc/get-started">Get Started</a> and <a href="https://dvc.org/doc/commands-reference/fetch">certain parts of Commands Reference</a> . So far most of our documentation has been written majorly by the engineering team and there is need for improving the overall structure and making some parts more friendly from a new user perspective. We have mostly complete <a href="https://dvc.org/doc/commands-reference">reference documentation</a> for each command, although some functions are missing good actionable examples. We also have a <a href="https://dvc.org/doc/user-guide/dvc-files-and-directories">User Guide</a>, however it is not in very good shape. We strive for making our documentation clear and comprehensive for users of various backgrounds and proficiency levels and this is where we do need some fresh perspective.</p><h4><strong>How DVC documentation is built</strong></h4><p>We have an open Github Apache-2 licensed repository for the <a href="https://github.com/iterative/dvc.org">DVC website</a>, the documentation engine and the <a href="https://github.com/iterative/dvc.org">documentation files</a>. The website is built with Node.js + React, including the documentation engine (built in-house).</p><p>Each documentation page is a static Markdown file in the repository, e.g. <a href="https://github.com/iterative/dvc.org/blob/master/static/docs/get-started/example-versioning.md.">example here</a>. It is rendered dynamically in the browser, no preprocessing is required. It means that tech writers or contributors need to write/edit a Markdown file, create a pull request and merge it into the master branch of the <a href="https://github.com/iterative/dvc.org">repository.</a> The complete <a href="https://github.com/iterative/dvc.org/blob/master/README.md#contributing">documentation contributing guide</a> describes the directory structure and locations for the different documentation parts.</p><h4><strong>DVC’s approach to documentation work</strong></h4><p>Documentation tasks and issues are maintained on our doc’s GitHub <a href="https://github.com/iterative/dvc.org/issues">issue tracker</a>. Changes to the documentation are made via pull requests on GitHub, and go through our standard review process which is the same for documentation and code. A technical writer would be trained in working with our current development process. It generally means that tech writers or contributors need to write/edit a Markdown file, use git and Github to create a pull request and publish it. The documentation <a href="https://github.com/iterative/dvc.org/blob/master/README.md#contributing">contributing guide</a> includes style conventions and other details. Documentation is considered of the same importance as code. Engineering team has a policy to write or update the relevant sections if something new is released. If it’s something too involved engineers may create a ticket and ask for help. There is one maintainer who is responsible for doing final reviews and merging the changes. In this sense, our documentation is very similar to any other open source project.</p><h3><strong>Project ideas for GSoD’19</strong></h3><p>We identified a number of ideas to work on and there are two major topics these ideas fall into. Both topics are pretty broad and we don’t expect we can completely cover them during this GSoD but hopefully we can make certain progress.</p><p>First of all, we want to bring more structure and logic to our documentation to improve user onboarding experience. The goal is for a new user to have a clear path they can follow and understand what takeaways each part of the documentation provides. In particular, improving how <a href="https://dvc.org/doc/get-started">Get Started</a>, <a href="https://dvc.org/doc/tutorial">Tutorials</a> and <a href="https://dvc.org/doc/get-started/example-versioning">Examples</a> relate to each other, restructuring the existing <a href="https://dvc.org/doc/user-guide">User Guide</a> to explain basic concepts, and writing more use cases that resonate with ML engineers and data scientists.</p><p>The other issue we would like to tackle is improving and expanding the existing reference docs — commands descriptions, examples, etc. It involves filling in the gaps and developing new sections, similar to <a href="https://dvc.org/doc/commands-reference/fetch">this one</a>. We would also love to see more illustrative materials.</p><h4><strong>Project 1: Improving and expanding User Guide</strong></h4><p><strong>Description and details: </strong>Reviewing, restructuring and filling major gaps in the User Guide (introductory parts of the basic concepts of DVC), e.g. have a look at <a href="https://github.com/iterative/dvc.org/issues/144">this ticket</a> or <a href="https://github.com/iterative/dvc.org/issues/53">this one</a>.</p><p><strong>Mentors: </strong><a href="https://github.com/shcheklein">@shcheklein</a> and <a href="https://github.com/dmpetrov">@dmpetrov</a></p><h4><strong>Project 2: Expanding and developing new tutorials and use cases.</strong></h4><p><strong>Description and details: </strong>We already have some requests for more tutorials, e.g. <a href="https://github.com/iterative/dvc.org/issues/96">this ticket</a>. Here is another good <a href="https://github.com/iterative/dvc.org/issues/194">use case request</a> . If you are going to work on this project you would need some domain knowledge, preferably some basic ML or data science experience.</p><p><strong>Mentors: </strong><a href="https://github.com/shcheklein">@shcheklein</a> and <a href="https://github.com/dmpetrov">@dmpetrov</a></p><h4><strong>Project 3: Improving new user onboarding</strong></h4><p><strong>Description and details:</strong> Analyze and restructure user walkthrough across <a href="https://dvc.org/doc/get-started">Get started</a>, <a href="https://dvc.org/doc/tutorial">Tutorials</a> and <a href="https://dvc.org/doc/get-started/example-versioning">Examples</a>. These three have one thing in common — hands-on experience with DVC. If you choose this project, we will work together to come up with a better location for the Examples (to move them out of the Get Started shadow), and a better location for the Tutorials (to reference external tutorials that were developed by our community members and published on different platforms).</p><p><strong>Mentors: </strong><a href="https://github.com/shcheklein">@shcheklein</a> and <a href="https://github.com/dmpetrov">@dmpetrov</a></p><h4><strong>Project 4: Improving commands reference</strong></h4><p><strong>Description and details: </strong>We will work on improving our<strong> </strong><a href="https://dvc.org/doc/commands-reference">Commands reference</a> section. This includes expanding and filling in the gaps. One of the biggest pain points right now are Examples. Users want them to be <a href="https://github.com/iterative/dvc.org/issues/198">easy to run and try</a> and here is a lot to be done in terms of improvement. We have a good example of how is should be done <a href="https://dvc.org/doc/commands-reference/fetch">here</a>.</p><p><strong>Mentors: </strong><a href="https://github.com/shcheklein">@shcheklein</a> and <a href="https://github.com/dmpetrov">@dmpetrov</a></p><h4><strong>Project 5: Describe and integrate “DVC packages”</strong></h4><p><strong>Description and details: </strong>Describe the brand new feature “DVC packages” and integrate it with the rest of the documentation. We have been working hard to release a few new commands to help with datasets management (have a look at <a href="https://github.com/iterative/dvc/issues/1487">this ticket</a>). It’s a major feature that deserves its place in the Get Started, Use cases, Commands Reference, etc.</p><p><strong>Mentors: </strong><a href="https://github.com/shcheklein">@shcheklein</a> and <a href="https://github.com/dmpetrov">@dmpetrov</a></p><p>The ideas we outline above are just an example of what we can work on. We are open for any other suggestions and would like to work together with the technical writer to make the contribution experience both useful and enjoyable for all parties involved. If you have any suggestions or questions we would love to hear from you =&gt;DVC.org/support and our DMs on <a href="https://twitter.com/DVCorg">Twitter</a> are always open!</p><p>Special thanks to the <a href="https://numfocus.org/">NumFOCUS</a> for the ideas list inspiration.</p><p>If you are a tech writer — check the <a href="https://developers.google.com/season-of-docs/docs/tech-writer-guide">Technical writer guide</a>. From April 30, 2019 you can see the list of participating open source organizations on the Season of Docs website at <a href="https://g.co/seasonofdocs">g.co/seasonofdocs</a>. The application period for technical writers opens on <strong>May 29, 2019</strong> and ends on June 28, 2019.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=defe3a73b248" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataversioncontrol/dvc-project-ideas-for-google-summer-of-docs-2019-defe3a73b248">DVC project ideas for Google Summer of Docs 2019</a> was originally published in <a href="https://medium.com/dataversioncontrol">Data Version Control</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[April’19 DVC❤️Heartbeat]]></title>
            <link>https://medium.com/dataversioncontrol/april19-dvc-%EF%B8%8Fheartbeat-296c71a59be4?source=rss----4e21975b393---4</link>
            <guid isPermaLink="false">https://medium.com/p/296c71a59be4</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[git]]></category>
            <category><![CDATA[version-control-system]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Svetlana Grinchenko]]></dc:creator>
            <pubDate>Thu, 18 Apr 2019 01:03:13 GMT</pubDate>
            <atom:updated>2019-04-24T21:02:18.143Z</atom:updated>
            <content:encoded><![CDATA[<h3>April’ 19 DVC❤️Heartbeat</h3><p>Every month we are sharing here our news, findings, interesting reads, community takeaways, and everything along the way.</p><p>Some of those are related to our brainchild <a href="https://dvc.org/">DVC </a>and its journey. The others are a collection of exciting stories and ideas centered around ML best practices and workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e_h3ZrXtWf3t-5eBJUwt3w.jpeg" /></figure><h3>News and links</h3><p>We have some exciting news to share this month!</p><p>DVC is going to <a href="https://us.pycon.org/2019/">Pycon 2019</a>! It is the first conference that we attend as a team. When we say ‘team’ — we mean it. Our engineers are flying from all over the globe to get together offline and catch up with fellow Pythonistas.</p><p>The <a href="https://us.pycon.org/2019/schedule/talks/list/">speaker pipeline</a> is amazing! DVC creator Dmitry Petrov is giving a talk on <a href="https://us.pycon.org/2019/schedule/presentation/176/">Machine learning model and dataset versioning practices</a>.</p><p>Stop by our booth at the Startup Row on Saturday, May 4, reach out and let us know that you are willing to chat, or simply find a person with a huge DVC owl on their shirt!</p><p>Speaking of the owls — DVC has done some rebranding recently and we love our new logo. Special thanks to <a href="https://99designs.com/">99designs.com</a> for building a great platform for finding trusted designers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BdIz7sGCeiYIiRmd6a2nSw.png" /></figure><p>DVC is moving fast (almost as fast as my two-year-old). We do our best to keep up and totally love all the buzz in our community channels lately!</p><p>Here is a number of interesting reads that caught our eye:</p><ul><li><a href="https://blog.codecentric.de/en/2019/03/walkthrough-dvc/"><strong>A walkthrough of DVC</strong></a><strong> by </strong><a href="https://www.linkedin.com/in/bert-besser-284564182/"><strong>Bert Besser</strong></a></li></ul><p><a href="https://blog.codecentric.de/en/2019/03/walkthrough-dvc/">A walkthrough of DVC - codecentric AG Blog</a></p><p>A great article about using DVC with a quite advanced scenario and docker. If you haven’t had a chance to try<a href="http://dvc.org/"> DVC.org</a> yet — this is a great comprehensive read on why you should do so right away.</p><ul><li><a href="https://github.com/EthicalML/state-of-mlops-2019"><strong>The state of machine learning operations</strong></a><strong> by </strong><a href="https://www.linkedin.com/in/axsaucedo/"><strong>Alejandro Saucedo</strong></a></li></ul><p><a href="https://github.com/EthicalML/state-of-mlops-2019">EthicalML/state-of-mlops-2019</a></p><p>A short (only 8 minutes!) and inspiring talk by Alejandro Saucedo at FOSDEM. Alejandro covers the key trends in machine learning operations, as well as most recent open source tools and frameworks. Focused on reproducibility, monitoring and explainability, this lightning talk is a great snapshot of the current state of ML operations.</p><ul><li><a href="https://hackernoon.com/interview-with-kaggle-grandmaster-senior-cv-engineer-at-lyft-dr-vladimir-i-iglovikov-9938e1fc7c"><strong>Interview with Kaggle Grandmaster, Senior Computer Vision Engineer at Lyft: Dr. Vladimir I. Iglovikov</strong></a><strong> by </strong><a href="https://twitter.com/bhutanisanyam1"><strong>Sanyam Bhutani</strong></a></li></ul><p><a href="https://hackernoon.com/interview-with-kaggle-grandmaster-senior-cv-engineer-at-lyft-dr-vladimir-i-iglovikov-9938e1fc7c">Interview with Kaggle Grandmaster, Senior CV Engineer at Lyft: Dr. Vladimir I. Iglovikov</a></p><blockquote>There is no way you will become Kaggle Master and not learn how to approach anew, the unknown problem in a fast hacking way with a very high number of iterations per unit of time. This skill in the world of competitive learning is the question of survival</blockquote><h3>Discord gems:</h3><p>There are lots of hidden gems in our Discord community discussions. Sometimes they are scattered all over the channels and hard to track down.</p><p>We are sifting through the issues and discussions and share with you the most interesting takeaways.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/552098155861114891"><strong>What are the system requirements to install DVC (type of operating system, dependencies of another application (as GIT), memory, cpu, etc).</strong></a></li></ul><p>- It supports Windows, Mac, Linux. Python 2 and 3.</p><p>- No specific CPU or RAM requirements — it’s a lightweight command line tool and should be able run pretty much everywhere you can run Python.</p><p>- It depends on a few Python libraries that it installs as dependencies (they are specified in the <a href="https://github.com/iterative/dvc/blob/master/requirements.txt">`requirements.txt`</a>).</p><p>- It does not depend on Git and theoretically could be run without any SCM. Running it on top of a Git repository however is recommended and gives you an ability to actually save history of datasets, models, etc (even though it does not put them into Git directly).</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/560212552638791706"><strong>Do I have to buy a server license to run DVC, do you have this?</strong></a></li></ul><p>No server licenses for DVC. It is 100% free and open source.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/560154903331340289"><strong>What is the storage limit when using DVC?</strong></a><strong> I am trying to version control datasets and models with &gt;10 GB (Potentially even bigger). Can DVC handle this?</strong></li></ul><p>There is no limit. None enforced by DVC itself. It depends on the size of your local or <a href="https://dvc.org/doc/commands-reference/remote">remote storages</a>. You need to have some space available on S3, your SSH server or other storage you are using to keep these data files, models and their version, which you would like to store.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/553731815228178433"><strong>How does DVC know the sequence of stages to run</strong></a><strong>? How does it connect them? Does it see that there is a dependency which is outputted from the first run?</strong></li></ul><p>DVC figures out the pipeline by looking at the dependencies and outputs of the stages. For example, having the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/02b968ad1f6da1f01655f538a5c7f551/href">https://medium.com/media/02b968ad1f6da1f01655f538a5c7f551/href</a></iframe><p>you will end up with two stages: `download.dvc` and `duplicate.dvc`. The download one will have `joke.txt` as an output . The duplicate one defined `joke.txt` as a dependency, as it is the same file. DVC detects that and creates a pipeline by joining those stages.</p><p>You can inspect the content of each stage file <a href="https://dvc.org/doc/user-guide/dvc-file-format">here</a> (they are human readable).</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/560022999848321026"><strong>Is it possible to use the same data of a remote in two different repositories?</strong></a><strong> (e.g. in one repo `run dvc pull -r my_remote` to pull some data and running the same command in a different git repo should also pull the same)</strong></li></ul><p>Yes! It’s a frequent scenario for multiple repos to share remotes and even local cache. DVC file serves as a link to the actual data. If you add the same DVC file (e.g. `data.dvc`) to the new repo and do `dvc pull -r remotename data.dvc`- it will fetch data. You have to use `dvc remote add` first to specify the coordinates of the remote storage you would like to share in every project. Alternatively (check out the question below), you could use ` — global` to specify a single default remote (and/or cache dir) per machine.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485586884165107734/559653121228275727"><strong>Could I set a global remote server, instead of config in each project?</strong></a></li></ul><p>Use ` — global` when you specify the remote settings. Then remote will be visible for all projects on the same machine. ` — global` — saves remote configuration to the global config (e.g. ~/.config/dvc/config) instead of a per project one — `.dvc/config`. See more details <a href="https://dvc.org/doc/commands-reference/remote-add">here</a>.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/554679392823934977"><strong>How do I version a large dataset in S3 or any other storage?</strong></a></li></ul><p>We would recommend to skim through our <a href="https://dvc.org/doc/get-started">get started</a> tutorial, to summarize the data versioning process of DVC:</p><p>- You create stage (aka DVC) files by adding, importing files (dvc add / dvc import) , or run a command to generate files (dvc run — out file.csv “wget https://example.com/file.csv&quot;).</p><p>- This stage files are tracked by git</p><p>- You use git to retrieve previous stage files (e.g. git checkout v1.0)</p><p>- Then use dvc checkout to retrieve all the files related by those stage files</p><p>All your files (with each different version) are stored in a .dvc/cache directory, that you sync with a remote file storage (for example, S3) using the dvc push or dvc pull commands (analogous to a git push / git pull, but instead of syncing your .git, you are syncing your .dvc directory)</p><p>on a remote repository (let’s say an S3 bucket).</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/558216007684980736"><strong>How do I move/rename DVC files?</strong></a></li></ul><p>If you need to move your dvc file somewhere, it is pretty easy, even if done manually:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8c72433012d97ef63564fe6a63690523/href">https://medium.com/media/8c72433012d97ef63564fe6a63690523/href</a></iframe><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/555431645402890255"><strong>I performed ‘dvc push’ of a file to a remote. On the remote there is created a directory called ‘8f’ with a file inside called ‘2ec34faf91ff15ef64abf3fbffa7ee’. The original csv file doesn’t appear on the remote. Is that expected behaviour?</strong></a></li></ul><p>This is an expected behaviour. DVC saves files under the name created from their checksum in order to prevent duplication. If you delete “pushed” file in your project directory and perform `dvc pull`, dvc will take care of pulling the file and renaming it to “original” name.</p><p>Below are some details about how DVC’s cache works, just to illustrate the logic. When you add a data source:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/686c49836ecd671187d2eb82eb8d7f37/href">https://medium.com/media/686c49836ecd671187d2eb82eb8d7f37/href</a></iframe><p>It computes the (md5) checksum of the file and generates a DVC file with related information:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/85b4ef986786c3d7b2184d2fac44aba0/href">https://medium.com/media/85b4ef986786c3d7b2184d2fac44aba0/href</a></iframe><p>The original file is moved to the cache and a link or copy (depending on your filesystem) is created to replace it on your working space:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/25cf483104145701a63ff089ae6f1dd3/href">https://medium.com/media/25cf483104145701a63ff089ae6f1dd3/href</a></iframe><ul><li><a href="https://discordapp.com/channels/485586884165107732/485586884165107734/553570391000481802"><strong>Is it possible to integrate dvc with our in-house tools developed in Python?</strong></a></li></ul><p>Absolutely! There are three ways you could interact with DVC:</p><p>1) Use <a href="https://docs.python.org/3/library/subprocess.html">subprocess</a> to launch DVC;</p><p>2) Use from `dvc.main import main` and use it with regular CLI logic like `ret = main(‘add’, ‘foo’)`</p><p>3) Use our internal API (see `dvc/repo` and `dvc/command` in our source to get a grasp of it). It is not officially public yet, and we don’t have any special docs for it, but it is fairly stable and could definitely be used for a POC. We’ll add docs and all the official stuff for it in the not-so-distant future.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485586884165107734/555750217522216990"><strong>Can I still track the linkage between data and model without using `dvc run`</strong></a><strong> and a graph of tasks? Basically what would like extremely minimal DVC invasion into my GIT repo for an existing machine learning application?</strong></li></ul><p>There are two options:</p><p>1. Use `dvc add` to track models and/or input datasets. It should be enough if you use `git commit` on DVC files produced by `dvc add`. This is the very minimum you can get with DVC and it does not require using DVC run. Check the first part (up to the Pipelines/Add transformations section) of the DVC <a href="https://dvc.org/doc/get-started">get started</a>.</p><p>2. You could use ` — no-exec` in `dvc run` and then just `dvc commit` and `git commit` the results. That way you’ll get your DVC files with all the linkages, without having to actually run your commands through DVC.</p><p>If you have any questions, concerns or ideas, let us know <a href="https://dvc.org/support">here</a> and our stellar team will get back to you in no time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=296c71a59be4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataversioncontrol/april19-dvc-%EF%B8%8Fheartbeat-296c71a59be4">April’19 DVC❤️Heartbeat</a> was originally published in <a href="https://medium.com/dataversioncontrol">Data Version Control</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DVC Heartbeat]]></title>
            <link>https://medium.com/dataversioncontrol/dvc-heartbeat-6301aebf5c96?source=rss----4e21975b393---4</link>
            <guid isPermaLink="false">https://medium.com/p/6301aebf5c96</guid>
            <category><![CDATA[dvc]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[reproducibility]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Svetlana Grinchenko]]></dc:creator>
            <pubDate>Tue, 05 Mar 2019 00:45:58 GMT</pubDate>
            <atom:updated>2019-04-24T21:02:01.156Z</atom:updated>
            <content:encoded><![CDATA[<h3>March ’19 DVC❤️Heartbeat</h3><p>This is the very first issue of the DVC❤️Heartbeat. Every month we will be sharing our news, findings, interesting reads, community takeaways, and everything along the way.</p><p>Some of those are related to our brainchild <a href="https://dvc.org/">DVC </a>and its journey. The others are a collection of exciting stories and ideas centered around ML best practices and workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jUoIve_Ff3QUjRsRsL_S2g.jpeg" /></figure><h3><strong>News and links</strong></h3><p>We read a ton of articles and posts every day and here are a few that caught our eye. Well-written, offering a different perspective and definitely worth checking.</p><ul><li><a href="https://veekaybee.github.io/2019/02/13/data-science-is-different/"><strong>“Data science is different now”</strong></a><strong> by </strong><a href="https://veekaybee.github.io/"><strong>Vicki Boykis</strong></a></li></ul><p><a href="https://veekaybee.github.io/2019/02/13/data-science-is-different/">Data science is different now</a></p><blockquote>‘What is becoming clear is that, in the late stage of the hype cycle, data science is asymptotically moving closer to engineering, and the <a href="https://www.youtube.com/watch?v=frQeK8xo9Ls">skills that data scientists need</a> moving forward are less visualization and statistics-based, and <a href="https://tech.trivago.com/2018/12/03/teardown-rebuild-migrating-from-hive-to-pyspark/">more in line with traditional computer science curricula</a>.’</blockquote><ul><li><a href="https://emilygorcenski.com/post/data-versioning/"><strong>“Data Versioning”</strong></a><strong> by </strong><a href="https://emilygorcenski.com/"><strong>Emily F. Gorcenski</strong></a></li></ul><p><a href="https://emilygorcenski.com/post/data-versioning/">Data Versioning · EmilyGorcenski.com</a></p><blockquote>‘I want to explore how the degrees of freedom in versioning machine learning systems poses a unique challenge. I’ll identify four key axes on which machine learning systems have a notion of version, along with some brief recommendations for how to simplify this a bit.’</blockquote><ul><li><a href="https://blog.mi.hdm-stuttgart.de/index.php/2019/02/26/reproducibility-in-ml/"><strong>“Reproducibility in Machine Learning”</strong></a><strong> by </strong><a href="https://blog.mi.hdm-stuttgart.de/index.php/author/pf023/"><strong>Pascal Fecht</strong></a></li></ul><p><a href="https://blog.mi.hdm-stuttgart.de/index.php/2019/02/26/reproducibility-in-ml/">Reproducibility in Machine Learning | Computer Science Blog</a></p><blockquote>‘…the objective of this post is not to philosophize about the dangers and dark sides of AI. In fact, this post aims to work out common challenges in reproducibility for machine learning and shows programming differences to other areas of Computer Science. Secondly, we will see practices and workflows to create a higher grade of reproducibility in machine learning algorithms.’</blockquote><h3><strong>DVC Discord gems</strong></h3><p>There are lots of hidden gems in our Discord community discussions. Sometimes they are scattered all over the channels and hard to track down.</p><p>We will be sifting through the issues and discussions and share the most interesting takeaways.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485586884165107734/541622187296161816"><strong>Edit and define DVC files manually, in a Makefile style</strong></a></li></ul><p>There is no separate guide for that, but it is very straight forward. See <a href="https://dvc.org/doc/user-guide/dvc-file-format">DVC file format</a> description for how dvc file looks inside in general. All `dvc add` or `dvc run` does is just computing md5 fields in it, that is all. You could write your dvc file and then run dvc repro that will run a command(if any) and compute all needed checksums … <a href="https://discordapp.com/channels/485586884165107732/485586884165107734/541622187296161816">read more</a></p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485586884165107734/547424240677158915"><strong>Best practices to define the code dependencies</strong></a></li></ul><p>…There’s a ton of code in that project, and it’s very non-trivial to define the code dependencies for my training stage — there are a lot of imports going on, the training code is distributed across many modules … <a href="https://discordapp.com/channels/485586884165107732/485586884165107734/547424240677158915">read more</a></p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485586884165107734/548495589428428801"><strong>Azure data lake support</strong></a></li></ul><p>DVC officially only supports regular Azure blob storage. Gen1 Data Lake should be accessible by the same interface, so configuring a regular azure remote for dvc should work. Seems like Gen2 Data Lake <a href="https://discordapp.com/channels/485586884165107732/485586884165107734/550546413197590539">has disable</a> blob API. If you know more details about the difference between Gen1 and Gen2, feel free to join <a href="https://dvc.org/chat">our community</a> and share this knowledge.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/542390986299539459"><strong>What licence DVC is released under</strong></a></li></ul><p>Apache 2.0. One of the <a href="https://opensource.org/licenses">most common</a> and permissible OSS licences.</p><ul><li><strong>Setting up S3 compatible remote (</strong><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/543445798868746278"><strong>Localstack</strong></a><strong>, </strong><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/541466951474479115"><strong>wasabi</strong></a><strong>)</strong></li></ul><pre>$ dvc remote add upstream s3://my-bucket<br>$ dvc remote modify upstream region REGION_NAME<br>$ dvc remote modify upstream endpointurl &lt;url&gt;</pre><p>Find and click the `S3 API compatible storage` on <a href="https://dvc.org/doc/commands-reference/remote-add">this page</a></p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/543914550173368332"><strong>Why DVC creates and updates `.gitignore` file</strong></a></li></ul><p>… it adds your datafiles there, that are tracked by dvc, so that you don’t accidentally add them to git as well you can open it with file editor of your liking and see your data files listed there.</p><ul><li><a href="https://discordapp.com/channels/485586884165107732/485596304961962003/545562334983356426"><strong>Managing data and pipelines with DVC on HDFS</strong></a></li></ul><p>… with dvc, you could connect your data sources from HDFS with your pipeline in your local project, by simply specifying it as an external dependency. For example let’s say your script `process.cmd` works on an input file on HDFS and then downloads a result to your local workspace, then with DVC it could look something like:</p><pre>$ dvc run -d hdfs://example.com/home/shared/input -d process.cmd -o output process.cmd</pre><p>… <a href="https://discordapp.com/channels/485586884165107732/485596304961962003/545562334983356426">read more</a>.</p><p>If you have any questions, concerns or ideas, let us know <a href="https://dvc.org/support">here</a> and our stellar team will get back to you in no time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6301aebf5c96" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataversioncontrol/dvc-heartbeat-6301aebf5c96">DVC Heartbeat</a> was originally published in <a href="https://medium.com/dataversioncontrol">Data Version Control</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>