<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Falk Tandetzky on Medium]]></title>
        <description><![CDATA[Stories by Falk Tandetzky on Medium]]></description>
        <link>https://medium.com/@falktan?source=rss-bc3dfdb706f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*7bEGD6Ys7Iki36SqU1cc5Q.jpeg</url>
            <title>Stories by Falk Tandetzky on Medium</title>
            <link>https://medium.com/@falktan?source=rss-bc3dfdb706f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 15:46:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@falktan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Keep passwords out of source code — why and how]]></title>
            <link>https://medium.com/twodigits/keep-passwords-out-of-source-code-why-and-how-e84f9004815a?source=rss-bc3dfdb706f------2</link>
            <guid isPermaLink="false">https://medium.com/p/e84f9004815a</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[passwords]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Falk Tandetzky]]></dc:creator>
            <pubDate>Fri, 02 Jul 2021 12:34:57 GMT</pubDate>
            <atom:updated>2021-07-02T12:34:57.001Z</atom:updated>
            <content:encoded><![CDATA[<h3>Keep passwords out of source code — why and how</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lntTzlPXhu8s3WGFHRVkwA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@iantuck?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Ian Tuck</a> on <a href="https://unsplash.com/s/photos/secret?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>A very common security misbehavior I see in my daily work is that credentials are checked into source code control (like git). This is often referred to as hard-coded credentials. This article provides an overview about why this should be avoided and how this can be achieved in different situations. As the solution depends a lot on the situation, I will mostly provide a rough overview about a topic and then provide links to sources where things are explained in depth.</p><p>Throughout the article I will use terms like passwords, credentials or secrets interchangeably. The latter two are the broadest terms. They include passwords used by users or technical users, API keys, certificates, ssh keys and really any other data that is secret.</p><h3>Why not store passwords in source code?</h3><p>Oftentimes, when code is developed for a company, this company owns this code and keeps it secret. This can lead to the impression, that also credentials can be stored with the code without problems. Probably anyone who is allowed to access the code is also allowed to access the credentials. So why bother?</p><p>The issue is, that access to code and who should have access to the credentials often changes over time. Once a project becomes more mature there are usually some developers who do not need to know and hence should not know certain credentials. In addition code may be reused for slightly different purposes or even become open source.</p><p>In addition, as the code base grows more complex, it becomes very tedious to identify credentials being buried somewhere in the middle of the code. So at some point people might not even be aware of credentials being distributed with the code. Finally, it can become difficult to find out in which places the same credentials are used, making it difficult to ever change credentials.</p><p>I did not find a compiled list of security breaches caused by clear-text passwords, but it seems safe to say that<strong> hundreds of millions of users have already been affected by issues caused by hard-coded credentials</strong>. <a href="https://www.beyondtrust.com/blog/entry/hardcoded-and-embedded-credentials-are-an-it-security-hazard-heres-what-you-need-to-know">Here is an article with some examples</a>.</p><h3>How to store your passwords</h3><h4>Injecting credentials into a container</h4><p>There are countless ways to host an application. To have some specific examples in mind, think of AWS, Azure, GCP or Heroku as platforms where you can host a web application.</p><p>Oftentimes your web-application needs to access some data source. This may be a database (managed by yourself or someone else) or another API, which may be protected by a password or API-key. Either way, you need to make it possible that your app can somehow prove that it is permitted to access the resource.</p><p>As we do not want to put credentials into the source code of the app, the process for this needs two steps.</p><ol><li>Credentials need to be provided to your application.</li><li>Your application should find the correct credential and use it</li></ol><p><strong>Credential provider</strong></p><p>Probably the most common approach for providing credentials is to inject them in the environment when starting the service. This can for instance be the operating system environment of a Linux server or of a docker container in which your application is running.</p><p>For AWS there are plenty of ways how to do this. Sometimes you can avoid passing credentials explicitly altogether by <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html">using instance roles</a> and authorization via IAM. In the case of third party APIs you need to pass the credentials more explicitly. Once again there are several ways. See e.g. <a href="https://medium.com/twodigits/dynamic-app-configuration-passing-secrets-and-sensitive-values-safely-to-applications-at-runtime-8c0e0ba63c0a">AWS: passing credentials to a docker container using AWS Fargate</a>.</p><p>To add another example here is how you can <a href="https://devcenter.heroku.com/articles/config-vars">inject credentials on Heroku</a>.</p><p>A noteworthy trick to manage multiple credentials is to just have one real secret passed to the container by the methods mentioned above and then use this secret to retrieve other secrets. This limits the number of parameters and credentials that need to be passed to the container through above methods while allowing to manage many parameters and secrets in a central place. Examples of tools that help with this approach are</p><ul><li><a href="https://www.vaultproject.io/">HCP Vault</a></li><li><a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html">AWS Parameter Store</a> / <a href="https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html">AWS Secrets Manager</a></li></ul><p><strong>Credential consumer</strong></p><p>If your credentials are provided in the environment you can easily access them. However, managing parameters and credentials throughout various environments can become challenging. To simplify this process there are some tools to make accessing the correct credentials in the appropriate environment smoother. Here are some examples for such tools</p><ul><li><a href="https://pypi.org/project/python-dotenv/">Python: How to manage variables from the environment using dotenv</a></li><li><a href="https://medium.com/twodigits/dynamic-app-configuration-inject-configuration-at-run-time-using-spring-boot-and-docker-ffb42631852a">Java: Spring boot, and how it uses credentials</a></li></ul><h4>Build Pipelines:</h4><p>Any reasonable CI/CD tool provides functionality that allows you to avoid putting credentials in the pipeline definition itself. Instead you configure credentials independent of the pipeline definition. This allows you to access them via special variables in the pipeline definition. Here are some examples</p><ul><li><a href="https://docs.github.com/en/actions/reference/encrypted-secrets">Github: how to handle secrets in build pipelines</a></li><li><a href="https://www.jenkins.io/doc/book/pipeline/jenkinsfile/#handling-credentials">Jenkins: how to handle secrets in build pipelines</a></li></ul><h4>Handling User Passwords on a Server:</h4><p>A common pattern of how to grant access to an application is to require a user to provide a password. First of all there are sometimes better alternatives to this approach, like single-sign-on or OIDC. Even if this approach is used, user passwords should never be hard coded and usually also not be stored on the server or a database.</p><p>A good approach is to not even store the password at all. Instead one should use so-called salt and hashing and only store the hash in a database. Of course make sure access to this DB is as limited as possible.</p><p>Here are some articles about how this works</p><ul><li><a href="https://www.baeldung.com/java-password-hashing">How and why to salt and hash passwords in Java</a></li><li><a href="https://nitratine.net/blog/post/how-to-hash-passwords-in-python/">How and why to salt and hash passwords in Python</a></li><li><a href="https://crackstation.net/hashing-security.htm">Detailed article about how and why to salt and hash (with some more links for multiple programming languages)</a></li></ul><h3>Summary</h3><p>There are many scenarios where credentials are needed in an application. Putting them directly into source code poses a security risk, as it makes it impossible to provide access to the source code without also providing access to those credentials.</p><p>For all steps of implementing an application, deploying it with automation tools and running them on a platform there are mechanisms to handle credentials in a save way.</p><p>Pretty much any platform and any build automation tool has its own specifics of how credentials are handled. However, with awareness that credentials need to be treated with care, it is often not difficult to find appropriate documentation.</p><h3>Further Reading</h3><ul><li>Top 10 of web service vulnerabilities every developer should be aware of: <a href="https://owasp.org/www-project-top-ten/">https://owasp.org/www-project-top-ten/</a></li><li><a href="https://deshpandetanmay.medium.com/the-easiest-way-to-remove-checked-in-credentials-from-a-git-repo-704a373b94e3">How to remove credentials from git history</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e84f9004815a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/twodigits/keep-passwords-out-of-source-code-why-and-how-e84f9004815a">Keep passwords out of source code — why and how</a> was originally published in <a href="https://medium.com/twodigits">NEW IT Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[3 Reasons Quantum Computing is Overrated]]></title>
            <link>https://medium.com/twodigits/3-reasons-quantum-computing-is-overrated-9d87d11aa248?source=rss-bc3dfdb706f------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d87d11aa248</guid>
            <category><![CDATA[quantum-computing]]></category>
            <category><![CDATA[limitations]]></category>
            <category><![CDATA[criticism]]></category>
            <category><![CDATA[google]]></category>
            <dc:creator><![CDATA[Falk Tandetzky]]></dc:creator>
            <pubDate>Mon, 09 Mar 2020 07:28:14 GMT</pubDate>
            <atom:updated>2021-07-02T13:33:54.866Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pPdOMM94OecRzXSc5eN8Dg.jpeg" /></figure><p>Some weeks ago I was enjoying a beer with a couple of colleagues. Then one of them told us that we should start learning how to use quantum computers and that they would soon change how we do software development. He was probably not suspecting that he was sitting on a table with someone who made his PhD in quantum physics. To be honest I have not looked into the field of quantum computing much during the last decade. Yet, I could immediately tell that the picture he was drawing was full of misconceptions. After that I came across all kinds of indications that he is not alone with his view. I saw training sessions on an IT conference that would teach quantum physics. I heard people talk about the topic a lot. I saw technology radars report a high relevance of the topic — not for fundamental research but for software engineering.</p><p>Time to clean up some misconceptions. Time to help people understand that quantum computing is still a matter of fundamental research and nowhere close to every-day applications.</p><h3>1: The reports on “quantum supremacy” are misleading</h3><p>The story google <a href="https://blog.google/technology/ai/computing-takes-quantum-leap-forward">published in October 2019 [1]</a> is impressive, but not as much a breakthrough as they would like to make us believe. The best <a href="https://www.ibm.com/blogs/research/2019/10/on-quantum-supremacy/">criticism came immediately from a competitor [2]</a>: The statement that what they solved on their system would take 10.000 year for a classical computer is not true. It would only take 2 or 3 days. In addition the problem they consider is of no practical relevance and rather picked so that it can be handled by their quantum computer rather well and is difficult for an ordinary computer.</p><p>Reports from D-Wave [3] focus on practical devices for optimization tasks. It is very misleading when they claim, that they handle so-called NP-Hard problems. What their devices can do is finding <em>approximate</em> solutions to those problems (a much easier task, which can also be done by ordinary computers), but a convincing proof, that they can guarantee to find the global optimum has not yet been delivered, and it seems likely, that they cannot do that [4, 5].</p><h3>2. Scaling</h3><p>The usual reason stated why quantum computers would eventually overtake ordinary computers is: The number of computational states that can be explored scales exponentially with the number of qubits. Well, that is true. But the question how the number of qubits increases over time is usually not discussed. The reasons are: a) no one knows and b) probably not so well.</p><p>The trouble is that the more qubits you have, the tougher it is to add another one due to a phenomenon called decoherence. In addition one needs to join multiple qubits into so-called error-corrected qubits [6]. As a result it is by no means clear if the number of qubits that can be realized scales better or worse than linear in time. If the number of <em>error-corrected</em> qubits increases by one every other year, quantum computers would all in all scale about as good (i.e. not exponentially better) as ordinary computers, at least as long as <a href="https://en.wikipedia.org/wiki/Moore%27s_law">Moore’s law [7]</a> continues. At the moment it is way to early to make a reasonable prediction on how the scaling of error-corrected qubits will look like, as to the best of my knowledge there has not been a demonstration of more than a handful of error corrected qubits.</p><h3>3. Quantum computers can only be applied to a very limited set of problems</h3><p>It is a common misconception that quantum computers can solve most exponentially scaling problems. They cannot.</p><p>Edit 07/2021: Here is a great article explaining why this misconception comes up and some explanation of how the limitations arise: <a href="https://www.quantamagazine.org/why-is-quantum-computing-so-hard-to-explain-20210608/">https://www.quantamagazine.org/why-is-quantum-computing-so-hard-to-explain-20210608/</a></p><p>Let’s go through the different classes of problems.</p><h4>Cryptography</h4><p>I wouldn’t call cryptography an application of quantum computing but rather an unwanted side-product. Not too surprisingly the NSA started investing tax money on this topic [8]. From their perspective this is an application of quantum computing. From my perspective this is a threat to information security. If quantum computers ever break the security of currently used encryption schemes then the end result of this will probably be that both sides, the ones protecting their data and the ones trying to break the security will invest quite some effort. And in the end we are in the same situation as before, just that everyone wasted a lot of work and time.</p><p>As discussed it is by no means clear if quantum computers scale better than ordinary computers. For breaking current crypto algorithms they would not only need to scale better, but need to scale <em>exponentially </em>better on this task. My feeling is that this will probably not happen at all and even if this happens, I would guess that this is still at least two decades away. Fortunately there are already numerous ideas how ordinary computers could implement better protocols that would be save against quantum computers [9].</p><h4>Simulation of quantum systems</h4><p>For simulating quantum systems the requirements are lower than for quantum computing, as one only needs to mirror the properties of a system of interest. Due to that this is something that actually works and in fact has been done for at least a decade. And it is something that is indeed helpful — for fundamental research. So when google claims [10]</p><blockquote>[we believe that] quantum computing can accelerate solutions for some of the world’s most pressing problems, from climate change to disease.</blockquote><p>then this is not completely made up. But the question is how much of an impact will it have. Any invention that saves some energy or in any way simplifies research, qualifies for the same statement. I have seen “might help cure cancer” and “might lay the foundations to improve batteries” on too many proposals for financing fundamental research. My point here is not that quantum computing cannot contribute to research, just that one should not expect it to make too much of a difference. Without going into the details, it should also be noted, that the systems quantum computers can simulate are also somewhat limited. In addition simulations of quantum systems have been done on classical computers for at least 40 years. It works with the same trick as for treating otherwise tough optimization tasks: One can often accept a limited accuracy in practice and that way simplify the problem considerably.</p><h4>Polynomial scaling</h4><p>Most every day problems scale linear (10 times the data to process means 10 times longer processing time needed) or quadratic (10 times more data means 100 times longer processing time needed) or so with input size. That includes anything that falls into the category of big data or data analysis, task like searching or sorting and pretty much any data transformation task and a lot of more things you might have in mind. For problems of this type quantum computers can not profit from the quantum nature of its qubits. Hence none of those problems can be treated on a quantum computer more efficiently than on ordinary computers.</p><p>What is left are problems that scale exponentially (for example: additional 10 data elements increase the total effort by a <strong>factor</strong> of 2¹⁰ = 1024).</p><h4>Optimization problems</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/274/1*dPrUcPQgCOzYxzBNc3WN4w.png" /><figcaption>The knapsack problem [11]. Finding the optimal solution is one example of the class of so-called NP-hard problems. Those problems cannot be solved with quantum computing [4, 5, 12].</figcaption></figure><p>Optimization problems are of very high practical relevance. Some optimization problems (for example finding a shortest path between two cities) have polynomial scaling, as described above. Pretty much <em>all other optimization problems of relevance</em> fall into a class of problems called “NP-hard”. An example would be the knapsack problem [11]: Given the weights and values of a bunch of objects, which ones should one choose to fill a knapsack (that can only hold a limited total weight) in order to maximize the total value of objects. Another famous example is the traveling salesman problem: Which is the shortest route to travel around a number of given cities?</p><p>Aaaand — drum roll — <em>quantum computers cannot solve any of those problems</em>. There is no algorithm known that can run on a quantum computer that solves a NP-<em>hard</em> problem [4, 5, 12].</p><p>And just as a side remark: most of the time there is a very reasonable workaround. It is often enough to limit accuracy. I.e. search for a solution that gets you to, say 99% of the optimum. Then somewhat surprisingly those problems are suddenly feasible on ordinary computers [11].</p><h4>Other problems</h4><p>I am not aware of any problems that are of high relevance and do not fall into any of the mentioned categories (where the section on optimization stands for all NP-hard problems). There are some problems known to be in NP but not in NP-hard (see e.g. [13]), but aside from cryptography I am not sure if there are any really important among them.</p><h3>Conclusion</h3><p>Quantum computing is an interesting research topic and may in the long run help to solve computational tasks that cannot be solved without it. However, as increasing the size of the computer gets harder with every qubit, is not yet clear if they can overtake classical computers any time soon if at all. Even if powerful quantum computers can be build, the set of problems that can be solved more efficiently on them will be very restricted due to the nature of how algorithms run on them. They would be able to break encryption, as it is currently in use. But aside from that the list of tasks that can be handled is very short.</p><h3>Acknowledgement</h3><p>Thanks a lot to Dr. Andreas Johannes and Dr. Enrico Seise for reviewing this article and for helpful discussions.</p><h3>Further Reading</h3><ul><li>Update 07/2021: Also have a look at this excellent article on the topic: <a href="https://www.quantamagazine.org/why-is-quantum-computing-so-hard-to-explain-20210608/">https://www.quantamagazine.org/why-is-quantum-computing-so-hard-to-explain-20210608/</a></li></ul><p>[1] Story by google: <a href="https://blog.google/technology/ai/computing-takes-quantum-leap-forward">https://blog.google/technology/ai/computing-takes-quantum-leap-forward</a><br>[2] Criticism on googles story: <a href="https://www.ibm.com/blogs/research/2019/10/on-quantum-supremacy/">https://www.ibm.com/blogs/research/2019/10/on-quantum-supremacy/</a><br>[3] <a href="https://www.dwavesys.com/">https://www.dwavesys.com/</a><br>[4] Joe Fitzsimons (https://physics.stackexchange.com/users/163/joe-fitzsimons), What can the D-Wave quantum computer do? <a href="https://physics.stackexchange.com/q/14604">https://physics.stackexchange.com/q/14604</a><br>[5] <a href="https://www.reddit.com/r/Physics/comments/631rjj/dwave_and_np_problems/">https://www.reddit.com/r/Physics/comments/631rjj/dwave_and_np_problems/</a><br>[6] <a href="https://arxiv.org/pdf/quant-ph/9802018.pdf">https://arxiv.org/pdf/quant-ph/9802018.pdf</a><br>[7] <a href="https://en.wikipedia.org/wiki/Moore%27s_law">https://en.wikipedia.org/wiki/Moore%27s_law</a><br>[8] <a href="https://www.washingtonpost.com/world/national-security/nsa-seeks-to-build-quantum-computer-that-could-crack-most-types-of-encryption/2014/01/02/8fff297e-7195-11e3-8def-a33011492df2_story.html">https://www.washingtonpost.com/world/national-security/nsa-seeks-to-build-quantum-computer-that-could-crack-most-types-of-encryption/2014/01/02/8fff297e-7195-11e3-8def-a33011492df2_story.html</a> <br>[9] <a href="https://en.wikipedia.org/wiki/Post-quantum_cryptography">https://en.wikipedia.org/wiki/Post-quantum_cryptography</a><br>[10] <a href="https://blog.google/perspectives/sundar-pichai/what-our-quantum-computing-milestone-means/">https://blog.google/perspectives/sundar-pichai/what-our-quantum-computing-milestone-means/</a><br>[11] <a href="https://en.wikipedia.org/wiki/Knapsack_problem">https://en.wikipedia.org/wiki/Knapsack_problem</a><br>[12] <a href="https://en.wikipedia.org/wiki/BQP">https://en.wikipedia.org/wiki/BQP</a><br>[13] <a href="https://cstheory.stackexchange.com/questions/79/problems-between-p-and-npc">https://cstheory.stackexchange.com/questions/79/problems-between-p-and-npc</a><br>Title Photo by <a href="https://unsplash.com/@cdr6934?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Chris Ried</a> on <a href="https://unsplash.com/s/photos/microchip?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d87d11aa248" width="1" height="1" alt=""><hr><p><a href="https://medium.com/twodigits/3-reasons-quantum-computing-is-overrated-9d87d11aa248">3 Reasons Quantum Computing is Overrated</a> was originally published in <a href="https://medium.com/twodigits">NEW IT Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Deepracer — How to train a model in 15 minutes]]></title>
            <link>https://medium.com/twodigits/aws-deepracer-how-to-train-a-model-in-15-minutes-3a0dca1175fb?source=rss-bc3dfdb706f------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a0dca1175fb</guid>
            <category><![CDATA[deepracer]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[accenture]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <dc:creator><![CDATA[Falk Tandetzky]]></dc:creator>
            <pubDate>Mon, 16 Dec 2019 18:46:35 GMT</pubDate>
            <atom:updated>2020-10-31T21:05:25.253Z</atom:updated>
            <content:encoded><![CDATA[<h3>AWS Deepracer — How to train a model in 15 minutes</h3><p>In a <a href="https://medium.com/twodigits/the-aws-deepraceing-championship-at-re-invent-in-las-vegas-and-how-i-got-there-2595394427d1">previous article</a> I described how I got into DeepRacer and what it is. This article is about the technical parts: How the scoring function was designed and why it works.</p><h3>What most people did</h3><p>Almost all scoring functions that I heard of are mostly scoring based on where the car currently is. The simplest approach is already provided as an example by AWS: just score high, if you are closer to the central line. More sophisticated approaches score high, if the car is close to a hand crafted more or less ideal line. E.g. a line that would be on the right before a left turn and on the left in the middle of the turn. The <a href="https://arxiv.org/abs/1707.06347">proximal policy optimization algorithm</a> is then able to learn what good actions have been and which not.</p><h3>What I did</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/467/0*A6UtKugYg3wGqhBj.png" /><figcaption>The point the car should aim for is determined by simple geometry.</figcaption></figure><p>My approach was different: I used the steering direction of the car (which is available as a parameter in the scoring function during training).</p><p><a href="https://github.com/TwoDigits/deepracer">My scoring function</a> works as follows:</p><p>Draw a circle with a fixed radius around the car.</p><p>Take the intersection of this circle with the central line.</p><p>Aim for this point. I.e. The score is highest if the steering direction is such that the front wheels point there. The more it deviates the lower the score.</p><p>I adjusted the radius of the circle depending on the track and how narrow the corners are. And that is it. This <a href="https://medium.com/twodigits/the-aws-deepraceing-championship-at-re-invent-in-las-vegas-and-how-i-got-there-2595394427d1">got me to the top 8 worldwide</a>. Other than that I only needed to adjust one of the training parameters: I set the discount factor to 0.5 rather than the default of 0.999.</p><h3>Why is this a good idea?</h3><p>After about 15 minutes of training the car completes its first lap. One minute later it completes another lap and after this it can run 50 more without a single crash. Other approaches usually took many hours if not days to converge to a somewhat stable model. But Why?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/0*L8tpAqgzV94sNXOs.png" /><figcaption>18 Minutes of training on the re:invent 2018 track. After 42 attempts the model learned to run the track with 100% success rate.</figcaption></figure><p><strong>Quick response leads to faster learning</strong>: Reinforcement learning is about rewarding good behavior. The clearer the connection between the behavior and the score is, the quicker it can learn. Imagine you try to train your dog something. You <em>could </em>do this by providing a reward every evening, say, by providing more or better food, if it behaved well. Though this does work in principle it will take pretty much forever. If you want to train some tricks, you pretty much have to provide a reward immediately after something was done well.</p><p>It is the same with the scoring function in DeepRacer: One could in principle exclusively provide a score for finishing a lap. The PPO-Algorithm would eventually work out how to steer. But it will only improve if it ever finishes a lap, and even then would it barely learn with each success. In fact doing slightly better by providing a constant score for not crashing already allows the algorithm to slowly (but visibly) learn. Of cause the scoring according to position of the car is much more direct. Even still it only provides some kind of average rating for the last couple of frames that lead the car to the place it is now. There may be a couple of bad steering commands in the past that were only compensated by other commands later on. Both would end up being encouraged and only eventually would good commands end up with a better statistic and hence get encouraged most often.</p><p>Pointing to the “right” direction is as direct as it gets. It immediately tells the learning algorithm, if this steering direction is good or not. There is no averaging over multiple frames needed. This is also the reason I reduced the discount factor: the scoring does by no means rate good actions in the past (in contrast to scoring according to position). I could probably have reduced the discount factor even further. I really did not have time to play around with it yet.</p><p>This direct scoring is the main reason this approach learns so fast.</p><p><strong>Using local information makes learning easier</strong>: Ultimately the car has to drive without any knowledge of way points or where it is exactly. It can only work with the camera image and make a decision. This means that it is much easier to learn a scoring function that is only based on information that can be <em>seen</em>. I am using only information about the central line right in front of the car, which can usually be seen. A perfectly optimized ideal line would (at least in principle) also depend on parts of the track that can not even be seen at the time of making the decision. That does not mean that it is not possible to learn this more complicated path, just it is more difficult.</p><p><strong>Cutting corners is good</strong>: This should not be too surprising. It is faster because the path gets shorter this way. It also reduces the risk of running out of the track. Finally this also makes it easier to keep the eyes on the road. If the car is on the right in a left turn, the camera mostly sees green and only little of the track, so it is harder to decide in which direction to steer. Increasing the radius in the scoring function described above leads to a direction that is cutting corners in a stronger way.</p><h3>Isn’t this cheating?</h3><p>Well it feels a bit like cheating at first. Right? After all there is an elaborate training algorithm that can figure out how to steer. An now we tell the car directly how it should steer using “complicated mathematics”.</p><p>That is true, but the challenge is <em>not </em>to learn what is a good direction to steer to <em>given the way points and the coordinates of the car</em>. The difficult part is to train a neuronal network in such a way that it can find this direction <em>based on the camera image</em>. It is only in the virtual environment that we can use the position of the car and the track because this information is so kindly provided by the virtual environment.</p><p>What providing scores based on position does is it puts the extra burden of learning how the steering direction affects the position on the neuronal network.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FHWWs7oN7rRc%3Ffeature%3Doembed&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DHWWs7oN7rRc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FHWWs7oN7rRc%2Fhqdefault.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/015a20e59d43ee0901f715dc3f4c65c3/href">https://medium.com/media/015a20e59d43ee0901f715dc3f4c65c3/href</a></iframe><p>In summary, one should put as much knowledge of the problem into the scoring function to minimize the amount of information that the neuronal net needs to learn and to make the feedback for the learning as direct as possible.</p><p>Edit (Oct.2020): Now that there are obstacles or other cars on the track the approach I have taken no longer works. However, it could be extended by taking into account where the obstacles are when deciding for a steering direction. I am curious to see if anyone implements this approach or what else people come up with.</p><p>The scoring function and parameters used for training can be found here (leave me a star if you have a github account ;)<br><a href="https://github.com/falktan/deepracer">https://github.com/falktan/deepracer</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a0dca1175fb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/twodigits/aws-deepracer-how-to-train-a-model-in-15-minutes-3a0dca1175fb">AWS Deepracer — How to train a model in 15 minutes</a> was originally published in <a href="https://medium.com/twodigits">NEW IT Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The AWS DeepRaceing Championship at re:invent in Las Vegas and how I got there]]></title>
            <link>https://medium.com/twodigits/the-aws-deepraceing-championship-at-re-invent-in-las-vegas-and-how-i-got-there-2595394427d1?source=rss-bc3dfdb706f------2</link>
            <guid isPermaLink="false">https://medium.com/p/2595394427d1</guid>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[accenture]]></category>
            <category><![CDATA[deepracer]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Falk Tandetzky]]></dc:creator>
            <pubDate>Mon, 16 Dec 2019 18:43:58 GMT</pubDate>
            <atom:updated>2019-12-19T17:07:02.169Z</atom:updated>
            <content:encoded><![CDATA[<iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FHWWs7oN7rRc%3Ffeature%3Doembed&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DHWWs7oN7rRc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FHWWs7oN7rRc%2Fhqdefault.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/015a20e59d43ee0901f715dc3f4c65c3/href">https://medium.com/media/015a20e59d43ee0901f715dc3f4c65c3/href</a></iframe><h3>What is DeepRacer?</h3><p>From the <a href="https://docs.aws.amazon.com/deepracer/latest/developerguide/what-is-deepracer.html">AWS-Documentation</a>:</p><blockquote><em>AWS DeepRacer is an integrated learning system for users of all levels to learn and explore reinforcement learning and to experiment and build autonomous driving applications. It consists of the following components:</em></blockquote><blockquote><em>An AWS Machine Learning service to train and evaluate reinforcement learning models in a simulated autonomous-driving environment.</em></blockquote><blockquote><em>AWS DeepRacer 1/18th scale RC vehicles capable of running inference on a trained AWS DeepRacer model for autonomous driving.</em></blockquote><blockquote><em>AWS DeepRacer for the participants to compete on model performances on a specific track in a simulated or real environment and have the results ranked on a virtual or physical leader board.</em></blockquote><p><strong>AWS-Robomaker </strong>provides a virtual environment. It creates a 3d-world with one of a couple of virtual race tracks and a car to race through it.</p><p><strong>AWS-Sagemaker </strong>provides an environment for training the car. The design of the neuronal network is provided. The input is the image received from a camera mounted on the car. The output is an action. One can choose which actions to use. For instance one might want two speeds, say 1 m/s and 2 m/s. Then one can choose for instance 5 steering angles that the car shall be able to steer to. In this example there would be a total of 2*5 = 10 possible actions available, each represented by an output neuron. The command associated with the neuron that receives the strongest input will be chosen to drive the car.</p><p>A 3 layer convolutional neuronal network is used, followed by a couple of fully connected layers. The algorithm to learn the weights is a <a href="https://arxiv.org/abs/1707.06347">proximal policy optimization algorithm</a>.</p><p>The AWS engineers left open which scoring function one wants to use and how to choose a couple of parameters for the learning process. However, they do provide examples for that that already kind of work.</p><h3>How did I get into this?</h3><p><strong>11/12:</strong> I receive an email saying “Are your AI skills in pole position? Then come to the Accenture AWS DeepRacer hackathon in Kronberg on 11/14”. I felt challenged. I look through the instructions and log into the AWS console to create my first virtual race car. The setup is super convenient. There is an AWS service that constructs all needed resources automatically.</p><p>I run the first couple of trainings with the default-scoring functions. I have a couple of ideas what one could do. But my feeling is that I am far too late to get into this topic simply because I started too late. Fortunately I read in some Q&amp;A section:</p><blockquote><em>Q: I do not have a working model. Can I participate anyway?<br>A: Just come around. You can still train models on the day and get in touch with other developers.</em></blockquote><p><strong>11/13:</strong> I do some client work. In the afternoon I finally decide to participate in the Accenture hackathon. I book a train and a hotel and drive to Frankfurt in the evening. At this point in time I do not even have a model that would manage to finish a single lap. I still only played around with a couple of sample reward functions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/0*W0RpIU_3w7IZ-vI9.png" /></figure><p>On the train I finish <a href="https://medium.com/twodigits/aws-deepracer-how-to-train-a-model-in-15-minutes-3a0dca1175fb">my first reward function</a>. I aim for a simple, robust model that can be trained quickly. When I arrive at the hotel I throw the code into the AWS console to look what happens. The result is: After about 15 minutes of training the car completes its first lap. One minute later it completes another lap and after this 50 more without a single crash! I am really happy with this result, but I still do not anticipate that this model would become the fastest in the worldwide Accenture competition the next day.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1nfc-k8UlFyNte7d.jpeg" /></figure><p><strong>11/14:</strong> I get up ridiculously early to get to the Accenture office in Kronberg and attend a call with a client. Then at 9:00 the actual race starts. I am surprised to see that my model works very well in the real world as well. It is a known issue, that transferring a model from a simulated environment to the real world can be very challenging. I try out a couple of ways to improve, but apart from some minor parameter adjustments I do not manage to find any better approach. I realize that starting off with a simple robust base model was the way to go for this competition.</p><p>And in the end that saves me the first place in the world’s largest private AWS DeepRacer league run by Accenture. I am a bit proud, after all this was a global competition consisting of 30 sites across 17 countries. This also is my entry pass for the AWS DeepRacer Championship Cup at re:invent in Las Vegas.</p><h3>Vegas Baby</h3><p><strong>12/01</strong>: I arrived in Vegas. My last training runs are two weeks ago. I did not have time to prepare much for what is ahead of me. I arrived at the hotel and finally find some time to train more models. After a little shock that AWS changed the user interface for the training and changed the behavior of the car in the virtual environment a bit, I manage to train a couple of models, none of which differs much from my initial approach.</p><p><strong>12/03</strong>: There are a total of 64 competitors from all over the world and from numerous companies. I am representing Accenture. All participants are assigned to one of 4 groups of 16. To make it to the next round I have to be among the first 4 in my group. After a couple of warm up runs, the race begins. Each participant gets 4 runs, 4 minutes each and the overall fastest lap counts. I end up making it to the 3rd place in my group. So I am in the final group of 16.</p><p><strong>12/04</strong>: The last 16 compete in a double elimination system. I drop out in the quarter final, i.e. I made it to the top 8 worldwide. I am pretty happy with making it that far. I am also very exhausted.</p><p><strong>12/05</strong>: Grand Final. Watch it on <a href="https://www.youtube.com/watch?v=xL34jRhg6ME">youtube</a>.</p><p>Details on how the model was trained can be found here:<br><a href="https://medium.com/twodigits/aws-deepracer-how-to-train-a-model-in-15-minutes-3a0dca1175fb">How to train a model in 15 minutes</a></p><p>The scoring function and parameters used for training can be found here (leave me a star if you learned something ;) <a href="https://github.com/TwoDigits/deepracer">https://github.com/TwoDigits/deepracer</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2595394427d1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/twodigits/the-aws-deepraceing-championship-at-re-invent-in-las-vegas-and-how-i-got-there-2595394427d1">The AWS DeepRaceing Championship at re:invent in Las Vegas and how I got there</a> was originally published in <a href="https://medium.com/twodigits">NEW IT Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Deepracer — How to train a model in 15 minutes]]></title>
            <link>https://falktan.medium.com/aws-deepracer-how-to-train-a-model-in-15-minutes-a07ab77fb793?source=rss-bc3dfdb706f------2</link>
            <guid isPermaLink="false">https://medium.com/p/a07ab77fb793</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[deepracer]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[accenture]]></category>
            <dc:creator><![CDATA[Falk Tandetzky]]></dc:creator>
            <pubDate>Mon, 16 Dec 2019 18:35:03 GMT</pubDate>
            <atom:updated>2020-10-31T21:07:16.096Z</atom:updated>
            <content:encoded><![CDATA[<h3>AWS Deepracer — How to train a model in 15 minutes</h3><p>In a <a href="https://medium.com/@falktandetzky/the-aws-deepraceing-championship-at-re-invent-in-las-vegas-and-how-i-got-there-33caa780898c">previous article</a> I described how I got into DeepRacer and what it is. This article is about the technical parts: How the scoring function was designed and why it works.</p><h3>What most people did</h3><p>Almost all scoring functions that I heard of are mostly scoring based on where the car currently is. The simplest approach is already provided as an example by AWS: just score high, if you are closer to the central line. More sophisticated approaches score high, if the car is close to a hand crafted more or less ideal line. E.g. a line that would be on the right before a left turn and on the left in the middle of the turn. The <a href="https://arxiv.org/abs/1707.06347">proximal policy optimization algorithm</a> is then able to learn what good actions have been and which not.</p><h3>What I did</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/467/1*YFuQ5_ivo46fYdwVa1ajiA.png" /><figcaption>The point the car should aim for is determined by simple geometry.</figcaption></figure><p>My approach was different: I used the steering direction of the car (which is available as a parameter in the scoring function during training).</p><p><a href="https://github.com/TwoDigits/deepracer">My scoring function</a> works as follows:</p><p>Draw a circle with a fixed radius around the car.</p><p>Take the intersection of this circle with the central line.</p><p>Aim for this point. I.e. The score is highest if the steering direction is such that the front wheels point there. The more it deviates the lower the score.</p><p>I adjusted the radius of the circle depending on the track and how narrow the corners are. And that is it. This <a href="https://medium.com/@falktandetzky/the-aws-deepraceing-championship-at-re-invent-in-las-vegas-and-how-i-got-there-33caa780898c">got me to the top 8 worldwide</a>. Other than that I only needed to adjust one of the training parameters: I set the discount factor to 0.5 rather than the default of 0.999.</p><h3>Why is this a good idea?</h3><p>After about 15 minutes of training the car completes its first lap. One minute later it completes another lap and after this it can run 50 more without a single crash. Other approaches usually took many hours if not days to converge to a somewhat stable model. But Why?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*R6kT5WNr0VF99YdgEaWapA.png" /><figcaption>18 Minutes of training on the re:invent 2018 track. After 42 attempts the model learned to run the track with 100% success rate.</figcaption></figure><p><strong>Quick response leads to faster learning</strong>: Reinforcement learning is about rewarding good behavior. The clearer the connection between the behavior and the score is, the quicker it can learn. Imagine you try to train your dog something. You <em>could </em>do this by providing a reward every evening, say, by providing more or better food, if it behaved well. Though this does work in principle it will take pretty much forever. If you want to train some tricks, you pretty much have to provide a reward immediately after something was done well.</p><p>It is the same with the scoring function in DeepRacer: One could in principle exclusively provide a score for finishing a lap. The PPO-Algorithm would eventually work out how to steer. But it will only improve if it ever finishes a lap, and even then would it barely learn with each success. In fact doing slightly better by providing a constant score for not crashing already allows the algorithm to slowly (but visibly) learn. Of cause the scoring according to position of the car is much more direct. Even still it only provides some kind of average rating for the last couple of frames that lead the car to the place it is now. There may be a couple of bad steering commands in the past that were only compensated by other commands later on. Both would end up being encouraged and only eventually would good commands end up with a better statistic and hence get encouraged most often.</p><p>Pointing to the “right” direction is as direct as it gets. It immediately tells the learning algorithm, if this steering direction is good or not. There is no averaging over multiple frames needed. This is also the reason I reduced the discount factor: the scoring does by no means rate good actions in the past (in contrast to scoring according to position). I could probably have reduced the discount factor even further. I really did not have time to play around with it yet.</p><p>This direct scoring is the main reason this approach learns so fast.</p><p><strong>Using local information makes learning easier</strong>: Ultimately the car has to drive without any knowledge of way points or where it is exactly. It can only work with the camera image and make a decision. This means that it is much easier to learn a scoring function that is only based on information that can be <em>seen</em>. I am using only information about the central line right in front of the car, which can usually be seen. A perfectly optimized ideal line would (at least in principle) also depend on parts of the track that can not even be seen at the time of making the decision. That does not mean that it is not possible to learn this more complicated path, just it is more difficult.</p><p><strong>Cutting corners is good</strong>: This should not be too surprising. It is faster because the path gets shorter this way. It also reduces the risk of running out of the track. Finally this also makes it easier to keep the eyes on the road. If the car is on the right in a left turn, the camera mostly sees green and only little of the track, so it is harder to decide in which direction to steer. Increasing the radius in the scoring function described above leads to a direction that is cutting corners in a stronger way.</p><h3>Isn’t this cheating?</h3><p>Well it feels a bit like cheating at first. Right? After all there is an elaborate training algorithm that can figure out how to steer. An now we tell the car directly how it should steer using “complicated mathematics”.</p><p>That is true, but the challenge is <em>not </em>to learn what is a good direction to steer to <em>given the way points and the coordinates of the car</em>. The difficult part is to train a neuronal network in such a way that it can find this direction <em>based on the camera image</em>. It is only in the virtual environment that we can use the position of the car and the track because this information is so kindly provided by the virtual environment.</p><p>What providing scores based on position does is it puts the extra burden of learning how the steering direction affects the position on the neuronal network.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FHWWs7oN7rRc%3Ffeature%3Doembed&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DHWWs7oN7rRc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FHWWs7oN7rRc%2Fhqdefault.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/015a20e59d43ee0901f715dc3f4c65c3/href">https://medium.com/media/015a20e59d43ee0901f715dc3f4c65c3/href</a></iframe><p>In summary, one should put as much knowledge of the problem into the scoring function to minimize the amount of information that the neuronal net needs to learn and to make the feedback for the learning as direct as possible.</p><p>Edit (Oct.2020): Now that there are obstacles or other cars on the track the approach I have taken no longer works. However, it could be extended by taking into account where the obstacles are when deciding for a steering direction. I am curious to see if anyone implements this approach or what else people come up with.</p><p>The scoring function and parameters used for training can be found here (leave me a star if you have a github account ;)<br><a href="https://github.com/falktan/deepracer">https://github.com/falktan/deepracer</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a07ab77fb793" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The AWS DeepRaceing Championship at re:invent in Las Vegas and how I got there]]></title>
            <link>https://falktan.medium.com/the-aws-deepraceing-championship-at-re-invent-in-las-vegas-and-how-i-got-there-33caa780898c?source=rss-bc3dfdb706f------2</link>
            <guid isPermaLink="false">https://medium.com/p/33caa780898c</guid>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deepracer]]></category>
            <category><![CDATA[accenture]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Falk Tandetzky]]></dc:creator>
            <pubDate>Mon, 16 Dec 2019 16:46:47 GMT</pubDate>
            <atom:updated>2019-12-18T12:20:25.827Z</atom:updated>
            <content:encoded><![CDATA[<iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FHWWs7oN7rRc%3Ffeature%3Doembed&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DHWWs7oN7rRc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FHWWs7oN7rRc%2Fhqdefault.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/015a20e59d43ee0901f715dc3f4c65c3/href">https://medium.com/media/015a20e59d43ee0901f715dc3f4c65c3/href</a></iframe><h3>What is DeepRacer?</h3><p>From the <a href="https://docs.aws.amazon.com/deepracer/latest/developerguide/what-is-deepracer.html">AWS-Documentation</a>:</p><blockquote>AWS DeepRacer is an integrated learning system for users of all levels to learn and explore reinforcement learning and to experiment and build autonomous driving applications. It consists of the following components:</blockquote><blockquote>An AWS Machine Learning service to train and evaluate reinforcement learning models in a simulated autonomous-driving environment.</blockquote><blockquote>AWS DeepRacer 1/18th scale RC vehicles capable of running inference on a trained AWS DeepRacer model for autonomous driving.</blockquote><blockquote>AWS DeepRacer for the participants to compete on model performances on a specific track in a simulated or real environment and have the results ranked on a virtual or physical leader board.</blockquote><p><strong>AWS-Robomaker </strong>provides a virtual environment. It creates a 3d-world with one of a couple of virtual race tracks and a car to race through it.</p><p><strong>AWS-Sagemaker </strong>provides an environment for training the car. The design of the neuronal network is provided. The input is the image received from a camera mounted on the car. The output is an action. One can choose which actions to use. For instance one might want two speeds, say 1 m/s and 2 m/s. Then one can choose for instance 5 steering angles that the car shall be able to steer to. In this example there would be a total of 2*5 = 10 possible actions available, each represented by an output neuron. The command associated with the neuron that receives the strongest input will be chosen to drive the car.</p><p>A 3 layer convolutional neuronal network is used, followed by a couple of fully connected layers. The algorithm to learn the weights is a <a href="https://arxiv.org/abs/1707.06347">proximal policy optimization algorithm</a>.</p><p>The AWS engineers left open which scoring function one wants to use and how to choose a couple of parameters for the learning process. However, they do provide examples for that that already kind of work.</p><h3>How did I get into this?</h3><p><strong>11/12:</strong> I receive an email saying “Are your AI skills in pole position? Then come to the Accenture AWS DeepRacer hackathon in Kronberg on 11/14”. I felt challenged. I look through the instructions and log into the AWS console to create my first virtual race car. The setup is super convenient. There is an AWS service that constructs all needed resources automatically.</p><p>I run the first couple of trainings with the default-scoring functions. I have a couple of ideas what one could do. But my feeling is that I am far too late to get into this topic simply because I started too late. Fortunately I read in some Q&amp;A section:</p><blockquote>Q: I do not have a working model. Can I participate anyway?<br>A: Just come around. You can still train models on the day and get in touch with other developers.</blockquote><p><strong>11/13:</strong> I do some client work. In the afternoon I finally decide to participate in the Accenture hackathon. I book a train and a hotel and drive to Frankfurt in the evening. At this point in time I do not even have a model that would manage to finish a single lap. I still only played around with a couple of sample reward functions.</p><figure><img alt="Improvement of model during 18 minute training" src="https://cdn-images-1.medium.com/max/748/1*R6kT5WNr0VF99YdgEaWapA.png" /></figure><p>On the train I finish <a href="https://medium.com/@falktandetzky/aws-deepracer-how-to-train-a-model-in-15-minutes-a07ab77fb793">my first reward function</a>. I aim for a simple, robust model that can be trained quickly. When I arrive at the hotel I throw the code into the AWS console to look what happens. The result is: After about 15 minutes of training the car completes its first lap. One minute later it completes another lap and after this 50 more without a single crash! I am really happy with this result, but I still do not anticipate that this model would become the fastest in the worldwide Accenture competition the next day.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fLrRcuHLoEwffYwXTHHE8w.jpeg" /></figure><p><strong>11/14:</strong> I get up ridiculously early to get to the Accenture office in Kronberg and attend a call with a client. Then at 9:00 the actual race starts. I am surprised to see that my model works very well in the real world as well. It is a known issue, that transferring a model from a simulated environment to the real world can be very challenging. I try out a couple of ways to improve, but apart from some minor parameter adjustments I do not manage to find any better approach. I realize that starting off with a simple robust base model was the way to go for this competition.</p><p>And in the end that saves me the first place in the world’s largest private AWS DeepRacer league run by Accenture. I am a bit proud, after all this was a global competition consisting of 30 sites across 17 countries. This also is my entry pass for the AWS DeepRacer Championship Cup at re:invent in Las Vegas.</p><h3>Vegas Baby</h3><p><strong>12/01</strong>: I arrived in Vegas. My last training runs are two weeks ago. I did not have time to prepare much for what is ahead of me. I arrived at the hotel and finally find some time to train more models. After a little shock that AWS changed the user interface for the training and changed the behavior of the car in the virtual environment a bit, I manage to train a couple of models, none of which differs much from my initial approach.</p><p><strong>12/03</strong>: There are a total of 64 competitors from all over the world and from numerous companies. I am representing Accenture. All participants are assigned to one of 4 groups of 16. To make it to the next round I have to be among the first 4 in my group. After a couple of warm up runs, the race begins. Each participant gets 4 runs, 4 minutes each and the overall fastest lap counts. I end up making it to the 3rd place in my group. So I am in the final group of 16.</p><p><strong>12/04</strong>: The last 16 compete in a double elimination system. I drop out in the quarter final, i.e. I made it to the top 8 worldwide. I am pretty happy with making it that far. I am also very exhausted.</p><p><strong>12/05</strong>: Grand Final. Watch it on <a href="https://www.youtube.com/watch?v=xL34jRhg6ME">youtube</a>.</p><p>Details on how the model was trained can be found here:<br><a href="https://medium.com/@falktandetzky/aws-deepracer-how-to-train-a-model-in-15-minutes-a07ab77fb793">How to train a model in 15 minutes</a></p><p>The scoring function and parameters used for training can be found here (leave me a star if you learned something ;) <a href="https://github.com/TwoDigits/deepracer">https://github.com/TwoDigits/deepracer</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=33caa780898c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>