<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Taka Ishikawa on Medium]]></title>
        <description><![CDATA[Stories by Taka Ishikawa on Medium]]></description>
        <link>https://medium.com/@taka2code?source=rss-d039534489e8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*90ynuYIiBMw4hFTpqwKERw.jpeg</url>
            <title>Stories by Taka Ishikawa on Medium</title>
            <link>https://medium.com/@taka2code?source=rss-d039534489e8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 11:09:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@taka2code/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Tech Stack Preliminary Research]]></title>
            <link>https://medium.com/@taka2code/tech-stack-preliminary-research-ebae7c87e83f?source=rss-d039534489e8------2</link>
            <guid isPermaLink="false">https://medium.com/p/ebae7c87e83f</guid>
            <dc:creator><![CDATA[Taka Ishikawa]]></dc:creator>
            <pubDate>Sat, 12 Sep 2020 13:12:39 GMT</pubDate>
            <atom:updated>2020-09-12T13:12:39.640Z</atom:updated>
            <content:encoded><![CDATA[<h3>Web Dev Tech Stack Preliminary Research</h3><p>Value-adding tech stack</p><p>As a software engineer working at a startup, I wanted to be more productive in learning new technologies. I’d like to learn by actions but before diving in, would like to keep some of the notes on the possible modern stack.</p><h3>React</h3><p>Javascript library to build UIs:</p><ul><li>Declarative: define the views and let it handle data updates and renderings.</li><li>Component-based: component encourages modularity and organizes the data flow.</li><li>Platform agnostic: developed on web and natives (React Native)</li></ul><p>I’ve been using React for the past two years for the web and also native applications development (React Native). The packaged javascript is so well abstracted to achieve the declarative coding. What you need to concern primarily for React development are the components structure and data flow and once you specify these clearly then the development goes so smoothly without spending extra time on templating and imperative implementations. The code reuse got so much improved with the introduction of the hooks, which enables extracting the logics to share across different components that can contain local states to be updated on.</p><p>This simplicity propagates team development as well that collaboration is each with communicating data flow before head and can split into a component or a group of components to assign. Once you set it upright, as long as you keep updating the best practices, the productivity gain on frontend development is huge.</p><h3>TypeScript</h3><p>TypeScript is a <em>static type checker.</em></p><p><em>Static checking: d</em>etecting errors in code without running</p><p><em>Type checking</em>: determining what’s an error and what’s not based on the kinds of values being operated on.</p><p>TypeScript is has a superset of syntax and types thus it’s easy to transition gradually. Runtime behaviors are guaranteed to be the same as the underlying javascript code. It’s just another layer on top of javascript that makes it code easier to avoid syntax and type error before runtime, which reduces development cost.</p><h3>Node.js</h3><p>Node.js is a run time environment of your browser. It integrates the event loop to achieve non-blocking run time and underlying engine that is written by C++ for the fast execution of your original JS code. Not only it supports the natively supported web API but npm provides ample packages to enrich your javascript application development.</p><h3>Apollo GraphQL</h3><blockquote>Essentially, you can think of GraphQL as a system for calling many nested endpoints in one request. Almost like a multiplexed REST.</blockquote><p>The quote is from the article written by Apollo called <a href="https://www.apollographql.com/blog/graphql-vs-rest-5d425123e34b/">GraphQL vs. REST</a> and it explains really well on pros and cons on GraphQL compared to REST.</p><p>RESTful application tends to complete resource fetching and constructs of the final shape of all data. We tend to add more endpoints to satisfy the growing needs of clients and oftentimes, the returned data has unnecessary data otherwise we add a new endpoint to slice the resource.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ebae7c87e83f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Principles in Acceleration of Software Development]]></title>
            <link>https://medium.com/@taka2code/accelerate-software-development-9eac256da30b?source=rss-d039534489e8------2</link>
            <guid isPermaLink="false">https://medium.com/p/9eac256da30b</guid>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[psychology]]></category>
            <dc:creator><![CDATA[Taka Ishikawa]]></dc:creator>
            <pubDate>Sun, 15 Jul 2018 04:35:44 GMT</pubDate>
            <atom:updated>2018-07-23T00:09:56.797Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why it’s important but hard to keep moving faster.</h3><p>Move fast. New features comes to the hands of users earlier and they would be happy if these are what they wanted. Having users feedbacks earlier reinforces product-market fit calibration. Eventually that’s key to make any project successful. But there’s one caution here. We should be aware of the second order of it: moving faster over time.</p><p>Our tendency to sacrifice the long-term efficiency for near-term one can be explained for an obvious reason: measuring how fast we currently are is substantially easier than how fast we will be in the future. We prefer certain choice and pick the former over the second. It accumulates over time and one day we realize we move as slow as a turtle does.</p><p>Also, as company scales, the complexity in organization, communication, product, and engineering increases, which disproportionate with the speed of development. Any system can be exposed to getting slowed down by the bottleneck.</p><p>How can we avoid moving slower in the future? The key is to set a principle and apply it as a set of strategies and tactics in whole levels. Here, let’s go over the principles of acceleration of software development.</p><h3>Generalize a set of resolutions: the principles</h3><p>Why principles? Because it gives greater extensibility to the one’s choices in different contexts in software development. It sets a paramount where anyone can construct a variety of concrete strategy, tactic, and practices according to a unique situation that one has.</p><ol><li>Anticipate Unknown of Unknown</li><li>Take compound interest into account</li><li>Make components moduled</li><li>Transparentise information</li><li>Automate process</li></ol><h4>1. Anticipate Unknown of Unknown</h4><p>The first principle should be something so fundamental that it constructs other principles. Anticipate unknown of unknown. What is unknown of unknown? Well, it’s easy to know that you are reading this sentence (known of known). It’s easy to know you don’t know what the author is thinking right now (known of unknown). It’s hard to know we don’t know what we don’t know (unknown of unknown). Unknown of unknown means something completely outside of our consciousness. It’s a total surprise when we know unknown.</p><p>We should always be aware that unknown of unknown exists and it requires unbounded time to notice known of unknown and turn it into known of known. We need to proactively expand the scope of expectation to capture the unknown of unknown as early as possible. This entails the necessity of consistent prospection, inspection, and retrospection. At any stage of development, we want to integrate the above among the whole team.</p><h4>2. Take compound interest into account</h4><p>The second principle is a subset of the first principle relates to the linearity in the mode of human cognition on any time scale. We tend to believe the rate of change in any observation is constant and dismiss the factors that have compound interests as its attributes.</p><p>Identifying these features and taking according actions are key to accelerate the development. As an instance, writing tests has a substantial compound interest. In the face of deciding whether we should write tests, firstly we should be aware that it compounds. Then try estimating its compound rate. Although it depends on the context, the larger the project is, the more significant compound rate will be. And needless to say, the effect reinforces over time.</p><h4>3. Make components moduled</h4><p>The third principle is a more practical interpretation of the above. Over time, the complexity of any system increases. This introduces the need of understanding the interaction between the components of the system. The number of the component and its size would impact the speed of development. Memory size of human brain is limited. The more components the system have, the harder developers to understand the interactions between them. The larger the size of a component is, the more restrictive the development will be.</p><p>At any layer of development, architecture, design, implementation, quality assurance, and deployment, apply and practice modularity. Module serves a specific functionality as a unit of system, hides internal details from outside of the world, provides an interface to let others to consume it’s functions, and reduces interdependency between components. Thus it improves understandability of the interaction and flexibility of the feature development in the system in long-term.</p><h4>4. Transparentise information</h4><p>The forth principle scopes out to the environment of development. Software development happens in a team environment in majority of the cases. No sufficient information is shared among the team leads to have more unknowns of unknown. The team will stumble upon the unexpected surprises more often and suffers the slower development.</p><p>Transparency is a key for a team to coordinate efficiently by understanding the updated context of the project. Frequent, concise, clear, and open communication should be consistently encouraged throughout the organization.</p><h4>5. Automate procecss</h4><p>The fifth principle addresses a side effect of practicing the above principles, which may decreases the current velocity of the development by adding extra work. Having rigorous meta analysis, writing tests, sharing status among the team, all takes time. At some point, it may become too repetitive and redundant.</p><p>Make repetitive tasks executable by a single command or scheduled as much as possible. From infrastructure level to coding level, lots of automation tasks are available but dismissed.</p><h3>Conclusion</h3><p>It is not trivial task to retain the velocity of software development. With that, accelerating the software development is far harder to achieve. However, it’s addressable. It’s useful to understanding the nature of cognitive limitation of human minds and construct the set of principles to apply strategy and tactics.</p><p>The benefit of incorporating ideas as a set of principles is the density and generality. It’s easy to retain in memory and apply to different set of contexts. A few entails many. The simple one adapt.</p><p>Hope you enjoyed.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9eac256da30b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Writing Good Tests]]></title>
            <link>https://medium.com/@taka2code/writing-good-tests-9be0b948da60?source=rss-d039534489e8------2</link>
            <guid isPermaLink="false">https://medium.com/p/9be0b948da60</guid>
            <category><![CDATA[testing]]></category>
            <dc:creator><![CDATA[Taka Ishikawa]]></dc:creator>
            <pubDate>Sun, 10 Jun 2018 23:54:48 GMT</pubDate>
            <atom:updated>2018-06-11T00:57:13.075Z</atom:updated>
            <content:encoded><![CDATA[<p>Why, How, and Evaluation</p><h3>Why?</h3><p>To summarize, the benefit of writing test is to make your program more scalable later time. Scalability of your program comes with its flexibility in design and implementation of every level of design and implementation, from statements to architecture. And when the program is not flexible, the road block is the amount of potential bugs that you may produce through a change made in your codebase. The opportunity cost you need to pay for fixing bugs introduced by a new feature accumulate if you do not think about this seriously as you add mode lines in your code.</p><p>The tests attest every units of your code to be adoptable with any potential behavioral expectations. Input data availability, data types, value range, configurations, constants, change all the time. If a function does not consider these factors, sooner or later, it bubble up as a bug to prevent your program to behave correctly. A test is a encapsulation of the expected state change of your program interacting with the part of your program you are testing against. It makes the unit of your program to be predictable for its behavior given variety of state change you may expect in the future.</p><p>Reducing the cost of sneaking bugs preventably is obvious benefit of writing tests but counter-intuitive one is that the test allows programmer to write better code in design and implementation faster. This is because tests encourage programmers to think of the usage of the method and its interaction with other components of the program before designing and implementing it. Clear understanding about the goal of the method and the anticipated state the method is utilized is the key to plan how it should behave in its context.</p><p>Test allows a programmer to assess the usability of the method objectively. Code of quality can be measured mainly in its correctness, performance, and readability but if there’s anything can be added in this list here, it would be usability. Test regards a portion of your codebase as if it is an small independent software that serves a goal of its user who consumes it. The user is the programmer, yourself or other programmer, who write a code along with the function. Tests gives better understanding the requirements and usage of the method and tells what quality should be satisfied, not too much or too less.</p><h3>How?</h3><p>When we should write a test?</p><p>Fowler said:</p><blockquote>Whenever you are tempted to type something into a print statement or a debugger expression, write it as a test instead.</blockquote><p>Whenever you’re implementing a class or method you realize the necessity to figure out what’s going internally. This indicate you have a blind spot about the internal implementation which introduced by its ambiguity of inputs, outputs, and behavior of public methods used in the method. Writing test for the method clarify the inputs/outputs requirements. If no test exists for the public method used then you now the time to write the test for it to disambiguate its inputs/outputs. Most of the case, the class or method you are going to write has a certain level of ambiguity before it is actually implemented, therefore, it’s much more likely you write a test before write a code.</p><p>What for should I test?</p><p>It’s important to know the scope of tests.</p><ul><li><strong>Unit</strong> test tests a class or method, independent unit of the functionality for a module.</li><li><strong>Integration</strong> test tests a interaction of a set of classes and methods, independent unit of the functionality of a feature.</li><li><strong>Regression</strong> test tests a new state brought by a change in its codebase, independent unit of the functionality of a entire codebase.</li><li><strong>Acceptance</strong> test tests a user experience, independent unit of the functionality of a product or service.</li></ul><p>As you write any code, if you expect a test to be unit test but write in the way it fallen in the category of integration tests (testing more than one class or method), it should raise a flag about how you test things.</p><p>For example, for unit test, it is written in the way that:</p><ol><li>test one method</li><li>provide some specific arguments to the method</li><li>confirm the expected result returned</li></ol><p>It should be designed and implemented independently satisfy the methods’ usage requirement. To achieve this:</p><ul><li>no use of external method to produce the expected result. hardcode it instead.</li><li>simplify a test. write as many test as you need to cover the interesting code path to see its coverage.</li></ul><p>You basically can apply the same principle of the independence to other scope of tests too. Never scope out too much (write unit test as if it’s integration test). Scope in as much as you can as long as it does not break down the purpose of the tests too much (write integration test as if it’s a set of unit tests).</p><h3>Evaluation?</h3><p>To keep track of improving the tests quality of a project, there should be a set of measurements to evaluate. This sounds tough but when we get back to the original reasoning behind why we should write test, ideas comes down more easier than you think.</p><p>One of the scoring measure for it is the time spent for bug fixes per a new feature introduced. If the tests strategy is well designed and executed, we see better score for the feature scalability in your code. If the team spends too much time to extinguish fires per new feature implementation as your codebase grow, it signals the poor quality of the code in design and implementation. If tests coverage is below a certain level, increasing it may improve this feature scalability score. Not only the coverage, quality of the tests should also be assessed using the measure of independence&#39;s for each scope of testing (unit, integration, regression, and acceptance).</p><p>The other is, thought it’s more qualitative measure, how the engineers in the team feel each component of code is usable and easy to develop upon. Majority of the code, if it severs as a production or working in staging, satisfy its correctness and mostly performance. Readability is a clearly measurable regardless of its existence of tests. But usability of the components of the codebase can vary with or without the tests and its quality. If you ask around the engineers, especially new engineers working on the codebase, then they raise complains about its codebase quality, it might be good time to consider utilizing tests as a strategy to improve it.</p><h3>Conclusion:</h3><p>Any software product is decomposed into a set of independent classes and methods. Regardless it’s complexity, without clearly define its usage and requirements beforehand, it’s quite easy to produce unexpected behaviors of your software in its interactions between those components. Tests are here for giving clearer understanding about the expected and actuals of those independent components and its interactions. It makes the software more scalable by reducing the cost of bugs and boost the velocity of development in long-term.</p><p>References:</p><ul><li><a href="https://blog.codinghorror.com/i-pity-the-fool-who-doesnt-write-unit-tests/">I Pity The Fool Who Doesn&#39;t Write Unit Tests</a></li><li><a href="https://blog.codinghorror.com/good-test-bad-test/">Good Test / Bad Test</a></li><li><a href="https://github.com/mockito/mockito/wiki/How-to-write-good-tests">How to write good tests</a></li><li><a href="https://medium.com/javascript-scene/what-every-unit-test-needs-f6cd34d9836d">5 Questions Every Unit Test Must Answer</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9be0b948da60" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Understand a Codebase]]></title>
            <link>https://medium.com/@taka2code/how-to-read-code-to-understand-it-well-2386ea586c9a?source=rss-d039534489e8------2</link>
            <guid isPermaLink="false">https://medium.com/p/2386ea586c9a</guid>
            <dc:creator><![CDATA[Taka Ishikawa]]></dc:creator>
            <pubDate>Tue, 29 May 2018 05:34:40 GMT</pubDate>
            <atom:updated>2018-07-01T00:36:03.714Z</atom:updated>
            <content:encoded><![CDATA[<h4>5 Minimal Variable Strategies</h4><h3>Why Understanding Code Matters</h3><p>Code is an expression of problem solving process to achieve a goal. It’s not trivial that depending on the quality of the code the outcome we gain does differ. Writing quality code means you think thoroughly and take care of its readability for other engineers. Knowing how the code works and judging whether the code is high quality or not is the crucial skill that every engineer want to have. Here I list some of my hypothetical strategy to understand the code better.</p><h3>Even before start looking at the code base</h3><h4>Understand the product or service from the user’s perspective</h4><p>Needless to say, understanding what’s the interpretation of the codebase is the most crucial. Who is user? How is it used? What the intension behind features and functionalities? Answering user-centered questions by tracing the current use flow gives a paramount of further codebase understanding.</p><p>Why software exists? To give value to users.</p><h3>Strategies</h3><h4>Strategy 1: Ask any engineer who are familiar with the architecture and design of the codebase</h4><p>An obvious but one of the most effective strategies to understand the codebase. The larger the code base is, the more effective this strategy becomes. Catch anyone who know well the codebase and start asking the thoughts process when they architect and design the codebase. Focus on the conceptual understanding first, architecture second, and design third. Document what you understand to reinforce the memory, bring better follow-up questions, and share learnings with other new engineers. Ask any question that pops in your head on the spot. No silly questions here.</p><h4>Strategy 2: Trace the usage from the integration tests</h4><p>Integration tests are great place to start since it describes how the components should orchestrate with each other. Given IDE, following the method definitions and method usages, you can clearly see the track of expected result from each input. Each modules are glued together in the way that each works independently to achieve a common goal if system is designed well. The key here is to establish the cohesive understanding about the connections between each module in the context of typical usage patterns.</p><h4>Strategy 3: Fix multiple bugs whose causes are already identified in the way that these cover all major usages</h4><p>Now actually taking a closer look at the codebase design and implementation. The best way to do is actually working on fixing well defined bugs. If the expectation and its actual result is clear and the approach to fix it is known, you can focus on understanding how the code is designed and implemented. Also the other engineers can easily sit aside and give you a further insights about the portion you are working on. This activity will give you broad and clear understanding on the codebase.</p><h4>Strategy 4: Adding test cases for the portion of code that you want to gain better understanding</h4><p>Adding tests are actually great way to dig in the further understanding about the codebase and even discover unknown issues. The core idea of testing code is to understand the set of expectations and actual results given variety of inputs. Given a portion of the codebase that you want to get familiar with, you now need to simulate the possible set of inputs and outputs under the logical instructions described there. Then the simulation should be attested through execution of tests cases. In either case of the tests pass or fail, you can gain better understanding about the implementation of the code.</p><h4>Strategy 5: Implement new features.</h4><p>If you gained baseline understanding on the codebase, implementing a new feature would give you critical view at the codebase. Since you now have a certain freedom in its design and implementation space for adding a new feature to the codebase, you may find interesting issues in its vertical scalability of the codebase. If you are experiencing a harder implementation process, some of the portion of the codebase has less flexible about adding new code. Then you get to suggest better design and implementation among those, which reinforces your understanding on the codebase.</p><h3>Conclusion:</h3><p>Reading code better precedents writing code better. Developing the judgement to see good and bad codes requires intentional effort to do so by the live code which is designed, implemented, and maintained by others. Though minimal variable strategies to understand the code base better might be helpful in some context, the most important key factor for it is the genuine curiosity about how the codebase works. Curiosity about how the codebase works flips other things: if you are curious enough, you want to use, test, find bugs, add fixes, refact, and design better.</p><p>Questioning indicates curiosity. If you have lots of questions in front of the codebase, you now are more than ready to understand the codebase.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2386ea586c9a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning Distributed Computing]]></title>
            <link>https://medium.com/@taka2code/learning-distributed-computing-problems-and-properties-229bec7d6bb9?source=rss-d039534489e8------2</link>
            <guid isPermaLink="false">https://medium.com/p/229bec7d6bb9</guid>
            <category><![CDATA[bitcoin]]></category>
            <category><![CDATA[distributed-computing]]></category>
            <dc:creator><![CDATA[Taka Ishikawa]]></dc:creator>
            <pubDate>Fri, 23 Feb 2018 11:23:57 GMT</pubDate>
            <atom:updated>2018-07-01T00:18:09.218Z</atom:updated>
            <content:encoded><![CDATA[<h4>— Problems and Properties</h4><h3>What and Why</h3><p>If we want to serve a product or service that requires many complex tasks at scale, single computer is not enough no matter how it’s powerful. So let’s make the computers work as a team! Distributed computing enables a solution that scales. But there’s a trade-off using distributed computing. Like us, computers can miscommunicate and draw a unpredictable outcome. This is called Consistency problem. How we can achieve the predictable scaling distributed system?</p><h3>Consistency</h3><p>— Attribute that Guarantee Everybody Get the Same Result.</p><p>Consistency is a promise made between a programmer and a system. If programmer follow the role, the system guarantee the results of operations such as reading, writing, updating, deleting are predictable. Technically, if the system is fully consistent, all nodes see the same data at the same time. In reality, complete consistency is hard to achieve because of the stochastic nature of the world.</p><h3>Consensus</h3><p>— Rules to Decide a Single Piece of Truth among Nodes of a Distributed System.</p><p>Consensus problem requires agreement among a number of processes for a single data value. There’s many reasons disagreements can happen such as dishonesty or unintentional error in one or more processes. A consensus protocol tolerating halting failures must satisfy the following properties.</p><ul><li>Termination: every correct process decides some value.</li><li>Validity: if all processes propose the same value v, then all correct processes decide v.</li><li>Integrity: every correct process decides at most one value, and if it decides some value v, then v must have been proposed by some process.</li><li>Agreement: every correct process must agree on the same value.</li></ul><h3>Atomicity</h3><p>— A Property of Writing of a Data: All or Nothing</p><p>When modifying an shared object, if the multiple operations applied, atomicity guarantees the operations complete perfectly as expected if condition met or do change nothing if something wrong. It guarantees all or nothing. It guarantees complete success or complete failure.</p><h3>Linearizability</h3><p>— A Consistency Model for High Consistency</p><p>Linearizability introduces more restrictive definitions of atomicity. Linearizability provides a real-time guarantee on the behavior of a set of single operations on a single object. Modification of a object appears to take place instantaneously and all later accesses are supposed to return the modified value or a value modified even later. This models has a strong guarantee on consistency however may draw a low availability to keep the premise.</p><h3>Eventual Consistency</h3><p>— A Consistency Model for Modest Consistency for High Availability</p><p>On the other hand, eventual consistency model guarantees high availability with low consistency. It does guarantee the reads return the same value at some point but not always so. Often it’s called BASE (<strong>B</strong>asically <strong>A</strong>vailable, <strong>S</strong>oft state, <strong>E</strong>ventual consistency). How it works is that it embeds a process of exchanging versions or updates of data between servers and choosing an final state if there’s any conflicts. A basic approaches are ‘last writer wins’, user-specified conflict handler, or Timestamps/vector clocks.</p><h3>Fault Tolerance</h3><p>— A Property of a System that Perform Still in the Failure of the Components</p><p>Fault tolerance system does not degrade or does so in smaller margin its performance in the event of failure in any component that belongs to the system. Adding fault tolerance to the system should be highly considered when it has critical, likely-to-fail component and the cost of installment is fairly low.</p><p>Now we learned about the major problems and properties in distributed computing system. In the next series, let’s cover some of the major protocols that is known to be fault-tolerance to achieve consistency.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=229bec7d6bb9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Grow Neural Nets — Part 2: Implementing First Hyperparameter — Learning Rate]]></title>
            <link>https://medium.com/@taka2code/grow-neural-nets-part-2-implementing-first-hyperparameter-learning-rate-12b89f1d3c8e?source=rss-d039534489e8------2</link>
            <guid isPermaLink="false">https://medium.com/p/12b89f1d3c8e</guid>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Taka Ishikawa]]></dc:creator>
            <pubDate>Wed, 22 Feb 2017 01:02:01 GMT</pubDate>
            <atom:updated>2017-02-22T01:02:01.090Z</atom:updated>
            <content:encoded><![CDATA[<h4>Start building your own neural nets and evolve it step by step to advanced machine learning models</h4><h3>Grow neural networks step by step</h3><p>In my past entry, <a href="https://medium.com/@tk2bit/a-lightning-path-to-understand-neural-networks-nns-1ba32d1b1785#.6mxyv3cyf">Learning Neural Nets</a>, I introduced how to start learning <strong>neural nets, one of the most important concepts in modern machine learning models.</strong> Once you start understanding how neural nets, you will wonder how advanced machine learning models such as Recurrent Neural Nets (RNNs), Convolutional Neural Nets (CNNs), or Generative Adversarial Networks (GANs).</p><p>Imagine you build your own advanced machine learning models from a minimal neural nets. Isn’t that cool? The purpose of this <strong>”Grow Neural Networks” </strong>series is to guide you through the journey of evolving the seed neural nets to more sophisticated models.</p><p>In first entry of this series, <a href="https://medium.com/@tk2bit/grow-neural-nets-part-1-intro-to-hyperparameters-39e59b212118#.jkrprdssq">Part1: Intro to Hyperparameters</a>, I introduced the concept of hyperparameters in machine learning and different kinds of ones commonly used in particularly neural nets trainings.</p><p>In this particular entry, I would like to introduce how to grow your neural nets by adding one of the hyperparameters, <strong>learning rate!</strong></p><p>Get excited? Let’s start!!</p><h3>Setup Virtual Environmen</h3><p>Before jumping in, I highly recommend setup virtual environment using the open data science platform, Anaconda. It comes with the necessary tools for data science and machine learning. The benefits of using Conda virtual environment are its isolation from your local system and easy package management. I put another post to set up and manage Conda environment on Mac OS X (&gt; El Captain): “<a href="https://medium.com/@tk2bit/how-to-set-up-and-maintain-conda-virtual-environment-for-your-python-projects-643c8f0c81a1#.at6s6vw9k">How to Set Up and Maintain Conda Virtual Environment for your Python Projects (on Mac OS X El Capitan (10.11.6))</a>”.</p><h3>The Base: 3-Layer Neural Nets</h3><p>Now, presumably you installed and configured the virtual environment and about to set off to the great journey of growing own neural nets. Where to start? We need a base where we start growing neural nets bit by bit. Here it is.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/37ca950f55090b719a3f0bc8d08e77c3/href">https://medium.com/media/37ca950f55090b719a3f0bc8d08e77c3/href</a></iframe><p>Please read throughly its implementation and try to pay attention to details, what each line represents and why it is like that. The base class include the very basic concept like activation, forward propagation, back propagation, etc.. We are going to grow the neural nets by adding more features to make the model powerful. Hyperparameters are the key ingredients to build a better machine learning model and understanding it gives greater joy and wisdom of learning neural nets.</p><h3>The First Growth: Learning Rate</h3><p>As we saw before in the neural nets GUI simulator, learning rate was one of the numeric hyperparams for us to tune. Let’s bring the description of this param up again from <a href="https://medium.com/@tk2bit/grow-neural-nets-part-1-intro-to-hyperparameters-39e59b212118#.ur4kct6d6">the previous post</a>:</p><p><strong>Learning Rate </strong>: 0.00001, 0.0001, 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10…</p><p>Step size of adjustments of parameters in each iteration. Though how it impacts is different between machine learning architectures, it should be optimized to find its best value to achieve faster training process and more accurate result. Higher value give faster learning but it may cause the model to fail to converge. Lower value gives higher chance to converge but may cause the model to learn too slow and even stuck in a local minima/maxima.</p><h3>Learning Rate in Intuition: Smart Shoes for Treasure Hunting</h3><p>All right. Let’s think through an example. Please imagine you are a treasure hunter. You heard that an ancient artefact is hidden in a dungeon. You have two tools to navigate yourself to the spot of the treasure: inclinometer and <strong>smart shoes</strong>. Inclinometer inform you the slope of the every direction where you are at. Smart shoes allows you to adjust the distance you can take for a step. An unit of your action to get closer to the treasure is like this: <strong>1. with the inclinometer, determine the direction to head next to 2. with the help of smart shoes take a step in the adjusted distance.</strong> Each unit of action above takes same amount of time and there’s a time limit due to capacity of food/water you can carry. By calculation, you knew the number of the actions you can take is 10,000 times. <strong>Throughout attempts, only before entering the dungeon you are allowed to tune the smart shoes to make a step shorter/longer. </strong>Assume the entrance is the highest point of the dungeon.</p><p>Now you are ready to explore the dungeon. <strong>For this first attempt, you adjusted the smart shoes power so small </strong>like its just a normal step size we take as a human<strong> </strong>since you are more likely to be careful<strong>.</strong> Now you entered in, in each action, you measured the slope to find the direction to go and with the help of smart shoes take an accurate step. As you go, you record what step you are in. This time, at 100 times, you cant reach to the treasure yet. At 1,000.. not yet. And at 10,000, you knew you failed for this attempt. You could not find the treasure, meaning you could not reach the deepest spot of this dungeon, this time. <strong>May be the step size was too small to reach to the treasure.</strong></p><p>Now you knew in the first attempt, the step size was too small. <strong>For the second time, you decided to increase the step size much larger than before </strong>as it’s almost like you can jump for 1 km like a superman<strong>. </strong>As you did before, you check the step as you go. At 100 time, you start realizing that you have seen the similar space in this attempt. At 1,000 step, you are more confident that you already have had same locations before. Finally at 10,000, you knew failed again. <strong>You learned you may had passed the deepest spot due to too large step size and go back end force between the opposite place in the dungeon.</strong></p><p><strong>In third attempt, from the lessons you learned from the previous two attempts, you decided to medium step size to take. </strong>With the configuration of smart shoes, you can hop about 10 meters for each iteration. This time, you could find the treasure at step 5,000. You passed some times the deepest spot though, step size was short enough to get down to the spot where you can see the treasure with your eyes.</p><h3>Learning Rate in Actual Modeling</h3><p><strong>In the example, the strength of smart shoes adjustment is equivalent to learning rate we are discussing. </strong>Remember the original goal of neural nets modeling is to minimize the error rate between the model predictions and labels. The locations close enough for you to get the treasure in sight is equivalent to optimized set of weights/params to produce minimal error rate between model prediction and labels in the neural nets modeling. <strong>The learning rate is the hyperparameters that helps you to find the spots entailing minimal error rates more accurate and/or faster.</strong></p><p>The implementation is quite simple. When you update the weights of synapse in back propagation, you multiply it with the preset learning rate, alpha in this example. Please note that now the input was slightly modified in the inputs and label. Why? This is for making a situation that learning rate affects the model’s ability to converge, meaning finding the minimal error rate. Wrong value of learning rate may make the model impossible to converge. Let’s see it through outputs in this model.</p><p><strong>Regarding to the inputs and label change, just image the modification of the inputs and labels as a change of structure of the dungeon you explore in example</strong>. In the previous inputs/label set, the structure was like just a <strong>slide where the slope is constant in its direction and without having inclinometer you know where you should head</strong>. With that, as long as you keep taking steps, you’ll find the spot of treasure. In the new implementation, now the structure of the dungeon is like <strong>U or V curve where depending on where you are, the slope may change. In this case, if you take a step too far, you may pass the minimal spot.</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/99e917e37494c0df440ab0541b74b717/href">https://medium.com/media/99e917e37494c0df440ab0541b74b717/href</a></iframe><p>In this example, the learning rate is set</p><ul><li>Small Step Size: Alpha =0.01</li><li>Medium Step Size: Alpha = 10</li><li>Large Step Size: Alpha = 100</li></ul><p>By comparing results in different learning rate, we can understand better the concept of it.</p><h3>Take a Closer Look of the Relationship between Learning Rate and Error Rate</h3><h4><strong>First Question: Did it converge enough to find minima?</strong></h4><p>If the model converges, the error rate should be very small. Let’s see the results.</p><ul><li>Small Step Size ( Alpha = 0 .01 )→Error: <strong>0.499 : Not converged..</strong></li><li>Medium Step Size ( Alpha = 10 )→Error: <strong>0.004 : Converged!</strong></li><li>Large Step Size ( Alpha = 100 ) → Error: <strong>0.500 : Not converged..</strong></li></ul><p>Like I explained in the treasure hunt example, the medium step size allowed you to find the minimal spot of the dungeon and to get the treasure in your sight. Small and large step size size not just gave you worse performance of neural nets learning but even made it impossible to find the minima!</p><h4><strong>Second Question: How the too small / large step failed?</strong></h4><p>Now the focus of research is the failed small and large step size. What we want to look is the pattern of the error rate change in the learning process.</p><p><strong>Error Rate Trend: Small Step Size (Alpha =0.01) Training,</strong></p><pre>Training With Alpha:0.01<br>Error at 0 times: 0.503589968097<br>Error at 1000 times: 0.502388238396<br>...<br>Error at 9000 times: 0.499455244084<br>Error at 10000 times: 0.499255792633</pre><pre>Output After Training:<br>[[ 0.48995169]<br> [ 0.49333611]<br> [ 0.50980554]<br> [ 0.51021314]]</pre><p><strong>Observation</strong>: you may notice the error rate is slightly decreasing over training periods. However, since it suppresses the weights update too much, it could not get close enough to the location of treasure. It <strong><em>undershoots</em></strong>.</p><p><strong>Error Rate Trend: Medium Step Size (Alpha =10) Training,</strong></p><pre>Training With Alpha:10<br>Error at 0 times: 0.503589968097<br>Error at 1000 times: 0.252422788538<br>Error at 2000 times: 0.251274160635<br>...<br>Error at 9000 times: 0.00467243365523<br>Error at 10000 times: 0.00431791192321<br><br>Output After Training:<br>[[ 0.00478068]<br> [ 0.99873911]<br> [ 0.99387162]<br> [ 0.00510169]]</pre><p><strong>Observation</strong>: the error rate is decreasing effectively over the training periods. The output after training looks preferable for the label [0,1,1,0].</p><p><strong>Error Rate Trend: Large Step Size (Alpha =100) Training,</strong></p><pre>Training With Alpha:100<br>Error at 0 times: 0.503589968097<br>Error at 1000 times: 0.5<br>Error at 2000 times: 0.5<br>...<br>Error at 9000 times: 0.5<br>Error at 10000 times: 0.5</pre><pre>Output After Training:<br>[[ 1.]<br> [ 1.]<br> [ 1.]<br> [ 1.]]</pre><p><strong>Observation</strong>: the error rate is not changing over training periods. The output after training shows very skewed result, which may indicate <strong><em>overshoot</em></strong> in the first few training periods.</p><h4><strong>Third and Final Question: Which learning rate among these works best?</strong></h4><p>Yes, the medium step size (10) for this neural nets configuration. It gives you the smallest error rate.</p><h3>How can we fix under/overshooting?</h3><p>The potential benefit of smaller learning rate is, <strong>if given enough step numbers</strong>, it converges in much closer to the optimum, like not just you can get the treasure in sight, but even get as close as it’s within the reach of your arm. On the other hand, larger learning rate may benefits you if the time does so matter as long as <strong>it does not overshoot too much</strong>.</p><p>In the treasure hunt example, I set a constraint about the value configuration of the smart shoes, which you are only allowed to adjust the parameter before entering the dungeon. <strong>This constrain can be liberated; you can change the learning rate after starting training. </strong>Considering the advantages of larger/smaller learning rates, a strategy you might want to use is <strong>set a larger learning rate at the beginning and make it smaller over time</strong>. This is technically called <strong>Learning Decay</strong>. <strong>You can make your smart shoes even smarter (auto step size adjustment over time)!</strong></p><p>I am going to pass implementing the Learning Decay in this entry since this has already been lengthy. I may cover it as an extra entry as of request!</p><h3>You Extended the Neural Nets!</h3><p>Thank you for reading through this entry. Now you implemented the first extension of your neural nets. Learning rate helps you to make your neural nets more precise (smaller step size) and/or fast-learner (larger step size). You may find the optimal learning rate tuning even including the Learning Decay and strategies beyond that.<strong> What’s the best smart shoes? It depends on the dungeon you explore.</strong></p><p>Now what if the problems are getting complex like you have to explore multi dimensions of the universe? Yes, your neural nets can gain more capability by introducing other hyperparameters.</p><h4>Any question or thoughts? <a href="http://takishikawa.com/">Reach out to me:)</a></h4><p>I am always welcome to answer any questions or feedback on my entries. I am also the one just started learning neural nets/machine learning and know more and better about it through interactions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12b89f1d3c8e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Grow Neural Nets —Part 1: Intro to Hyperparameters]]></title>
            <link>https://medium.com/@taka2code/grow-neural-nets-part-1-intro-to-hyperparameters-39e59b212118?source=rss-d039534489e8------2</link>
            <guid isPermaLink="false">https://medium.com/p/39e59b212118</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[science]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[Taka Ishikawa]]></dc:creator>
            <pubDate>Sun, 19 Feb 2017 07:48:26 GMT</pubDate>
            <atom:updated>2017-02-19T12:25:14.102Z</atom:updated>
            <content:encoded><![CDATA[<h4>Start building your own neural nets and evolve it step by step to advanced machine learning models</h4><h3>Grow neural networks step by step</h3><p>In my previous post, <a href="https://medium.com/@tk2bit/a-lightning-path-to-understand-neural-networks-nns-1ba32d1b1785#.6mxyv3cyf">Learning Neural Nets</a>, I introduced how to start learning <strong>neural nets, one of the most important concepts in modern machine learning models.</strong> Once you start understanding how neural nets, you will wonder how advanced machine learning models such as Recurrent Neural Nets (RNNs), Convolutional Neural Nets (CNNs), or Generative Adversarial Networks (GANs).</p><p>Imagine you build your own advanced machine learning models from a minimal neural nets. Isn’t that cool? The purpose of this <strong>”Grow Neural Networks” </strong>series is to guide you through the journey of evolving the seed neural nets to more sophisticated models.</p><p>In this particular entry, I would like to introduce how to add <strong>hyperparameters</strong> commonly used in neural networks.</p><p>Get excited? Let’s start!!</p><h3>Hyperparameters vs Parameters</h3><p>Before diving into understanding hyperparameters in depth, it is necessary to distinguish hyperparameters from parameters. In short, <strong>hyperparameters are fixed before training to tune the training model itself; parameters are dynamically adjusted while training within the training model</strong>.</p><p><strong>In neural nets training, weights of synapses are parameters</strong>. Remember, while neural nets training, what we are updating is the weights of synapses. More generally, machine learning is the process of adjusting parameters given sample input and output so that the resulting model can predict an output from a new input.</p><p>What we are going to taking a closer look is hyperparameters; variables of a machine learning model that we tune <strong>before a training.</strong> Hyperparameters impact the efficiency of the training process and the accuracy of the resulting model. Understanding hyperparameters allows you to build more complex customised neural nets and unleash its further potential.</p><p>Now, what are hyperparameters and how they work then?</p><h3>First Look: Hyperparameters in Action</h3><p>To feel is an effective way to assimilate new concepts. Above all, before diving into the trench of machine learning theories, I would like you to play around with <a href="http://playground.tensorflow.org/#activation=tanh&amp;batchSize=10&amp;dataset=circle&amp;regDataset=reg-plane&amp;learningRate=0.03&amp;regularizationRate=0&amp;noise=0&amp;networkShape=4,2&amp;seed=0.50098&amp;showTestData=false&amp;discretize=false&amp;percTrainData=50&amp;x=true&amp;y=true&amp;xTimesY=false&amp;xSquared=false&amp;ySquared=false&amp;cosX=false&amp;sinX=false&amp;cosY=false&amp;sinY=false&amp;collectStats=false&amp;problem=classification&amp;initZero=false&amp;hideText=false">TenserFlow’s Neural Network Playground</a> with full of joy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*funYWGYQRhaZdyy96EAIoA.png" /><figcaption>Tensor Flow’s Playground</figcaption></figure><p>Yeah! I understand your excitement by just watching what’s happening in the process of neural nets learning with this beautiful GUI. By rightly tuning, it can solves complex problems (did you try solving the spiral?). Now, I would love you to take a closer look what parameters you can tune here. These all are fairly interesting and effective ideas regarding tuning neural nets. Your wisdom and creativity will come in play around with that. Let’s dive in!</p><h3>Table of Contents of Hyperparameters</h3><p>In neural nets model, by tuning hyperparameters, we will see significant difference for its performance and accuracy. To understand these deeply is the key to be neural nets wizard. It is fast-forward though, I am going to briefly introducing each hyperparameters we saw in the simulator. We are going to implement each of them so do not worry even if you do not understand at this point.</p><p>I separated the hyperparameters into three categories:</p><ol><li><strong>Neural Nets Structure: </strong>the most intuitive hyperparams we can observe in the model. Generally speaking, more layers and nodes get the model powerful but of course there’s drawback with over complexity.</li><li><strong>Numeric Operations: </strong>not obvious but definitive secret source of sophisticated neural nets. You would have plenty choice of hyperparams under this category. May be hard to grasp at the beginning, it worth to stick to learn this.</li><li><strong>Dataset Preconditioning: </strong>ground-truth of any machine learning processes is the data. Without preparing data well, the model cannot succeed.</li></ol><p><strong>Neural Nets Structures</strong></p><ul><li><strong>Choice/Tweak of Features</strong> : X1, X2, X1², X2², X1X2, sin(X1), sin(X2)…</li></ul><p>A <a href="https://en.wikipedia.org/wiki/Feature_(machine_learning)">feature</a> is <em>“an individual measurable property of a phenomenon being observed”</em>. We can assume X1 as a collection of features that contains different n individual features than X2. We can apply mathematical operations before feeding it in like the list of features X1², X2², X1X2, sin(X1), and sin(X2) to make the model more powerful. Normalize entire features? Reduce the number of features? Generate new features from the currently available features? Degree of freedom on this hyperparams is large and its impact is so.</p><ul><li><strong>Number of hidden layers</strong> : 0…</li></ul><p>A hidden layer is an intermediate set of neurons made by mapping weights and non-linear transformation (activation function) onto neurons in the previous layer. If your data is simple like you can separate them by drawing a straight line, you wont need any hidden layer. Beyond that, it is said one hidden layer is sufficient for the large majority of conventional problem. However, we need more of them when we apply it to more complex problems that we are going to explore.</p><ul><li><strong>Number of nodes in each hidden layer</strong> : 1…</li></ul><p>This means the number of nodes in a layer. In a layer, each nodes holds different intermediate computational values since each is mapped with different weights configuration. According to the author of “Introduction to Neural Networks for Java”, Jeff Heaton, ‘<em>the optimal size of the hidden layer is usually </em><strong><em>between the size of the input and size of the output layers</em></strong>’. This only can be applied for simpler and conventional model of neural nets though, it is good guidance to consider often times.</p><p><strong>Numeric Operations</strong></p><ul><li><strong>Learning Rate </strong>: 0.00001, 0.0001, 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10…</li></ul><p>Step size of adjustments of weights/parameters in each iteration. Though how it impacts is different between machine learning architectures, it should be optimized to find its best value to achieve faster training process and more accurate result. Higher value give faster learning but it may cause the model to fail to converge. Lower value gives higher chance to converge but may cause the model to learn too slow and even stuck in a local minima/maxima. Step size should be fixed? No, over time we may vary it. It is one of techniques.</p><ul><li><strong>Activation function </strong>: ReLU, Tanh, Sigmoid, Linear…</li></ul><p>Different activation function changes how we convert weighted values through synapses for a better training. Generally speaking, <strong>ReLU is a primary choice (avoid gradient vanishing, efficient without pre-training in deep neural nets).</strong> Leaky ReLU or Maxout comes second and Tanh third choice. Sigmoid and linear cannot be used if the output should be multi-class. Activation functions are really interesting topic when we microscope it actually.</p><ul><li><strong>Regularization </strong>: L1, L2…</li></ul><p>Regularization controls the capacity of neural nets to prevent overfitting, which a model gets too optimized for the training data set and loose generality to predict accurately for new data. Sampling data is constrained in real-world. There’s no perfect data set and often times it contains ill-posed variance. Regularization injects our preferences on weights into available data in order to mitigate the real-world sampling limits. Or just we suspect the model is too powerful, we can introduce regularization as a counterforce against it.</p><ul><li><strong>Regularization rate</strong> : 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10…</li></ul><p>Strengthen of regularization applied with the type of reguralization. Of course, if we set this too high, leaner will not be able to learn enough and its predictions be suffered.</p><ul><li><strong>Problem type</strong> : classification or regression</li></ul><p>In classification problem, the output is supposed to be discrete value like a label (0 or 1, red, green, or blue, cat or dog etc…). In regression problem, the output is supposed to be continuous value (-0.1234, -0.0012, 1.2345, 12345 etc…)</p><p><strong>Dataset Preconditioning</strong></p><ul><li><strong>Ratio of training to test </strong>: 10%, 20%, … , 90%</li></ul><p>This item does not directly related to the training process. In short, the test set is not used to train the model but to validate the generalization of it after training. As suggested in regularization, it is important to check whether the model is generalized enough for new inputs and the ratio determines how careful we become on the validation.</p><ul><li><strong>Noise</strong> : 0, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50…</li></ul><p>Often times, to prevent overfitting, noise is injected to a training process. This is a set of examples inserted in the original training data set and it makes it more difficult for neural nets to learn. The more noise added, the stronger the degree of overfitting prevention. However, if you make it too noisy then a training may fail to build sufficient model to accurately predict.</p><ul><li><strong>Batch size</strong> : 1, 2, 3, 4, 5, …. 10, … , 20, … , 30</li></ul><p>Batch size defines how many samples that going to be propagate through network. The smaller batch size is, the less memory is required. But if it is set too small, the estimate of gradients gets less accurate.</p><h3>Hyperparameters as a set of powerful tools</h3><p>Each hyperparameters are there for making the machine learning, in particular the above set of them; the neural nets learning, better. In practice, these hyperparameters are mutual dependent and by combining them produces much different results. We would not know the best set of tuning values without understanding mutual impacts. However, for the sake of learning, starting learning one hyperparmeter at a step will help you to become a wizard of neural nets faster.</p><p>From now one, I will introducing each of hyperparameters one by one. Eventually, we will realilze we are actually learning and implementing the more advanced model of neural nets like RNNs, CNNs, and GANs!</p><p>Stay tune!</p><h4>Any question or thoughts? <a href="http://takishikawa.com/">Reach out to me:)</a></h4><p>I am always welcome to answer any questions or feedback on my entries. I am also the one just started learning neural nets/machine learning and know more and better about it through interactions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39e59b212118" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Grow Neural Nets — Part 0: The Seed Neural Nets]]></title>
            <link>https://medium.com/@taka2code/grow-neural-nets-part-0-the-seed-neural-nets-bc9a14a5d029?source=rss-d039534489e8------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc9a14a5d029</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Taka Ishikawa]]></dc:creator>
            <pubDate>Thu, 19 Jan 2017 03:42:00 GMT</pubDate>
            <atom:updated>2017-01-19T03:42:00.108Z</atom:updated>
            <content:encoded><![CDATA[<h4>Develop your own neural nets and advance it step by step</h4><p>Imagine you build your own advanced machine learning models from a minimal neural nets. Isn’t that cool? The purpose of this <strong>”Grow Neural Networks” </strong>series is to guide you through the journey of evolving the seed neural nets to more sophisticated models.</p><h3>The Seed Neural Nets</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/37ca950f55090b719a3f0bc8d08e77c3/href">https://medium.com/media/37ca950f55090b719a3f0bc8d08e77c3/href</a></iframe><p>It covers fundamentals of neural nets:</p><ul><li>Binary class input and output</li><li>Activation function: Sigmoid</li><li>Forward propagation</li><li>Back propagation</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc9a14a5d029" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning Neural Networks]]></title>
            <link>https://medium.com/@taka2code/a-lightning-path-to-understand-neural-networks-nns-1ba32d1b1785?source=rss-d039534489e8------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ba32d1b1785</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[Taka Ishikawa]]></dc:creator>
            <pubDate>Sun, 15 Jan 2017 22:29:10 GMT</pubDate>
            <atom:updated>2017-01-25T22:50:57.806Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learning materials and methods to understand how neural networks works for machine learning beginners.</h4><h3>Why understanding Neural Nets matters:</h3><p>It is obvious that Artificial Intelligence is a key technology in 21st century. Adoption of this technology is enormously undergoing in any industry and it is unstoppable because of its potential. However, the question here is “what exactly Artificial Intelligence?” Actually, what is happening behind it is advancement of a particular field of Artificial Intelligence, machine learning. In the most of cases, when we talk about AI technically, we talk about machine learning. <strong>Understanding machine learning allows you to build modern AI applications and beyond.</strong></p><p>From what I have studied about machine learning, <strong>if I have to pick the most important concept to learn in this field, <em>Neural Networks</em> is my answer.</strong> The reason of this is because most of architectures of recently discovered and applied machine learning methods uses this concept. Without understanding what is neural networks and how it works, we cannot understand more complex machine learning models such as Recurrent Neural Networks (RNNs) —Auto Image Captioning, Convolutional Neural Networks (CNNs) — Complex-Game AI (AlphaGo), or Generative Adversarial Networks (GANs) — Auto Image Generation.</p><h3>Let’s Learn Neural Nets :</h3><p>What’s is neural networks then? Throughout the history of artificial intelligence research, one of the major goal to achieve this field has been designing and implementing a system that has as smart as or even smarter than human intelligence. In the course of this, it is not hard to imagine that researches get a hint out of our brain, which we believe where our intelligence come from.</p><p>Neural nets was one of the inventions in this process. Imitating how brain works when they learn, neural nets give machines ability to do so. Grasping the fundamental concepts is not that complicated and it’s actually fun. Let’s start.</p><h4><strong>0. Machine Learning 101 :</strong></h4><p>Before going further to learn neural nets, I would like to review fundamental concept of machine learning.</p><ul><li>Machine learning in one sentence: “design and implement algorithms that allow a machine to <strong>recognise patterns in data</strong> and <strong>adjust its program based on given preference.</strong>”</li><li><strong>Any data in machine learning process is represented in numeric values</strong> so that machines can handle it. An image can be represented in a collection of pixel values, audio can be represented in a collection of temporal sound wave values, etc..</li><li>Fundamental process of machine learning: <strong>1. Train, 2. Test, 3. Apply</strong></li><li><strong>1. Train</strong>: given training data, find patterns (model)</li><li><strong>2. Test</strong>: given test data, evaluate the model</li><li><strong>3. Apply</strong>: apply the model to new data (annotate an image, predict weather tomorrow, etc.. )</li><li><strong>Training</strong> is the phase that the most of machine learning designer’s creativity and effort put.</li><li>There’s three types of learning: <strong>1. Supervised Learning, 2. Unsupervised Learning , 3. Reinforcement Learning</strong></li><li><strong>1. Supervised Learning</strong>: given a set of problems and answer, predict an answer from new problem</li><li><strong>2. Unsupervised Learning</strong>: given a set of data, describe a hidden pattern</li><li><strong>3. Reinforcement Learning</strong>: given a set of environment and preference on states, find the optimal policy of actions</li><li><strong>Neural Networks is so generic computational approach</strong> that it can be flexibly used for many types of machine learning and therefore more complex real-world applications.</li></ul><p>If you want to explore more about machine learning itself in general especially from application perspective, I highly recommend to read through the <a href="https://medium.com/@ageitgey/machine-learning-is-fun-80ea3ec3c471#.wwakioflw"><em>“Machine Learning is Fun!”</em></a> series by <a href="https://medium.com/@ageitgey?source=user_popover">Adam Geitgey</a>:</p><h4>1. Comprehensive Intro to Neural Networks — <em>“</em><a href="https://www.youtube.com/watch?v=4PjBAO7Uy_Y"><em>Writing Neural Networks from Scratch</em></a><em>” by Eric Schles [Youtube Video]</em></h4><p>If you have zero knowledge on machine learning, you may feel getting lost. I you are feeling so in the first 5 minutes of this video, I highly recommend to go over the <a href="https://www.youtube.com/watch?v=bxe2T-V8XRs">“Neural Networks Demystified” series</a> by Welch Labs first and then come back again. This is not exact note used in this presentation but I found he shared <a href="https://github.com/EricSchles/intro_to_ml/blob/master/lectures/intro_to_neural_networks.md">a note on NNs</a>.</p><p>For an aid, this is a list of key ideas and its minimum explanations with a link to a related resource included in this section:</p><ul><li><strong>Neural Network is a mathematical approach to learn complex decision boundaries.</strong></li><li><a href="http://www.ece.utep.edu/research/webfuzzy/docs/kk-thesis/kk-thesis-html/node19.html"><strong>Linear separability</strong></a>: if drawing a straight line in the plane could separate data points, the space is linearly separable. With complex data sets, the space tends to be non-linearly separable. Operations in Neural Networks turn non-linearly separable space into linearly separable space to separate data points.</li><li><a href="https://en.wikipedia.org/wiki/Activation_function"><strong>Activation function</strong></a>: = non-linear transformation to apply to your data set in some of layers in the neural networks.</li><li><a href="https://en.wikipedia.org/wiki/Sigmoid_function"><strong>Sigmoid function</strong></a>: one of the activation functions commonly used in neural networks. It can be used for binary class (a value of output is 0 or 1). If you want to handle multi-class ( more than two classes like a value of output is 0, 1 , 2, or 3 ), you should incorporate other activation function such as Softmax function in neural nets.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/313/1*ZYtW_dqiAuqQJwGQMapJ8w.png" /><figcaption>Typical Sigmoid Function</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/127/1*clWpnBYi1Mn8co0WIZ517g.png" /></figure><ul><li><a href="https://www.youtube.com/watch?v=UJwK6jAStmg"><strong>Forward Propagation</strong></a>: apply a set of weights to the input data and calculate an output. The set of weights is selected randomly.</li><li><a href="https://www.youtube.com/watch?v=GlcnxUlrtek"><strong>Back Propagation</strong></a>: measures the margin of error of the output and adjust the weights accordingly to decrease the error.</li><li><a href="https://www.youtube.com/watch?v=5u0jaA3qAGk"><strong>Steepest Gradient Descent</strong></a>: one of the optimization techniques, whose purpose is to find a maximum or minimum value of a function. It uses the gradient, generalized concept of derivative for multiple variables, that has the highest rate of change as a guide to approach the solution in each iteration.</li></ul><p>At the end of video, there are some recommended learning materials on neural nets introduced and I inspected each to learn further.</p><h4>2. Designing and Implementing a Neural Networks — <a href="http://iamtrask.github.io/2015/07/12/basic-python-network/">“A Neural Network in 11 lines of Python (Part 1)</a>” by I am Trask</h4><p>This is the part of learn by doing. I recommend you to implement code on jupyter notebook with any insight you gain over the instructions in this entry. I suggest playing around in a virtualenv virtual environment which is a full isolated pythonic environment. If you need an assistant, I have a different entry to prepare <a href="https://medium.com/@tk2bit/jupyter-notebook-virtualenv-83764c93ea2#.wev2k8vs2">virtualenv + jupyter notebook environment.</a></p><p>This is an example of learning the material with jupyter notebook:</p><p>2 Layers:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b8d9ac47891fcef38fc94e51b63d0c8c/href">https://medium.com/media/b8d9ac47891fcef38fc94e51b63d0c8c/href</a></iframe><p>3 Layers:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/739921ef72a7401347f7f315a9ffbd48/href">https://medium.com/media/739921ef72a7401347f7f315a9ffbd48/href</a></iframe><p>I highly recommend to complete the next part “<a href="http://iamtrask.github.io/2015/07/27/python-network-part2/">A Neural Network in 13 lines of Python (Part 2 — Gradient Descent)</a>” too. This includes how to improve the neural nets with optimization technique, gradient descent introduced in the first part of material.</p><h4>3. Mini Project: Designing and Implementing Own Neural Networks from Scratch</h4><p>At this point, you have enough knowledge to code your own neural nets from scratch. These are some ideas about more advanced neural nets that you might build:</p><ul><li><strong>with</strong> <strong>deeper layers</strong> (more than three layers).</li><li><strong>in</strong> <strong>different languages</strong>, especially ones with more efficiency like C/C++.</li><li><strong>with different kinds of activations functions</strong>.</li><li><strong>with more advanced optimization techniques (regularization, dropout, etc..).</strong></li></ul><p>I initiated this stage from implementing base neural nets implementation from scratch to review previous exercises and use it as seed to add more features upon:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/37ca950f55090b719a3f0bc8d08e77c3/href">https://medium.com/media/37ca950f55090b719a3f0bc8d08e77c3/href</a></iframe><p>In the last sections of <a href="http://iamtrask.github.io/2015/07/12/basic-python-network/">“A Neural Network in 11 lines of Python (Part 1) by I am Trask”</a> lists up great suggestions for the further exploration of neural nets learning.</p><h3>Conclusion</h3><p>As you may notice, neural nets itself is not so complex to understand and implement. The complexity comes when we build deeper neural nets with many advanced techniques. Needless to say, to understand more complex models of neural nets, mastering how neural nets works is the most effective thing you can do. I hope this entry helps you to learn neural nets and take a dive into further exciting machine learning journey.</p><p>Further references:</p><p><strong>Linear Algebra:</strong></p><ul><li><a href="https://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-spring-2010/">Linear Algebra at MIT Opencourse</a></li></ul><p><strong>Deep Learning in general:</strong></p><ul><li><a href="http://www.wildml.com/">WildML</a> — AI, Deep Learning, NLP</li></ul><p><strong>Recurrent Neural Networks (RNNs):</strong></p><ul><li><a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/">The Unreasonable Effectiveness of Recurrent Neural Networks</a></li></ul><p><strong>Convolutional Neural Networks (CNNs):</strong></p><ul><li><a href="https://www.tastehit.com/blog/google-deepmind-alphago-how-it-works/">Google DeepMind’s AlphaGo: How it works</a></li></ul><p><strong>Generative Adversarial Networks (GANs):</strong></p><ul><li><a href="https://arxiv.org/abs/1701.00160">NIPS 2016 Tutorial: Generative Adversarial Networks [Paper]</a></li><li><a href="https://arxiv.org/abs/1406.2661">Generative Adversarial Networks [Paper]</a></li><li><a href="https://openai.com/blog/generative-models/">Generative Models</a> by OpenAI</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ba32d1b1785" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>