<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Henry Han on Medium]]></title>
        <description><![CDATA[Stories by Henry Han on Medium]]></description>
        <link>https://medium.com/@hanzhongliu1990?source=rss-f80de040b8c1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ilA9lQLYW0gHSdiu5QE0dw@2x.jpeg</url>
            <title>Stories by Henry Han on Medium</title>
            <link>https://medium.com/@hanzhongliu1990?source=rss-f80de040b8c1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 13:43:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@hanzhongliu1990/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Learn Python]]></title>
            <link>https://medium.com/@hanzhongliu1990/%E5%AD%A6%E4%B9%A0python-7f72841b52b5?source=rss-f80de040b8c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/7f72841b52b5</guid>
            <dc:creator><![CDATA[Henry Han]]></dc:creator>
            <pubDate>Sat, 17 Nov 2018 18:29:09 GMT</pubDate>
            <atom:updated>2018-12-08T18:08:38.308Z</atom:updated>
            <content:encoded><![CDATA[<h4>started learning on Nov 12, 2018, materials sourced from imooc</h4><h4>Python适合的领域</h4><ol><li>Web网站和各种网络服务</li><li>系统工具和脚本</li><li>作为胶水语言把其他语言开发的模块包装起来方便实用</li></ol><h4>Python不适合的领域</h4><ol><li>贴近硬件的代码 比如C</li><li>移动开发</li><li>游戏开发 C/C++</li></ol><p>Python语言的特点</p><ol><li>类型：解释执行</li><li>运行速度： 慢</li><li>代码量：少</li></ol><p>安装Python时勿忘添加系统环境变量（Python.exe文件所在目录）</p><h4>第一个程序</h4><p>运行时 切换到程序所在目录</p><p>Python hello.py</p><h4>Python中的数据类型</h4><ol><li>整数</li><li>浮点数</li><li>字符串</li><li>布尔值</li><li>空值</li></ol><h4>print语句</h4><p>逗号可以连接多个字符串，例如：</p><p>Print ‘The quick brown fox’, ‘jumps over’, ‘the lazy dog’</p><h4>注释</h4><p>#</p><h4>变量</h4><p>变量名必须是大小写英文、数字和下划线的组合，且不能用数字开头</p><p>X1=1, d=3, n=100</p><p>X100 = x1+(n-1)*d</p><p>S = (x1+x100)*n/2</p><h4>定义字符串</h4><p>‘Learn “Python” in imooc’</p><p>‘Bob said \“I\’m OK\”.’</p><p>\n 表示换行</p><p>\t 表示制表符</p><p>\\ 表示\字符本身</p><h4>raw字符串与多行字符串</h4><p>对整段进行转义，不需要\</p><p>Print r’’’ …………………’’’</p><p>换行不要\n 直接下一行</p><h4>Unicode字符串</h4><p>Print u’中文’</p><p>Print ur’’’……………’’’</p><p>或者直接在第一行添加注释 就不需要u了</p><p># -*- coding: utf-8 -*-</p><h4>整数和浮点数</h4><p>同Java</p><h4>布尔类型</h4><h4>创建list</h4><p>L = [‘Adam’,‘Lisa’,’Bart’]</p><p>Print L</p><h4>按照索引访问list</h4><p>Print L[0]</p><p>Print L[1]</p><p>Print L[2]</p><h4>倒序访问list</h4><p>想要输出Bart</p><p>Print L[-1]</p><h4>添加新元素</h4><ol><li>L.append(‘Paul’)</li><li>L.insert(0,‘Paul’)</li></ol><h4>从list删除元素</h4><p>L.pop #删除最后的元素</p><p>L.pop(2) #删除index为2的元素</p><h4>创建tuple</h4><p>Tuple是一种有序的列表，中文翻译成”元组”。Tuple和List非常类似，但是tuple一旦创建完毕，就不能修改。</p><p>T = (‘Adam’, ‘Lisa’, ‘Bart’)</p><p>T不能改变，tuple没有append()或insert()或pop()方法。可以访问，但无法修改。</p><h4>创建单元素tuple</h4><p>包含0个元素的tuple</p><p>T=()</p><p>包含1个元素的tuple</p><p>T=(1)会输出整数的1而不是tuple的(1)</p><p>所以要多加一个逗号 T=(1,)避免歧义</p><p>多元素tuple不必加逗号</p><h4>“可变”的tuple</h4><p>T = (‘a’, ‘b’, [‘A’, ‘B’])</p><p>L = T(2)</p><p>修改</p><p>L[0] = ‘X’</p><p>L[1] = ‘Y’</p><p>Print T</p><p>输出 (‘a’, ‘b’, [‘X’, ‘Y’])</p><blockquote>总结: []可变 ()不可变</blockquote><h4>If语句</h4><p>Age = 20</p><p>If age&gt;=18:</p><p>Print ‘your age is ’, age</p><p>Print ‘adult’</p><p>Print ‘END’</p><blockquote>Python的缩进规则为四个空格，不能使用Tab， 退出缩进需要多敲一行回车</blockquote><h4>if-else</h4><p>if not age &gt;=18:</p><p>print ‘teenager’</p><p>if age &gt;= 18:</p><p>print ‘adult’</p><p>else:</p><p>print ‘teenager’</p><h4>If-elif-else</h4><p>不要忘记：</p><h4>For循环</h4><p>L=[‘Adam’, ‘Lisa’, ‘Bart’]</p><p>for name in L:</p><p>print name</p><h4>While循环</h4><p>N=10</p><p>X=0</p><p>While X&lt;N:</p><p>Print X</p><p>X= X+1</p><h4>Break退出循环</h4><p>计算1到100的整数和</p><p>sum=0</p><p>X=1</p><p>While True:</p><p>Sum = sum + x</p><p>X = x+1</p><p>If x&gt;100:</p><p>Break</p><p>Print sum</p><h4>Continue继续循环</h4><p>利用continue跳过后续循环代码，继续下一次循环。</p><h4>多重循环</h4><p>同Java</p><h4>什么是dict</h4><p>名字和分数关联起来，给定一个名字，就可以直接查到分数。</p><p>d={</p><p>‘Adam’:95,</p><p>‘Lisa’:85,</p><p>‘Bart’:59</p><p>}</p><p>Key:value</p><p>Len(d)表示dict的集合大小 输出3</p><h4>访问dict</h4><p>D[key] to get the value</p><p>E.g. Print d[‘Adam’]</p><ol><li>判断是否存在</li></ol><p>If ‘Paul’ in d:</p><p>Print d[‘Paul’]</p><p>2. 当key不存在时，返回none</p><p>Print d.get(‘Paul’)</p><h4>dict的特点</h4><ol><li>查找速度快，但占用内存大，还有会浪费很多内容。list占用内存小，但查找速度慢。</li><li>key-value序对是没有顺序的。</li><li>作为key的元素必须不可变。</li></ol><p>字符串、整数、浮点数都是不可变的都可以作为key，list可变所以不能作为key。</p><h4>更新dict</h4><p>dict是可变的</p><p>添加新的key-value</p><p>D[‘Paul’] = 72</p><h4>遍历dict</h4><p>for key in d</p><p>Print key</p><h4>Set</h4><p>Dict的作用是建立一组key和一组valve的映射关系，dict的key是不能重复的。如果只想要dict的key，不关心对应的valve时，使用set。</p><p>set持有一系列元素，这一点和list很像，但set的元素没有重复，而且是无序的，这点和dict的key很像。</p><p>创建set的方式是调用set()并传入一个list，list的元素将作为set的元素。</p><p>s=set([‘A’,’B’,’C’])</p><p>Print s 输出</p><p>Set([‘A’,’C’,’B’])</p><p>Len(s)</p><p>如果set传入包含重复元素的list，set将会自动去掉重复的元素。</p><h4>访问set</h4><p>‘Bart’ in s 输出</p><p>True</p><p>注意大小写！</p><h4>set的特点</h4><p>set的内部结构和dict很像，唯一区别的是不存储valve，因此判断一个元素是否在set中速度很快。</p><p>set存储的元素和dict的key类似，必须是不变对象，因此任何和变对象是不能放入set中的。</p><p>set存储的元素也是没有顺序的。</p><p>事先创建好一个set，然后判断用户输入是否在set里面。</p><h4>遍历set</h4><p>for name in s:</p><p>Print name</p><blockquote>总结：list[] tuple() dict{} set([])</blockquote><h4>更新set</h4><p>s.add(4)</p><p>s.remove(4)</p><h4>编写函数</h4><p>def my_abs(x):</p><p>If x&gt;=0:</p><p>Return x</p><p>Else</p><p>Return -x</p><h4>函数之返回多值</h4><p>函数返回多值其实就是返回一个tuple，但在语法上返回一个tuple可以省略括号。</p><p>import math</p><p>def quadratic_equation(a, b, c):<br> delta = b*b — 4.0*a*c<br> if delta &gt;=0:<br> t = math.sqrt(delta)<br> return (-b+t)/(2.0*a),(-b-t)/(2.0*a)<br> else:<br> return</p><p>print quadratic_equation(2, 3, 0)<br>print quadratic_equation(1, -6, 5)</p><h4>递归函数</h4><pre>fact(n) = n! = 1 * 2 * 3 * ... * (n-1) * n = (n-1)! * n = fact(n-1) * n</pre><pre>def fact(n):<br>    if n==1:<br>        return 1<br>    return n * fact(n - 1)</pre><p>使用递归函数需要注意防止栈溢出。</p><p><strong>汉诺塔问题</strong></p><p>def move(n, a, b, c):<br> if n == 1:<br> print(a + “ →” + c)<br> else:<br> move(n-1,a,c,b)<br> move(1,a,b,c)<br> move(n-1,b,a,c)</p><p>move(4, ‘A’, ‘B’, ‘C’)</p><h4>默认参数</h4><p>函数的默认参数的作用是简化作用</p><p>默认参数只能定义在必需参数的后面</p><p>e.g.</p><p>#OK</p><p>def fn1(a,b=1,c=2):</p><p>Pass</p><p>#Error</p><p>Def fn2(a=1,b)</p><p>Pass</p><h4>可变参数</h4><p>def fn(*args):</p><p>print args</p><p>args前有*号，可以传入0、1或多个参数给可变参数</p><p>可直接把args看成一个tuple就好了</p><p>def average(*args):<br> sum = 0.0<br> if len(args) == 0:<br> return sum<br> else:<br> for x in args:<br> sum = sum + x<br> return sum/len(args)</p><p>print average()<br>print average(1, 2)<br>print average(1, 2, 2, 3, 4)</p><h4>对list进行切片</h4><p>L[起始位置:结束位置:步长]</p><p>把list换成tuple，切片操作完全相同，只是切片的结果也变成了tuple。</p><p>L = range(1, 101)</p><p>print L[0:10] #前十个数<br>print L[2::3] #3的倍数 相当于从index2开始每次走3步<br>print L[4:51:5] #不大于50的5的倍数 相当于从index4开始index51结束 每次走5步</p><h4>倒序切片</h4><p>L = range(1, 101)<br>print L[-10:] #最后10个数<br>print L[4::5][-10:] #最后10个5的倍数 相当于从index4开始 每次走5步 然后取最后10个元素</p><h4>对字符串切片</h4><p>&gt;&gt;&gt; ‘ABCDEFG’[:3]</p><p>‘ABC’</p><p>&gt;&gt;&gt; ‘ABCDEFG’[-3:]</p><p>EFG</p><p>&gt;&gt;&gt; ‘ABCDEFG’[::2]</p><p>ACEG</p><p>把字符变成大些字母</p><p>&gt;&gt;&gt; ‘abc’.upper()</p><p>‘ABC’</p><p>Def firstCharUpper(s):</p><p>Return s[:1].upper() + s[1::]</p><h4>迭代</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f72841b52b5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Joomla!]]></title>
            <link>https://medium.com/@hanzhongliu1990/starting-learning-joomla-on-october-30-2018-677f72a169d2?source=rss-f80de040b8c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/677f72a169d2</guid>
            <category><![CDATA[joomla]]></category>
            <dc:creator><![CDATA[Henry Han]]></dc:creator>
            <pubDate>Wed, 31 Oct 2018 06:23:41 GMT</pubDate>
            <atom:updated>2018-10-31T06:24:46.685Z</atom:updated>
            <content:encoded><![CDATA[<p>Joomla!</p><p>Starting learning Joomla on October 30, 2018</p><h3>Introduction to the Joomla beginner class</h3><p>Workflow</p><ol><li>Plan</li><li>Install</li><li>Content</li><li>Extend</li><li>Design</li><li>People</li><li>Manage</li></ol><h3>What is Joomla?</h3><p>A content management system (CMS)</p><p>Joomla’ s goal is to make it easy to manage your content online.</p><p>What is a CMS?</p><p>Content – text, editing, links</p><p>Template – site design and page layout, information architecture, usability design</p><p>Framework/CMS – database, PHP, Ajax, JavaScript, etc.</p><p>When?</p><p>“Mambo” in 2000, changed to “Joomla” in 2005</p><h3>How to plan your Joomla site?</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=677f72a169d2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Big Data — Hadoop]]></title>
            <link>https://medium.com/@hanzhongliu1990/big-data-hadoop-ed3e53c6406c?source=rss-f80de040b8c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed3e53c6406c</guid>
            <dc:creator><![CDATA[Henry Han]]></dc:creator>
            <pubDate>Tue, 13 Mar 2018 23:04:49 GMT</pubDate>
            <atom:updated>2018-03-30T17:50:50.653Z</atom:updated>
            <content:encoded><![CDATA[<p>Introducing Apache Hadoop</p><h4>What is Apache Hadoop?</h4><blockquote>The Apache™ Hadoop® project develops open-source software for reliable, scalable, distributed computing.</blockquote><blockquote>The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.</blockquote><h4>Download Hadoop</h4><p>go the following website, and select the latest version and binary file.</p><p><a href="http://hadoop.apache.org/releases.html">Apache Hadoop</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_t8z7S7Tp4GhdZKevmdlCw.png" /></figure><h4>Hadoop installation and environment setup</h4><p><strong>Environment</strong></p><p>I used Ubuntu 16–64 bit as the system environment.</p><p><strong>Create a Hadoop user</strong></p><p>If you’re not using “hadoop” as the user, you need to create a user named “hadoop”.</p><p>Open a new terminal, then type in the following command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*jRWal98BtemtO2cbD9nnlw.png" /><figcaption>add a new user “hadoop”</figcaption></figure><p>Type in the password for the current logged-in user when password request prompts.</p><blockquote>sudo command</blockquote><blockquote>sudo is a program for Unix-like computer operating system that allows users to run programs with the security privileges of another user, by default the superuser.</blockquote><p>Use the following command to set up the password for “hadoop” user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/452/1*Wshs7kdy6BCZbNcPJEr8gQ.png" /><figcaption>set up password for “hadoop”</figcaption></figure><p>Give “hadoop” admin privilege to solve unpredictable problems for the beginners.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*W8qOAPdGgNf7MFLZ9ANizA.png" /><figcaption>give “hadoop” admin privilege</figcaption></figure><p><strong>Update apt</strong></p><p>Log in as “hadoop” first</p><p>Open up a new terminal, type in the following command to update apt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*L1hQRRfpXlcdG15ee2NG_A.png" /><figcaption>update apt</figcaption></figure><blockquote>We’re using apt to install apps. Some application cannot be installed without updating apt.</blockquote><p>You can also install “vim”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*IITMrC-abjk_fehfEe227w.png" /><figcaption>install vim</figcaption></figure><p>Type “y” when being asked if you want to continue.</p><p><strong>Install and configure SSH to login without password</strong></p><blockquote>Secure Shell <strong>(</strong>SSH<strong>) </strong>is a cryptographic network protocol for operating network services securely over an unsecured network.</blockquote><p>Ubuntu has already installed SSH client by default, we also need to install SSH server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*a76GzjVFkdob2nXStj_rFQ.png" /><figcaption>install SSH server</figcaption></figure><p>Type “y” when being asked if you want to continue.</p><p>After installation, type in the following the command and see if you can login to the local machine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/467/1*_XBIx91jYzJDvJasWOHKkA.png" /></figure><p>Type “yes” password for “hadoop”</p><p>We need to type in password every time we log in, then how to configure to log in without typing in the password?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/381/1*eUPkR4LzJAbbl_63mTFBSg.png" /><figcaption>exit the connection</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*aDdihVLcD18fG1wXVx_erw.png" /><figcaption>go to /.ssh/ folder</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*fpRh-xkhfJYmx1vh5xljOw.png" /><figcaption>generate public/private rsa key pair</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/1*pi4Xs0L2MEAlQBdixWIf6w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/1*5DoE2a4-2Ia9O-FuHYOVZQ.png" /><figcaption>login without password</figcaption></figure><p><strong>Configure Java environment</strong></p><p>Issue the following command to install Java.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*Y4iY8ruLfGmvqh-SZGAKDw.png" /><figcaption>install Java</figcaption></figure><p>Type “y” when being asked if you want to continue.</p><p>Set the value of variable JAVA_HOME, type in the following command to edit bashrc file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/409/1*JJ1BMA3shoSVHP0Ww9eKKQ.png" /><figcaption>edit bashrc file</figcaption></figure><p>Add “export JAVA_HOME=/usr/lib/jvm/default-java” to the first line.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/506/1*G8fkWCl-_5d2zXt55_Y-Pg.png" /><figcaption>add a line</figcaption></figure><p>Run the following command to update the file immediately.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/1*_dRZWI-urpQ5d-MY0hRQ9A.png" /><figcaption>update bashrc file</figcaption></figure><p>To verify if Java has been successfully installed, type in the following command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/1*8Gx6mFsdrdt5R6Za_HP6EQ.png" /><figcaption>verify $JAVA_HOME</figcaption></figure><p><strong>Install Hadoop 3</strong></p><p>Make sure the hadoop installation file is in the Downloads folder</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/1*MfhyWGY0fv8rl9mbADQYhw.png" /></figure><p>Extract installation file to /usr/local folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*vjMY89cjrWtqqcpKqni_ng.png" /></figure><p>Go to /usr/local/ folder</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/514/1*Q29mpLUchYJbN0UPgDhOtw.png" /></figure><p>Change the name of the folder to hadoop</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/1*i1lwcIJnSQyPGOAS5prCjQ.png" /></figure><p>Change the folder to be read-only.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*XP-qZ6pozBKQoVHwHVpHlw.png" /></figure><p>Make sure you are in the /usr/local/hadoop folder, type in the following command to verify if Hadoop has been successfully installed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/1*2ZYi6UFlpB_w-accnl1QyA.png" /></figure><p>If Hadoop has been successfully installed, the following message should be displayed on the screen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/733/1*FPI2RpFl9gIsWGSncprDEQ.png" /></figure><p><strong>Hadoop Standalone mode (non-distributed mode)</strong></p><p>Create a folder called input inside /usr/local/hadoop folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*wGwVYNXaWQX2ZCpQGbGbcA.png" /></figure><p>Copy all xml files from /etc/hadoop folder into input folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/1*VyFIE6nMzUiq9p4R4mEltA.png" /></figure><p>Run the built-in program — hadoop-mapreduce-examples-3.0.0.jar</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*sgaGF4o3K8lrTukjgXlSRw.png" /></figure><p>To see the output,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*5rz3TJI-JK78Xwiow19YiA.png" /></figure><p><strong>Hadoop pseudo-distributed mode</strong></p><p>The configuration files of Hadoop are located in the /usr/local/hadoop/etc/hadoop/, you need to edit 2 configuration files — core-site.xml and hdfs-site.xml to switch to pseudo-distributed mode.</p><p>Run the following command to edit core-site.xml file</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*j_d5rp6v5A3-qbt-WYxvkg.png" /></figure><p>And change the configuration into the following settings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*WNxHLJ-B7Db2lfz1x3T8sw.png" /></figure><p>To edit hdfs-site.xml file</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/719/1*7g6Wxz0PiBfZ0z8oRhZhjw.png" /></figure><p>And change the configuration into the following settings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/1*tspl1bf352asxo47fLm2AQ.png" /></figure><p>Format NameNode</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*P_wnhMe3xFYdM53mXu8c5A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*oMD74l-0WD3J1LMCzE1hVg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*BXf8wKHnLna4G8jKKI2WBQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/455/1*on6KxwwMJVU2jI-tnV7iAw.png" /></figure><p>Open a web browser, and go to <a href="http://localhost:9870/">http://localhost:9870</a> to view the information of NameNode and DataNoe, as well as the online files in HDFS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*j9xwV_v-bRvFDzhJ65HxXw.png" /></figure><p>Follow the steps to run a Hadoop pseudo-distributed mode instance.</p><p>Create a folder on HDFS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*Ju9A4G1XnGjQmCZ3rGuo6A.png" /></figure><p>Create a folder called input.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*p6bbMGBQXKzPxbKLGnnlHw.png" /></figure><p>Copy all xml files from /etc/hadoop folder into input folder on HDFS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*olWqZoi5jZiwFD2NWmpHRg.png" /></figure><p>To view the file list in the input folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*bZcOoym62_QH-ZZRr5XwnQ.png" /></figure><p>Run the built-in program — hadoop-mapreduce-examples-3.0.0.jar</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/1*joISwjStQMrnJlcrpG5P-g.png" /></figure><p>To see the output</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*W2MKXpnnWlgX5g2zt2iONQ.png" /></figure><p>You can also take the output to local</p><p>Remove the local output folder first</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*X0Rzw0qX9q7QmrD9jhU4Rg.png" /></figure><p>Take the output from HDFS to local</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*N5VZDHU1270-VOcb_k4mpA.png" /></figure><p>To see the output on the local</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*Kmw81SjbPmyVhD2c3DgfOA.png" /></figure><p>To remove the output folder on HDFS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*e4EG_JSiNZCLVw3t5FeuuA.png" /></figure><p>To stop Hadoop</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/1*MoqDV4pu3Rh2_pcIl9hKxQ.png" /></figure><p>To start Hadoop</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*B_Jrf4YsD26hDUR2p9SvVA.png" /></figure><h4><strong>Introduction of MapReduce</strong></h4><p><strong>What is MapReduce?</strong></p><blockquote>Hadoop MapReduce is a software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) in-parallel on large clusters (thousands of nodes) of commodity hardware in a reliable, fault-tolerant manner.</blockquote><p><strong>Algorithm</strong></p><blockquote>A MapReduce <em>job</em> usually splits the input data-set into independent chunks which are processed by the <em>map tasks</em> in a completely parallel manner. The framework sorts the outputs of the maps, which are then input to the <em>reduce tasks</em>. Typically both the input and the output of the job are stored in a file-system. The framework takes care of scheduling tasks, monitoring them and re-executes the failed tasks.</blockquote><p>MapReduce paradigm is based on sending the computer to where the data resides.</p><p>MapReduce program executes in 3 stages, map stage, shuffle stage and reduce stage.</p><p>Map stage: the map or mapper’s job is to process the input data. The input data is in the form of file or directory and is stored in the HDFS. The input file is passed to the mapper function line by line. The mapper processes the data and creates several small chunks of data.</p><p>Reduce stage: is the combination of shuffle and reduce. The reducer’s job is to process the data that comes from the mapper. It produces a new set of output, which will be stored in the HDFS.</p><p><strong>Terminology</strong></p><ol><li>NameNode — Node that manages the Hadoop Distributed File system (HDFS).</li><li>DataNoe — Node where JobTracker runs and which accepts job requests from clients</li><li>JobTracker — Schedules jobs and tracks the assign jobs to Task tracker.</li><li>Task Tracker — Tracks the task and report status to JobTracker.</li><li>Job — A program is an execution of a Mapper and Reducer across a dataset.</li><li>Task — An execution of a Mapper or a Reducer on a slice of data.</li><li>Mapper — Maps the input key/value pairs to a set of intermediate key/value pair.</li></ol><p>Please refer to <a href="https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html">https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html</a> for more details and examples about Hadoop MapReduce.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed3e53c6406c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Big Data Applications]]></title>
            <link>https://medium.com/@hanzhongliu1990/big-data-applications-10c2b93d7be7?source=rss-f80de040b8c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/10c2b93d7be7</guid>
            <dc:creator><![CDATA[Henry Han]]></dc:creator>
            <pubDate>Sun, 18 Feb 2018 18:31:52 GMT</pubDate>
            <atom:updated>2018-02-25T16:14:07.482Z</atom:updated>
            <content:encoded><![CDATA[<h3>Big Data Applications in Various Industries</h3><h4>Overview</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*BHwFTpTQamNJEW7Agr631A.png" /><figcaption>Figure 1: Big Data application Overview</figcaption></figure><h4>Some examples of big data applications in each industry</h4><ol><li>Daily Life</li></ol><ul><li>Customized service</li></ul><p>2. Government</p><ul><li>Election</li></ul><p>3. Security</p><ul><li>National security</li><li>Defense network attacks</li><li>Prevention of crime</li></ul><p>4. Sports &amp; Entertainment</p><ul><li>Team training</li><li>Investment in films and television programs</li><li>Forecast result of games</li></ul><p>5. Energy</p><ul><li>Smart grid</li></ul><p>6. Telecom</p><ul><li>Analysis of telecom customer churn</li></ul><p>7. Food</p><ul><li>Dining o2o (online to offline)</li></ul><p>8. Internet</p><ul><li>Recommendation system</li></ul><p>9. Biomedicine</p><ul><li>Forecast epidemic diseases</li><li>WIT120 (Wise Information Technology of 120)</li><li>Bioinformatics</li></ul><p>10. Logistics</p><ul><li>Intelligent logistics</li><li>“Green hand” (China Smart Logistics Network) — CSN</li></ul><p>11. City Management</p><ul><li>Intelligent Transport System</li><li>Environmental monitoring</li><li>Urban planning</li><li>Security &amp; Protection system</li></ul><p>12. Finance</p><ul><li>Block trade</li><li>Market sentiment analysis</li><li>Credit risk analysis</li></ul><p>13. Vehicle</p><ul><li>Driverless cars</li></ul><p>14. Retail</p><ul><li>Discover consumer behaviour</li><li>Customer segmentation</li><li>Supply chain management</li></ul><h3>Study Case: Internet — recommendation system</h3><h4>What is recommendation system?</h4><p>The rapid development of Internet let us enter the age of overloaded information. The search engine can help us find information, but can only solve specific and clear problems.</p><p>In order to allow users to effectively obtain what they need from massive information, the recommendation system was brought into being. The recommendation system is a typical application of big data used on the Internet. It understands users’ preferences by analyzing users’ history so that it will actively recommend interesting information and satisfy users’ personal recommendation needs.</p><p>A recommendation system is a tool that automatically connects users and items. To compare with a search engine, recommendation system studies users’ interest preferences and conducts personalized computing. It can find users’ interest point and help users to explore potential needs.</p><p><strong>Recommendation system definition on Wikipedia</strong></p><blockquote>A recommender system or a recommendation system (sometimes replacing “system” with a synonym such as “platform” or “engine”) is a subclass of information filtering system that seeks to predict the “rating” or “preference” a user would give to an item.</blockquote><h3>The long tail</h3><p>Recommendation system can create new business and economic patterns which help push or develop the sales of “long tail” items.</p><p>A long tail item is used to describe the retailing strategy of selling a large number of unique items with relatively small quantities sold of each (the “long tail”) — usually in addition to selling fewer popular items in larger quantities (the “head”).</p><p>The “long tail” was popularized by Chris Anderson in an October 2004 Wired magazine article, in which mentioned Amazon, Apple and Yahoo as examples of businesses applying this strategy. Anderson elaborated the concept in his book “The Long Tail: Why the Future of Business Is Selling Less of More”.</p><p>There are various kinds of products selling on e-commerce websites. Although most of them are not popular, they do have large quantities and potential considerable cumulative sales.</p><p>Recommendation system will discover these long tail items, push them to the users who may have interest, and increase sales eventually. This requires personalized recommendations to implement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/274/1*9kwhUMlwXC1a5T7eXwA4lA.png" /><figcaption>Figure 2: An example of a power law graph showing popularity ranking. To the right (yellow) is the long tail; to the left (green) are the few that dominate. In this example, the cut-off is chosen so that areas of both regions are equal</figcaption></figure><p>“Hot offers” is the most popular way of recommendation and widely used in all kinds of websites, such as “the best hit” in music websites or apps. The main disadvantage of “hot offers” is the limited range of recommendation, and the content of recommendation in particular period is relatively unchangeable or fixed.</p><p>The customized recommendation can be implemented by the recommendation system. The recommendation system finds users’ customized needs and spending propensities by discovering users’ records of behaviour (history). The long tail items will be accurately recommended to whoever needs them. The recommendation system achieves the win-win situation between users and merchants.</p><h4>Recommendation algorithms</h4><p>The nature of a recommendation system is to establish the connection between users and items. There are five kinds of recommendations (systems) described as the follows:</p><ol><li>Expert recommendation</li></ol><ul><li>The recommendation list is created and filtered by real people, especially experts.</li></ul><p>2. Statistics-based recommendation</p><ul><li>Same idea as “hot offers” based on the number of views or visits ranking from high to low</li><li>Easy to implement but no customization</li></ul><p>3. Content-based recommendation</p><ul><li>Recommend similar items based on the features of the items</li></ul><p>4. Collaborative filtering</p><ul><li>One of the earliest and successful ways of recommendation</li><li>Based on the comments from similar users</li></ul><p>5. Hybrid recommendation system</p><ul><li>a mix of the above</li></ul><h4>Recommendation system models</h4><p>A complete recommendation system model consists of 3 modules:</p><ol><li>The user module</li></ol><ul><li>Model user</li><li>Analyze users’ interests and needs based on user behaviour data and user attribute data set</li></ul><p>2. The item module</p><ul><li>Model item based on item attribute data set</li></ul><p>3. The algorithm module</p><ul><li>Apply recommendation algorithms to obtain the items that users may have interests</li><li>Based on user characteristics and item features</li><li>Adjust the recommendation result based on the recommendation scene and push the final result to users</li></ul><h4>Recommendation system applications</h4><p>Currently, the recommendation system has been widely in e-commerce, online video, online music and social media websites and applications.</p><p>Amazon recommends items based on user browsing history, and most of the recommended items are those that users have never browsed but users may have interests and potential purchase possibility.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*3yjb1Ek-q3hBYNMbLDw4HQ.png" /><figcaption>Figure 3: Amazon recommends products based on user browsing history</figcaption></figure><p>The recommendation system gradually plays a role in online music. Music has much larger quantities than movies and more obvious personal preferences, so it’s not enough to only rely on “hot offers”.</p><p>A Chinese music web/app site called Xiami uses music library/collection to analyze users’ music preferences to make recommendations, such as recommend similar-style music or music from the same singer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*Bc1BzjIGcmtne6QdEHHquA.png" /><figcaption>Figure 4: Xiami recommendation system (Guess what you like) based on personal music library</figcaption></figure><h4>Collaborative Filtering (CF)</h4><p>Recommendation technology has been developed for more than 10 years, and many new recommendation algorithms have been created during the development. Collaborative filtering, as the earliest and most famous algorithm, has been further delved and widely used so far.</p><p>Collaborative filtering can be divided into UserCF and ItemCF.</p><h4>UserCF</h4><p>The collaborative filtering algorithm based on user was brought forward in 1992, and it is the oldest recommendation algorithm.</p><p>To achieve UserCF:</p><p>1. Find the set of users who have similar interests with the target user</p><p>2. Find all the items which the users in the set above like and target user has never heard of</p><p>3. Recommend those items to the target user</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*2sW_IUV61jH4FiiU7510EQ.png" /><figcaption>Figure 5: UserCF example</figcaption></figure><p>In the figure above, User A likes Item A and Item C, User B like Item A and Item B. Because both User A and User B like Item A, they’re considered as “similar users”. Since User B has never heard of Item C, Item C will be recommended to User B.</p><p>The key step of achieving UserCF is to calculate the similarity among users (correlation). The most popular ways to measure the linear correlation are:</p><ul><li>Person correlation coefficient</li><li>Cosine-based similarity</li><li>Adjusted cosine similarity</li></ul><p>The formula of cosine-based similarity is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/179/1*IQHB4ssAsRpKbnIUo40r7w.png" /><figcaption>Figure 6: Cosine-based similarity formula</figcaption></figure><p>Given User A and B, n(A) is the set of items which User A likes, and n(B) is the set of items which User B likes. n(A∩B) is the set of items which both User A and B like, and n(A)×n(B) is the set of all the items which either User A or B likes or both.</p><p>The problem of using this method is that many users do not have any behaviours on the same item, so the numerator of the formula above is 0 and the similarity (correlation) is 0.</p><p>To solve this problem, use the inverted list from item to user. See the figures below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/324/1*gl466AikpW_brBqKBHd2JQ.png" /><figcaption>Figure 7: Inverted list</figcaption></figure><p>In this way, we can only calculate similarity only among users who have at least “1” similarity.</p><p>After obtaining the similarity K, use the following formula to calculate the degree of interest that User u has on Item i, which is P(u, i).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/1*JpSI7fL7Hi2XuDMybXlTVA.png" /><figcaption>Figure 8: Formula of calculating degree of interest</figcaption></figure><p>S(u, m) is the set of m users who have the most similar interests. N(i) is the set of users who like Item I, K is the similarity of User u and v. r is the implicit feedback representing the degree of interest that User v has on Item i. To simplify the calculation, let r = 1.</p><p>After calculating P(u, i) on all the items, list them in descending order, take the first n items as the recommendation result and display to User u (called Top-n recommendation)</p><h4>ItemCF</h4><p>The collaborative filtering algorithm based on item (ItemCF) is the most popular and widely used recommendation algorithm. Both Amazon and Netflix use ItemCF as their recommendation system algorithm.</p><p>ItemCF provides similar items that target user used to like. It calculates the similarity among items by analyzing a user’s behaviours and history.</p><p>The assumption of this algorithm is: having a large similarity between Item A and Item B is because most of the users who like Item A also like Item B.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*RTwNO87tJgch7-eEFp7s2A.png" /><figcaption>Figure 9: ItemCF example</figcaption></figure><p>To achieve ItemCF:</p><p>1. Calculate similarity among items</p><p>2. Create recommendation list based on item similarity and user behaviour history.</p><p>ItemCF uses the inverted list from user to item to calculate the item similarity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*wRyxtiB-Gf4qyGxsBdicow.png" /><figcaption>Figure 10: Inverted list</figcaption></figure><p>ItemCF calculates the item similarity and then use the following formula to calculate the degree of interest that User u has on Item j</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/208/1*CrKMnW7wL5GT47P33aQQWg.png" /><figcaption>Figure 11: Formula of calculating degree of interest</figcaption></figure><p>S(j, K) is the set of K items that are the most similar to Item j. N(u) is the set of items that User u likes. w is the similarity of Item i and Item j. r is the implicit feedback representing the degree of interest that User u has on Item i. To simplify the calculation, let r =1.</p><h4>UserCF vs ItemCF</h4><p>The algorithms and the calculation processes of UserCF and ItemCF are similar.</p><p>The main difference:</p><p>UserCF recommends the items from other users who have common interests with the target user.</p><p>ItemCF recommends the items which are similar to those items target users used to like.</p><p>UserCF is more “socialized” while ItemCF is more “personalized”.</p><p>UserCF is suitable for news recommendation and Twitter topics, and the recommendation result has some advantages in novelty.</p><p>The disadvantage of UserCF: With the rapid increase of users, the similarity calculation becomes more and more complex. The correlation of the recommendation is relatively weak, and it is hard to explain the result. The recommendation result is easily influenced by the public.</p><p>ItemCF is suitable for application scenes like e-commerce, movies and books. It takes the advantage of the user’s history to explain the recommendation result and provide conviction to the user.</p><p>The disadvantage of ItemCF: It prefers to recommend similar items that the user has already purchased. The lack of variety and novelty is the main problem of this algorithm.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=10c2b93d7be7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is Big Data?]]></title>
            <link>https://medium.com/@hanzhongliu1990/what-is-big-data-1bcb2535ca53?source=rss-f80de040b8c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/1bcb2535ca53</guid>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Henry Han]]></dc:creator>
            <pubDate>Sun, 28 Jan 2018 05:42:56 GMT</pubDate>
            <atom:updated>2018-02-04T03:54:04.583Z</atom:updated>
            <content:encoded><![CDATA[<p>The concept of Big Data is getting more and more popular recently and you may have heard of that, but what is Big Data?</p><p><strong>Definition</strong></p><p>There are many definitions of Big Data out there on the Internet, but I would like to introduce this concept from my personal understanding. Big Data is a very large volume of data sets hard to capture and manage using regular analysis (data processing) software within a certain amount of time. It requires some new processing methods or technologies that have more powerful functions.</p><p>Another name for Big Data is Mega Data.</p><p>The importance of this concept is not about how Big the data is, it is about how you process that big amount of data. In other words, if Big Data can be described as an industry, the key to making profits is improving the performance of processing data and adding value to existing data.</p><p>In terms of technology, Big Data is closely connected with Cloud Computing. It is impossible to process Big Data using only one computer. It must use distributed architecture. It operates data mining on a tremendous amount of data. It also relies on cloud computing’s distributed processing, distributed database, cloud storage and virtualization.</p><p>With the coming of cloud computing, Big Data gets more and more attraction. Big Data usually describes a large amount unstructured data and semi-structured data that a company created, and these data would waste too much time and money during the analysis process when downloaded to a relational database.</p><p>Big Data requires some special technologies, in order to effectively process that amount of data within the tolerance time. The technologies include massively parallel processing database, data mining, distributed file system, distributed database, cloud computing platform, Internet, and extendable storage system. (These technologies will be discussed in the “technical terms” section below)</p><p>The base unit of Big Data is the same as regular data.</p><p><strong>History</strong></p><p>Is Big Data a new term? The answer is no. The term, Big Data, has been in use since the 1990s, with some giving credit to John Mashey for coining and make it popular. While this term is still relatively new to people, the act of gathering and storing large amounts of data or information for analysis is actually old.</p><p><strong>Characteristics</strong></p><p>Big Data has a so-called “5-V” features:</p><p>1. volume</p><p>The quantity of generated and stored data determines the value of data and potential information being considered.</p><p>2. Velocity</p><p>This describes how fast data is generated, collected and processed. Does it meet the demands?</p><p>3. Variety</p><p>The type and nature of data being collected help analysts effectively use the resulting insight.</p><p>4. Variability</p><p>The inconsistency of the data set can hamper processes to handle and manage it.</p><p>5. Veracity</p><p>The data quality of collected data can greatly affect the accuracy of the analysis.</p><p>More (additional) features:</p><p>6. Complexity</p><p>The enormous size of data should come from various sources.</p><p><strong>Architecture</strong></p><p>Big Data includes structured, semi-structured and unstructured data, and unstructured data has become the major component of the data. According to the research report by IDC (a premier global market intelligence firm), 90% of the data from enterprises is unstructured data, and this increases by 60% every year. <a href="https://www.intel.com/content/dam/www/public/us/en/documents/solution-briefs/big-data-101-brief.pdf">https://www.intel.com/content/dam/www/public/us/en/documents/solution-briefs/big-data-101-brief.pdf</a> (PDF document from Intel)</p><p>Big Data is just a representation and characteristic of the current Internet, and there is no need to myth it. With the use of clouding computing and other newer technologies, these data that seemed to be difficult to collect has been easily used. With the constant innovations in various industries, Big Data will create more values to people gradually.</p><p>If you would like to know Big Data system systematically, you need to break it down into 3 steps, which is very similar to when you study a new technology or skill.</p><p>1: Theory</p><p>The theory is the base of everything when you would like to know something before really digging into it.</p><p>2: Technology</p><p>Technology is the way of showing the value of Big Data. Big Data involves many other technologies mentioned above.</p><p>3: Practice</p><p>Practice is true worth of Big Data. From the uses of Internet Big Data, Government Big Data, Enterprise Big Data and personal Big Data, it already shows the value of Big Data and the blueprint for the future.</p><p><strong>Significance</strong></p><p>The current society is a society with high-speed development, technology advances, information flows, communication among people is getting closer and closer, life is getting more and more convenient, and Big Data is the product of this high-tech age. The founder and CEO of Alibaba, Jack Ma mentioned in one of his speeches, “future will not be IT time but DT time”, and DT stands for Data Technology. Apparently, data plays a decisive role in Alibaba Corporation.</p><p>Again, the importance of big data is not about how much data you have, but what you do with it. You can take data from any source and analyze it to find answers that enable 1) cost reductions, 2) time reductions, 3) new product development and optimized offerings, and 4) smart decision making. When you combine big data with high-powered analytics, you can accomplish business-related tasks such as:</p><ul><li>Determining root causes of failures, issues and defects in near-real time.</li><li>Recalculating entire risk portfolios in minutes.</li><li>Detecting fraudulent behaviour before it affects your organization.</li></ul><p><strong>Trends</strong></p><p>Big Data has the following trends:</p><p>1. Data Resourcelization</p><p>Data resourcelization means Big Data has become an important strategic resource and a new focus of both enterprises and society. In order to capture the market opportunities ahead, an enterprise must make plans on marketing strategy specifically for Big Data.</p><p>2. Combined with Cloud Computing</p><p>Big Data can never work without could processing, and cloud processing provides Big Data flexible and extensive infrastructure. Cloud processing is one of the platforms for producing Big Data. Since 2013, Big Data has started to work with cloud computing, they will work much closer in the future as predicted. Other than cloud computing, Internet of Things (IoT), mobile Internet and other emerging technologies will also help with the revolution of Big Data, and let Big Data marketing make more influences.</p><p>3. Breakthrough in Scientific Theories</p><p>Similar to computers and Internet, Big Data might be the next evolution of technology. With other technologies, like data mining, machine learning, and artificial intelligence, Big Data might also change a lot of algorithms and basic theories, and make breakthroughs in science and technology eventually.</p><p>4. Foundation of Data Science and Data Alliance</p><p>In the future, data science will be a specialist subject and will be recognized by more and more people. Many universities will establish a department of data science, and more related jobs will be created.</p><p>Meanwhile, data sharing platforms of cross-industry will be founded. Data sharing will be extended to the level of enterprise and become the core the industry in the future.</p><p>5. Data Leakage/breach</p><p>The growth rate of data leakage incidents may reach 100% unless data could be secured on the source side. All companies, regardless of size, need to re-examine the definition of “safety”. Data should be protected when it is created, not when it is saved at last.</p><p>6. Data Management Becomes Core Competence</p><p>Data management directly influences the financial performance of a company. This will be discussed in the applications section in the next blog.</p><p>7. Data quality Is the Key to Success of Business Intelligence</p><p>The companies using self-servicing business intelligence tools will make themselves stick out from the crowd. One of the challenges they are going to face is that many data sources bring large amounts of the low quality of data. To succeed, companies need to understand the gap between original data and data analysis in order to eliminate low quality of data and make better decisions using business intelligence.</p><p>8. Degree of Data Ecosystem Compounding Is Strengthened</p><p>The world of Big Data is not only just a single, enormous computer network but an ecosystem composed of active members and multiple participant elements. This includes terminal equipment providers, infrastructure providers, Internet service providers, network access service providers, data service enablers, data service providers, contact services, data service retailers and other participants. Currently, the prototype of this ecosystem has been formed, and the next step will be towards subdividing the system internal roles.</p><p><strong>Technical terms mentioned above</strong></p><p>Massively parallel processing is the coordinated processing of a program by multiple processors working on different parts of the program. Each processor has its own operating system and memory. MPP speeds the performance of huge databases that deal with massive amounts of data.</p><p>Massively parallel processing databases use multi-core processors, multiple processors and servers, and storage appliances equipped for parallel processing. That combination enables reading many pieces of data across many processing units at the same time for enhanced speed. This method is necessary because the frequencies of processors are hitting the limits of the technologies used and are slow to increase.</p><p>Clouding computing is an information technology paradigm that enables ubiquitous access to shared pools of configurable system resources and higher-level services that can be rapidly provisioned with minimal management effort, often over the Internet.</p><p>Data mining is the process of discovering patterns in large data sets involving methods at the intersection of machine learning, statistics, and database systems.</p><p>Distributed processing is a phrase used to refer to a variety of computer system that uses more than one computer (or processor) to run an application. This includes parallel processing in which a single computer uses more than more than one CPU to execute programs.</p><p>A distributed database is a database in which storage devices are not all attached to a common process. It may be stored in multiple computers, located in the same physical location; or may be dispersed over a network of interconnected computers.</p><p>A distributed file system is a client/server-based application that allows clients to access and process data stored on the server as if it were on their own computer. When a user accesses a file on the server, the server sends the user a copy of the file, which is cached on the user’s computer while the data is being processed and is then returned to the server. Extendable storage system</p><p>Cloud storage is a model of data storage in which the digital data is stored in logical pools, the physical storage spans multiple servers (and often locations), and the physical environment is typically owned and managed by a hosting company. These cloud storage providers are responsible for keeping the data available and accessible, and the physical environment protected and running.</p><p>Virtualization refers to the act of creating a virtual (rather than actual) version of something, including virtual computer hardware platforms, storage devices, and computer network resources.</p><p>Internet of Things is the network of physical devices, vehicles, home appliances, and other items embedded with electronics, software, sensors, actuators, and network connectivity which enables these objects to connect and exchange data.</p><p>Business Intelligence (BI) comprises the strategies and technologies used by enterprises for the data analysis of business information. BI technologies provide historical, current and predictive views of business operations.</p><p><strong>Applications</strong></p><p>Applications of Big Data will be discussed in the next blog post.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1bcb2535ca53" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>