<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Madhu Kochar on Medium]]></title>
        <description><![CDATA[Stories by Madhu Kochar on Medium]]></description>
        <link>https://medium.com/@madhu_kochar?source=rss-322b52404cfb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*XcZi-h3YW0q1II-F.jpg</url>
            <title>Stories by Madhu Kochar on Medium</title>
            <link>https://medium.com/@madhu_kochar?source=rss-322b52404cfb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 14:33:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@madhu_kochar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Applying Machine Learning to Data Stewardship]]></title>
            <link>https://medium.com/inside-machine-learning/applying-machine-learning-to-data-stewardship-d3ff9f4874e1?source=rss-322b52404cfb------2</link>
            <guid isPermaLink="false">https://medium.com/p/d3ff9f4874e1</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-stewardship]]></category>
            <dc:creator><![CDATA[Madhu Kochar]]></dc:creator>
            <pubDate>Mon, 12 Aug 2019 20:40:37 GMT</pubDate>
            <atom:updated>2019-08-12T20:50:51.256Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Madhu Kochar and Martin Oberhofer</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*yqFQMoOmL11e_bW8F7G1Iw.png" /><figcaption><a href="https://www.burningsettlerscabin.com/burningsettlerscabin/tag/Edward+Weston">photo: Herbert Ponting</a></figcaption></figure><p>Master Data Management (MDM) solutions are designed to effectively manage your master data. Let’s assume for example a bank A requires another bank B doing a merger &amp; acquisition. Then in the IT departments you need to reconcile the systems managing your master data such as customer, products, employees, suppliers, etc. An MDM system helps you with that process by using a matching algorithm to decide whether or not for example a person record of John Doe from bank A is the same person as a person record Jim Doe from bank B. This is shown in Figure 1 below. To determine an answer to this question, the matching algorithm is comparing attributes like names, nationality, data of birth, social security numbers, etc. and depending on how similar or distant the values in these fields are, assigns weights to them — these are the numbers like 1.2, 7.4, etc.</p><p>In IBM MDM, there are many different individual functions the matching algorithm uses like edit distance, phonetics, nickname resolution, geocode resolution, transliteration, etc. to allow a deep, fuzzy inspection in real-time at big data scale — our largest customers run on billions on records. The total score 16.2 in the example in the Figure is the sum of the individual weights on the attributes. The total score is then compared against the lower threshold (in the example of the Figure that’s 10.0) and the upper threshold (in the example of the Figure that’s 20).</p><p>Depending on that comparison, the match result is classified as no link, clerical or autolink case (autolink are also known a merge cases). The results in the clerical category are also known as duplicate suspect tasks is the task list for the data steward.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*pgU8gj-FgzYm56Z9ZSGJmg.png" /><figcaption>Figure 1</figcaption></figure><p>Now looking at this task list, a day in the life of a data steward resolving duplicate suspect tasks might feel like Phil Conners played by Bill Murray in <a href="https://www.themoviedb.org/movie/137-groundhog-day?language=en-US">Groundhog Day</a> — or Charlie Chaplin in the movie <a href="https://www.imdb.com/title/tt0027977/">Modern Times</a> working at the assembly line — doing the same thing over and over and over again. As a result, the satisfaction with work is low and we heard from our clients there is a significant amount of turnover in their data stewardship teams. Digging into the that problem we found:</p><ul><li>When a master data management (MDM) system gets deployed — it’s a net new technology providing a sophisticated function known as matching which in essence is a capability to identify whether or not two or more data records from the same or different sources are duplicates and should be reconciled.</li><li>No matter which matching technology is used, there is a risk of false negatives and false positives. Too many false negatives could create the issue that you might not find cases which are relevant for Anti-Money Laundry (AML) compliance. False positives might cost you reputation if customers get upset if their data has been erroneously merged with the data of somebody else. For fear of such cases — the thresholds for the boundary non-link to clerical task for a data steward and the boundary of clerical task for a data steward to auto-link are initially set in such a way — that much more decisions then necessary are routed to data stewards. In other words — a lot of cases end up in the clerical task bucket which could have been decided automatically which creates boredom with data stewards when they realize its more of the same every day.</li><li>Making matters worse, consider the following: A medium size MDM project on initial deployment might load 20 million records into MDM. If only 1% of these records would produce a match result in the clerical range for clerical review by data stewards, that’s 200.000 tasks. If a data steward is able to resolve lets say 200 tasks a day — a single data steward would need roughly three years (no weekends) to clean up that task list. Keep in mind — during these 3 years new tasks might be created if additional sources are onboarded or new customers records by a growing business might be created — so its very hard to catch up ever.</li></ul><p>To address these problems we applied Machine Learning in IBM MDM to reduce the amount of work for data stewards significantly. As input data for training machine learning algorithms we used the resolution history of tasks data steward have already done. For each duplicate suspect task based on the data steward resolving it we had the information if it ultimately was a no link or a merge case which makes it labeled data. The key insight here is that with the help of machine learning we could capture the wisdom of the data stewards in a prediction model which allows for new tasks to predict with a high degree of certainty how a human data steward is going to do resolve this particular task.</p><p>This is insight you can derive by looking at what the data stewards have actually done using machine learning. With that insight — you can then assist data stewards in two ways: First, while introducing the new functionality, you can show the data stewards the recommendation on how to resolve the clerical task using the machine learning prediction. This already reduces the work per task significantly — mostly review and approve.</p><p>Once they trust the system, the data stewards can enable a flag that the machine learning backend takes the action automatically if the prediction is above a certain prediction threshold, e.g. lets say more then 95% certain of the recommendation. Once this flag is enabled the task is not even showing in the task list of the data stewards anymore.</p><p>We compared over a dozen different machine learning in terms of prediction quality for this particular problem and found that a Random Forest model produced the best results. However, to get very good prediction quality, such a model required approximately 5000 tasks resolved by data stewards as training data. Exploring clustering and active learning approaches, we could decrease this requirement to approximately 250 tasks to train the Random Forest model to the same level of accuracy.</p><p>Figure 2 shows conceptually the flow of the solution which we released earlier in 2019.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/817/1*OL1MzgRM7DEdFc8G6sQrAg.png" /><figcaption>Figure 2</figcaption></figure><p>Here’s a figure that combines the figure above with the figure at the start of the post:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w7HMcC8lcEHeOnA_P3C_Hw.png" /><figcaption>Figure 3</figcaption></figure><p>In summary, by bringing machine learning to MDM, we could reduce the amount of tasks created for data stewards by up to 2/3 which yields a significant reduction of work for the data stewardship team and hence also a significant amount of labor costs. At the same time, the remaining tasks are the ones which are less repetitive in nature and more complex allowing to focus the data stewards on more interesting tasks. If you want to learn more about this IBM MDM innovation, you can find more here: <a href="https://www.ibm.com/support/knowledgecenter/en/SSWSR9_11.6.0/com.ibm.swg.im.mdmhs.ml.stewardship.doc/topics/mlds_overview.html">IBM MDM Machine Learning</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d3ff9f4874e1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inside-machine-learning/applying-machine-learning-to-data-stewardship-d3ff9f4874e1">Applying Machine Learning to Data Stewardship</a> was originally published in <a href="https://medium.com/inside-machine-learning">Inside Machine learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Love what you do]]></title>
            <link>https://medium.com/inside-machine-learning/love-what-you-do-ca8a2d8eafae?source=rss-322b52404cfb------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca8a2d8eafae</guid>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Madhu Kochar]]></dc:creator>
            <pubDate>Tue, 09 Apr 2019 16:08:23 GMT</pubDate>
            <atom:updated>2019-04-15T18:41:00.554Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*6terAJj4X21EyCD7aFZ6zw.png" /></figure><p><strong><em>Madhu Kochar and Jay Limburn</em></strong></p><p>I was chatting to a friend recently who is an air traffic controller. He explained that since childhood he has been fascinated by all things aviation. Not just the thrill of flight, but the mechanics behind the incredible machines that make the world such a small place. He loves his job because he gets to spend all day working directly with these machines and using his knowledge of them to directly affect the path each one takes, being able to understand individual aircraft performance, climb rates, air speed, weights and suitable airports as well as having a direct line to the captains. However, with the enjoyment his chosen profession brings comes a degree of mundaneness.</p><p>A key part of the job is to ensure that the data being received from adjoining sectors of airspace is accurate and clear. As a flight enters his sector he has to check and recheck that the data received on that flight is accurate and has his trust, before he can even start to focus on the fun part of how to direct that flight. This involves a huge amount of knowledge about the procedures that need to be followed and constant recertification to the latest regulations and processes. This part can be tedious, mundane yet essential to ensure safe passage of aircraft through his airspace. As air traffic increases and the number of available data points increase this task is ever growing, increasingly Air Traffic Controllers will need to turn to intelligent data management techniques to ease this burden to allow air traffic controllers to spend more time orchestrating their city in the sky.</p><p>The same is true of data scientists — The “2017 Data Scientist Report”¹ from CrowdFlower showed that 64% of Data scientists responding agree that they are working in the 21st century’s sexiest job. This is validation that Data Scientists love what they do. They love the key aspects of delivering value from their data, but like my air traffic controller friend a large proportion of their work requires them to spend time on tasks that are not so much fun. Tasks that if they could minimize or eradicate altogether might give them more time to spend on the things that they love doing.</p><h3><strong>Being the Janitor</strong></h3><p>So, what is it that data scientists love and hate? Data scientists I have interacted with seem to have gotten into data science to do cool things with data and math. They love going deep into data and identifying non obvious patterns that they could use. They love building and tailoring the best algorithm to deliver the most efficient model to answer questions, identify patterns in images and ultimately prove they are the top of the data science pyramid by delivering something far cooler and deriving more value than their peer sat across the table. From the same report, the three tasks data scientists reported enjoying the least are collecting data assets (48%), labelling data (51%) and cleaning and organizing data (60%). This becomes the “trusted data” that data scientists need before they can embark on the exciting elements of developing, training and deploying their models. We can refer to these earlier tasks as ‘Janitorial Tasks’ — necessary but no fun. Data Scientists didn’t get into data science to be wasting time searching across disorganized, random systems to find data. Nor do they desire to spend hours arguing with the IT guys explaining the actual data they requested is not what they were provided with.</p><h3><strong>Make Artificial Intelligence your Janitor</strong></h3><p>With data volumes and variety of data expected to grow at an accelerated rate for the foreseeable future, the time spent by data scientists carrying out those janitorial tasks is only likely to increase. Not only is this frustrating to data scientists, it’s inefficient to the organizations hiring the data scientists to deliver business value. So how can a data scientist decrease the time spent carrying out some of those more mundane janitorial tasks? Ironically the answer might be Artificial Intelligence.</p><p>Augmenting Data Management with AI is an emerging discipline that uses traditional data management techniques infused with AI to provide an intelligent way to organize your data that allows your data science community to be able to more easily find, understand and extract it so they can get on with the good stuff of using it.</p><h3><strong>Helping to Accelerate the Journey to AI</strong></h3><p>Here at IBM we are on a mission to help organizations put in place efficient Data and AI teams, allowing them to reduce the mundane tasks throughout the data and AI lifecycle and focus on delivering value to the business. We refer to this as AI for Governance.</p><p>As we explore a typical Data &amp; AI life-cycle we can call out the areas where data scientists benefit from the use of AI for Governance. This is part of what IBM refers to as the “AI Ladder” as show in figure #1 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nYqWFwcRmKiGBv4GXvphbw.png" /><figcaption><strong>Figure #1: IBM AI Ladder</strong></figcaption></figure><ol><li><strong>AI Powered discovery of data. </strong>Intelligent discovery of sources of information. Being able to discover and collate enterprise and external data regardless of type or structure and transforming into a standardized format providing consumers a ‘DNS for their data’. Allowing them to easily be pointed to the best data for their purpose.</li><li><strong>AI powered profiling, quality and bias detection of the data. </strong>Utilizing AI to understand the data that you are cataloguing. Pre trained models that can inspect data elements and determine where sensitive data resides, and which business entities exist across your data landscape. Automatized detection of data quality issues, whilst determining if data could include bias if used in a certain manner.</li><li><strong>Intelligent findability of data. </strong>Utilize the richness of the metadata in your catalog and use it to train an AI model to help findability of data. Unlock dark data sets and allow AI to recommend them to your users based upon their previous usage of the system.</li><li><strong>AI Powered suggestions on data prep operations. </strong>Data preparation is a key part of effective data science. AI can assist the data scientist with this preparation. Making suggestions as to which data needs to be wrangled to provide the best set of training data for a model.</li><li><strong>Policy Activation on the data. </strong>Being able to activate your data protection policies so that data can be masked on the fly, at the point of consumption, dependent on who is using it and what their intent is. Simply put, policy activation puts more data in the hands of your knowledge workers then before whilst helping to ensure your data protection rules are enforced.</li></ol><h3><strong>Wrapping it all up</strong></h3><p>Data Scientists love what they do and can be the king makers in an organization’s future. However, they need to be relieved of the mundane tasks that take them away from delivering value and diminish their enthusiasm for their passion.</p><p>Data Management capabilities that use AI for governance to better provide organized data, that is easily accessible to the data science community can be a key factor to improve the overall outcome of your AI projects — such as those listed below.</p><ul><li><a href="https://www.ibm.com/uk-en/cloud/watson-knowledge-catalog">IBM Watson Knowledge Catalog</a> provides enterprise catalog and governance capabilities designed to drive productivity from data scientists.</li><li><a href="https://www.ibm.com/cloud/watson-studio">IBM Watson Studio</a> provides an enterprise grade data science builder’s environment designed to quickly deliver value through AI</li><li><a href="https://www.ibm.com/products/cloud-private-for-data">IBM Cloud Private for Data</a> provides an end to end data and AI platform with integrated capabilities making your data simple and accessible to power AI at scale.</li></ul><p>These are built to help infuse AI into Governance and support the Data &amp; AI Lifecycle, helping organizations to unshackle their data scientists and turn them from spending time as the janitor to being the rock stars of their organization.</p><p>Register for our upcoming webinar to learn how AI can be used to underpin your Governance initiatives: <a href="https://event.on24.com/wcc/r/1964130/0B5952F69C16E4003F90C8612E19F3A7?partnerref=jblog">https://event.on24.com/wcc/r/1964130/0B5952F69C16E4003F90C8612E19F3A7?partnerref=jblog</a></p><p>Sign up for a demo here and discover how IBM Watson can transform the way you do Data Science and Analytics.</p><p>¹ “2017 Data Scientist Report” from CrowdFlower.</p><p><strong>Jay Limburn — Director, Offering Management, IBM Data &amp; AI<br>Madhu Kochar — VP, IBM Data &amp; AI</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca8a2d8eafae" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inside-machine-learning/love-what-you-do-ca8a2d8eafae">Love what you do</a> was originally published in <a href="https://medium.com/inside-machine-learning">Inside Machine learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analytics Powered by ML — Watson Explorer Community Edition]]></title>
            <link>https://medium.com/ibm-data-ai/analytics-powered-by-ml-watson-explorer-community-edition-5a3aedac74fe?source=rss-322b52404cfb------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a3aedac74fe</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Madhu Kochar]]></dc:creator>
            <pubDate>Fri, 27 Oct 2017 09:25:26 GMT</pubDate>
            <atom:updated>2017-11-21T15:17:38.862Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Eg0VJb86IWtIC6FKXGSsmw.jpeg" /></figure><p>Welcome to the first in a series of posts about how we’re embedding machine learning capabilities across our analytics offerings. IBM is at the leading edge of this trend, which comes from a recognition that any product that stores, manages, or governs data will multiply in value for customers if it can also perform machine learning analytics on the data locally and in real time.</p><p>That said, the machine learning capabilities can’t just be bolted on the side; they have to be carefully woven into the architecture. Let’s start with <a href="https://www.ibm.com/us-en/marketplace/content-analytics">Watson Explorer Community Edition</a>, our text mining and content analytic solution.</p><p>One of its strengths is an ability to help organizations to uncover and anticipate the underlying reasons for costly, reputation-crippling issues. For example, an auto manufacturer could apply the machine learning capabilities of Watson Explorer Community Edition to mitigate recalls, poor market fit, and angry customers. (By the way those aren’t just nice-to-haves. Currently, the auto industry spends $45 billion every year on recalls and warranty claims. Just for recalls, the average expense per vehicle is close to $600 — a huge hit to their balance sheets.)</p><p>Other sectors like retail, finance, and healthcare, might use the capabilities in different ways, but the common thread is uncovering insights into processes at their deepest level, in order to find the inefficiencies and the opportunities.</p><p>Watson Explorer Community Edition enables content administration and analytics with the support of three different personas. A system administrator persona uses the admin console to create a data set and to set up a crawler with the data sources. In this new edition, which is available for free download, users can set up their local file systems and directories to crawl while installing the product or after it is installed and configured.</p><h3><strong>Automotive Accidents: A Use Case</strong></h3><p>The business analyst/data scientist persona can create collections, analyze content, and save his analysis for further use and sharing. A line-of-business user can consume these insights using the dashboard. Let’s consider the use case where Chad, a data analyst, needs to analyze car accident reports for early quality warning. The first productive use cases are built into the product with guided popups, but I’ll outline the steps here.</p><ol><li>Chad <a href="https://www.ibm.com/account/reg/us-en/signup?formid=urx-20195">downloads Watson Explorer Community Edition</a> and installs the product on his laptop (Mac or Windows) in less than 15 minutes.</li><li>He launches the Content Miner web app from the landing page and logs in. He’s presented with a popup with options to run the Market Analysis tour or the Early Quality warning tour.</li><li>He chooses Early Quality Warning tour and clicks on the collection. A UI shows multiple facets generated by the tool.</li><li>Chad picks the facets, applies date range filter and clicks apply.</li><li>He follows the guided tutorial and selects a car model called “Hill Walker”. The search returns 51 documents.</li><li>Chad now clicks Analyze trends and anomaly. Not only does the tool show that this particular model of car had higher numbers of accidents in December and January, but the tool also answers “why” by showing Topics and Trends.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gtJwJ4LEM7u_aYdMpdTeyw.png" /><figcaption><em>Figure 1: Watson Explorer answers not just “what” but “why”.</em></figcaption></figure><p>Now that Chad knows December is the month accidents started to increase, he selects December and clicks Analyze more. Doing so retrieves characteristic words for the model and the accidents. To see more, Chad can select words such as “rust” and “severely” and click the documents that appear to dive deeper into the quality warnings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*97i8Pv9WJPa7GLngV6n-8Q.png" /><figcaption><em>Figure 2: Continuing the investigation</em></figcaption></figure><h3><strong>Larger trends</strong></h3><p>The use case just described is typical of larger trends. Like so many other industries, auto manufacturers are embracing tools like Watson Explorer Community Edition hoping to cut costs — and they’ve already had great success. By <a href="http://www.autonews.com/">one estimate</a>, data mining and other track-and-trace software solutions have already cut warranty costs by 20 to 30 percent, administration costs by 25 to 45 percent, and compliance costs by 30 percent.</p><p>Those numbers are a testament to the kind of transformation that’s possible with data analytics. As you’ve seen, Watson Explorer delivers insights by combining search and content analytics to help users find and understand the unstructured data that can tell the true story of the business. The current release uses machine learning to do Natural Language Processing (NLP), to cluster documents, and to expand on keyword queries to build out full pictures of what’s at play.</p><p>This edition expands potential target users of Watson Explorer by delivering easy-to-use functions and new Content Miner that offers guided navigation to improve the experience for novice business users.</p><p>Stay tuned for the next in this series on our efforts to embed machine learning, where we’ll look at a use case for Data classifications and automated Metadata discovery in our Unified Governance portfolio. In the mean time, feel free to <a href="https://www.ibm.com/account/reg/us-en/signup?formid=urx-20195">download</a> Watson Explorer Community Edition and try it yourself.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a3aedac74fe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ibm-data-ai/analytics-powered-by-ml-watson-explorer-community-edition-5a3aedac74fe">Analytics Powered by ML — Watson Explorer Community Edition</a> was originally published in <a href="https://medium.com/ibm-data-ai">IBM Data Science in Practice</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Quest for Total Information Awareness]]></title>
            <link>https://medium.com/ibm-analytics-development/the-quest-for-total-information-awareness-ed3f7469d0ef?source=rss-322b52404cfb------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed3f7469d0ef</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Madhu Kochar]]></dc:creator>
            <pubDate>Mon, 17 Apr 2017 19:56:32 GMT</pubDate>
            <atom:updated>2017-05-02T20:10:46.708Z</atom:updated>
            <content:encoded><![CDATA[<p>I’m sure you’ve experienced the feeling of having made a decision — and then discovering information that might result in a different outcome and saying to yourself — “If I knew then what I know now …..” <em>Buyer’s remorse, post cognitive dissonance, hindsight, wiser after the event</em> — all expressions of regret in some form or another.</p><p>With so much data created every second, we can never have every bit of information available at our fingertips so we make decisions based on what we think we know and have at the time.</p><p>In theory, the more information we have (provided it is accurate) the better the decision and outcome, since information is tied to risk, revenue, growth — and their associated antonyms. But, total information awareness is more an aspiration than a reality as we can never know everything and its context. I think of information awareness as four states of knowing:</p><ul><li>Know what you know — Information you know and have.</li><li>Don’t know what you know — Information you don’t know you have</li><li>Know what you don’t know — Information you know you need but don’t have.</li><li>Don’t know what you don’t know — Information you don’t even know exists.</li></ul><h3><strong>The Where and How</strong></h3><p>You might ask “Where and how do I start discovering what information is out there? And how do I expand that information into the different forms of knowledge so that I can encompass more and more data moving forward?” The first step is all about exploration and discovery of your data: “Know what you know”.</p><p>Watson Explorer (WEX) enables customers to start simple with basic search and discovery of information and then work up to advanced data and text mining, natural language processing, cognitive insight and visibility across multiple data sources as summarized in figure 1 below. It can help explore, analyze and interpret information across an organization and beyond, bringing all the information together in one place without physically moving it or changing it. Think of it as integration and analysis “at the glass” then applying advanced analytics to reveal deep insights, discover patterns and trends that you didn’t know existed. The “<a href="https://www.ibm.com/watson/whitepaper/forrester-wave-explorer.html">Forrester Wave™:Big Data Text Analytics Platforms, 2Q 2016</a>” report provides an analyst’s view on some of Watson Explorer’s capabilities:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*ee7dmuM_JkD3eDViEv0-vw.png" /><figcaption><em>Figure 1: Explore, analyze and interpret with Watson Explorer</em>.</figcaption></figure><p>Essentially, WEX helps deliver a 360 degree view of information you need in relation to a particular set of needs and brings it together in an interactive application through a set of real-time artifacts and services. But it’s also able to make tailored recommendations based on a set of variables chosen and configured by the user or recommended by the “system”. WEX then ranks the recommendations by importance — from geographic parameters, past outcomes, time-sensitive issues, trends, risk assessments of individuals, demographics, etc.</p><p>Cognitive capabilities are core to every aspect of the Explorer product and hence the link to machine learning and decision optimization [links to previous blogs]. This means that the product can leverage statistical algorithms to provide smarter results and enable users to explore more information and discover further insights. The cognitive search functionality works with other components such as Knowledge Studio, Natural Language — which enables non-technical users to find answers with ease. Simplicity is a key design principle so that users have access to advance capabilities without having to be domain experts. Business unit executives shouldn’t have to be data scientists to ask or gain insights from their data.</p><p>In addition, Watson APIs offer a cognitive interface to many other Watson services such as Personality Insights, Visual Recognition, Watson Discover Service, Watson Natural Language Understanding — so really the world is your oyster in terms of the amount and types of data that can be reached and analyzed. WEX and its ecosystem of services therefore can help organizations discover information that they don’t even know exists: “Don’t know what you don’t know”.</p><h3><strong>Exploring in a Private Cloud Environment</strong></h3><p>A hybrid cloud strategy can help organizations achieve flexibility and longevity. Many organizations are still primarily concerned with exploiting the untapped potential of the data behind their firewalls, but may want to be able to reach out to public clouds when ready, reusing their private cloud investments. To facilitate this need, WEX’s core capabilities are architected as shown in Figure 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*PxBpyHBxfW_wk38pBc_c9w.png" /><figcaption><em>Figure 2 : Watson Explorer — hybrid cloud high level architecture.</em></figcaption></figure><p>You’ll notice that the architecture is designed to draw data and capabilities from across the topology, from public to private. It’s up to you to set the boundary or what information and services you are interested in.</p><p>With such power and versatility at your fingertips comes great responsibility. It’s fitting then that the product ensures that users have access only to information that has been granted or permitted using the product’s sophisticated security and governance model.</p><h3><strong>Start Exploring Today</strong></h3><p>If you want to experience the power of Watson Explorer, I invite you to spend 5 minutes on this <a href="https://www.ibm.com/watson/explorer/assessment/index.html">calculator</a> to gain a better understanding of the time, resources and money your company could save by using this cognitive search and data analytics platform. And for those who are more visually stimulated here’s a very informative 3-minute <a href="https://www.youtube.com/watch?v=feLLE-MrLhE">video</a>.</p><p><em>Follow me on Twitter: @madhu_kochar</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed3f7469d0ef" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ibm-analytics-development/the-quest-for-total-information-awareness-ed3f7469d0ef">The Quest for Total Information Awareness</a> was originally published in <a href="https://medium.com/ibm-analytics-development">IBM Analytics Development</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>