<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Snehal Gawas on Medium]]></title>
        <description><![CDATA[Stories by Snehal Gawas on Medium]]></description>
        <link>https://medium.com/@snehalgawas?source=rss-c3235dc190fa------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*jtPB841E1aAbnHIV.</url>
            <title>Stories by Snehal Gawas on Medium</title>
            <link>https://medium.com/@snehalgawas?source=rss-c3235dc190fa------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 02:49:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@snehalgawas/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Top 4 Applications of AI Governance]]></title>
            <link>https://medium.com/ibm-data-ai/top-4-applications-of-ai-governance-3c0f6dcde1ef?source=rss-c3235dc190fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c0f6dcde1ef</guid>
            <category><![CDATA[ai-governance]]></category>
            <dc:creator><![CDATA[Snehal Gawas]]></dc:creator>
            <pubDate>Mon, 13 Mar 2023 10:32:05 GMT</pubDate>
            <atom:updated>2023-03-13T10:32:05.159Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This is Part 3 of a series of blogs on IBM AI Governance.</em></p><p>We are in a unique phase right now, where everyone is excited about AI and its potential impact on humanity. Along with bright future prospects, there is also a recognition of significant risks associated with growing AI innovations and adoption.</p><p>There is a significant increase in AI rules, regulations, and the <a href="https://medium.com/ibm-data-ai/the-need-for-ai-governance-8d88f2f3b941">need for AI Governance</a><strong> </strong>to manage this risk. Organizations are trying to understand this rapidly changing AI landscape and looking for ways to drive responsible use of AI.</p><p>The <a href="https://www.ibm.com/products/cloud-pak-for-data/ai-governance">IBM AI Governance solution</a> provides a set of tools and processes to help enterprises drive trustworthy and responsible use of AI. Learn about the three key pillars of AI Governance <a href="https://medium.com/ibm-data-ai/3-key-areas-of-ai-governance-8676f1acf0f2">here</a>.</p><p>In this blog, let&#39;s look at what is possible with the help of the IBM AI Governance solution. We will focus on top problems that can be solved with IBM AI Governance solutions which are applicable across multiple industries.</p><h3><strong>Automate AI lifecycle governance at scale with effective collaboration among growing stakeholders</strong></h3><p>AI governance is about putting the right guardrails in place. AI Lifecycle is a complex process involving a lot of tools, procedures, and people. Enterprises often start small and are able to handle the end-to-end process efficiently but they face problems when they want to scale.</p><p>How can you make sure that you can scale your models from 10 to hundreds with standardized governance practices, effective collaboration, and minimum efforts? That’s where automation comes into the picture. IBM can help you with two types of automation to implement AI Governance at scale.</p><p><strong>Automation of Model Tracking</strong></p><p>Catalog and organize all your machine learning models within your organization at a central place in a model inventory. This gives you the ability to track models across their lifecycle from request to evaluation and production deployment.</p><p><strong>Automation of Model Auditing</strong></p><p>Automatically document important key characteristics of models across their lifecycle to facilitate enterprise validation.</p><p>Generate customizable reports and share them with stakeholders and regulatory authorities to keep business processes auditable and compliant with regulations</p><p>If you want, you can choose to automate the build and deploy phases of your lifecycle with <a href="https://www.ibm.com/products/cloud-pak-for-data/data-science-mlops">IBM Data Science and MLOps </a>capabilities. But even if you are using other ML vendors you can still use the IBM AI Governance solution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*87lrie5pm1BO7PdkTzxpoQ.png" /></figure><h3><strong>Implement responsible AI in business processes by utilizing fair models</strong></h3><p>One of the important pillars of responsible AI is fairness and eliminating bias when you are dealing with people&#39;s attributes. Whether you are recruiting people as part of the HR team or giving loans within the banking industry this is applicable to everyone. Making sure that your AI systems are not making favorable outcomes for one group over another is critical for your business outcomes and the reputation of the brand.</p><p><strong>Fairness evaluation for direct and indirect bias</strong></p><p>You can configure fairness monitoring for protected attributes like age, sex, and race to ensure fair outcomes among different groups. Evaluation results are available on a single, easy-to-read dashboard. You can set up a fairness threshold and receive alerts if fairness drops below acceptable value.</p><p>If you don’t have typically protected attributes in your data, the monitoring service will look for indirect bias by finding associated values in the training features.</p><p><strong>De-bias Techniques</strong></p><p>In case of bias is detected, you can choose to de-bias your model actively or passively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LGnW4Qz8mgv1Ta3JVzI-fg.png" /></figure><h3><strong>Accelerate business outcomes by adopting transparent and explainable AI</strong></h3><p>Explainability plays a crucial role in building trust in AI systems which leads to the adoption of AI with confidence. It is important to understand how AI is making decisions that are impacting critical business outcomes. If you belong to a regulated industry like banking then you need to provide an explanation for your decisions example reasons for a declining a loan to an individual.</p><p><strong>Global and Local Explainability</strong></p><p>IBM AI Governance solution can help you understand factors influencing your model’s decision on a local (specific model transaction level) and global level (overall impact on model outcomes).</p><p>You get access to explainability algorithms that are enterprise ready with proprietary enhancements from IBM Research to generate accurate, resilient, and cost-effective explanations.</p><p><strong>What if Analysis</strong></p><p>As a line of business user, you can do an analysis to understand how different values can change the outcome of the model without writing a single line of code.</p><h3><strong>AI regulations enforcement with custom and auditable model workflow management</strong></h3><p>AI adoption is growing and so are regulations around it. The question is are we ready to deal with this frequently changing and growing world of AI regulations and frameworks?</p><p><strong>Custom Governance, Risk, and Compliance</strong> (<strong>GRC</strong>) <strong>workflows</strong></p><p>IBM AI Governance solution is here to help you in this journey. You can create highly customizable GRC workflow as per current regulations and your enterprise requirements. It is easy to add new checkpoints and branches in the existing workflow to enforce new regulations. This allows you to standardize AI processes across your organization with the flexibility to accommodate new changes in the world of AI.</p><p>If this is exciting, try a demo or speak to an IBMer <a href="https://www.ibm.com/products/cloud-pak-for-data/ai-governance">here</a>.</p><p>Learn more about IBM AI Governance</p><ul><li>Part 1: <a href="https://medium.com/ibm-data-ai/the-need-for-ai-governance-8d88f2f3b941">The Need for AI Governance</a></li><li>Part 2: <a href="https://medium.com/ibm-data-ai/3-key-areas-of-ai-governance-8676f1acf0f2">3 Key Areas of AI Governance</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c0f6dcde1ef" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ibm-data-ai/top-4-applications-of-ai-governance-3c0f6dcde1ef">Top 4 Applications of AI Governance</a> was originally published in <a href="https://medium.com/ibm-data-ai">IBM Data Science in Practice</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What’s new in Watson Studio on IBM Cloud Pak for Data 4.6]]></title>
            <link>https://medium.com/@snehalgawas/whats-new-in-watson-studio-on-ibm-cloud-pak-for-data-4-6-de69f56f453b?source=rss-c3235dc190fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/de69f56f453b</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai-governance]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ibm-data-and-ai]]></category>
            <dc:creator><![CDATA[Snehal Gawas]]></dc:creator>
            <pubDate>Wed, 30 Nov 2022 22:02:40 GMT</pubDate>
            <atom:updated>2022-11-30T22:02:40.511Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://www.ibm.com/cloud/watson-studio">IBM Watson Studio </a>empowers customers to build, run and manage AI models, and optimize decisions on <a href="https://www.ibm.com/products/cloud-pak-for-data">IBM Cloud Pak® for Data</a> platform.</p><p>We are happy to announce new Watson Studio features on IBM Cloud Pak for Data 4.6</p><h4>IBM Watson Pipelines</h4><p><a href="https://medium.com/ibm-data-ai/introducing-ibm-watson-studio-pipelines-beta-on-cloud-pak-for-data-4-5-eda0325b4429">IBM Watson Pipelines</a> are generally available on IBM Cloud Pak for Data v4.6. We have added a lot of new features to support your complex orchestration use cases. Here are some of them.</p><ul><li><strong>Nested Pipelines: </strong>Invoke<strong> </strong>one Pipeline into another. This is an important feature to keep your workflow modular and reuse components.</li><li><strong>Pipeline custom components: </strong>IBM Watson Pipelines provides many ready-to-use out-of-the-box features. Now you can bring your custom logic into Pipeline via python scripts. These reusable components are easy to create and use within multiple pipelines as nodes.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h3dEf5O3u_hvJRVqtmvuJw.png" /><figcaption>Configure Pipelines node for selected custom component</figcaption></figure><h4>Natural Language Processing</h4><p>Watson Natural Language Processing library is available with python runtimes for your language processing tasks such as sentiment analysis, keyword extraction, text classification, etc.</p><p>You can create your own models or take advantage of pre-trained models available in over 20 languages. These models are curated by a dedicated team of experts at IBM and evaluated for quality in each specific language. Learn more about this feature with the help of <a href="https://github.com/IBMDataScience/sample-notebooks/tree/master/CloudPakForData/notebooks/4.6">sample notebooks</a>.</p><h4>Watson Studio extension for Visual Studio Code</h4><p>Watson Studio extension allows you to securely connect your Cloud Pak for Data cluster with VS code editor. You can now use VS Code editor on your desktop and debug your code remotely in a runtime environment on Cloud Pak for Data. This feature enables you to use your VS Code IDE while taking advantage of computing resources and secure, managed access to data<strong> </strong>available on the Cloud Pak for Data.</p><h4>Training and Deployments</h4><ul><li><strong>AutoAI</strong>: Accelerate your AI lifecycle by automating model building process. Now you can train an AutoAI experiment with a large data set up to 100 GB.</li><li><strong>Customization:</strong> Watson Machine Learning conda and pip environments can be customized to bring additional packages from private repositories.</li><li><strong>Accelerated Computing: </strong>Watson Machine Learning Accelerator now supports A100 Multi-Instance GPU (MIG) to train and deploy deep learning models.</li></ul><h4>Model Monitoring</h4><p>To get the most value out of your AI investments, it is essential to be able to deploy models in production with trust and confidence.</p><p><strong>Fairness</strong>: IBM Watson OpenScale supports 8 new fairness metrics at runtime to evaluate your model for biases and ensure fair outcomes among different groups.</p><p><strong>Explainability</strong>: Generate global explanations as part of your model evaluation to understand which features contributed to the model’s predicted outcome.</p><p>You can monitor your models for these metrics via a simple user interface without coding.</p><h4>AI Factsheets</h4><p>AI Factsheets enable <a href="https://www.ibm.com/products/cloud-pak-for-data/ai-governance">AI Governance</a> for enterprises by allowing them to easily document key AI technology characteristics<strong> </strong>to facilitate subsequent enterprise validation. This means your Data Science teams can now focus on ML tasks rather than manual documentation and you still get a singular view<strong> </strong>of accurate, always up-to-date<strong> </strong>model facts across the model lifecycle.</p><p><strong>Modularity:</strong> In this release, we have added enhancements to make the AI factsheet modular. Based on your use case you can decide to use factsheets with IBM Watson Studio (track model entries without catalog) or IBM Watson Knowledge Catalog (track model entries with catalog).</p><p><strong>Ease of Sharing:</strong> Factsheets can be exported in pdf/docx/html format to further share with stakeholders.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*kSbk47eNgJjrrrlwF6oZuw.png" /></figure><h4>SPSS Modeler</h4><p>You can now use Python for scripting in the Extension nodes and invoke Python APIs from your scripts to interact with SPSS Modeler.</p><h4>Frameworks Upgrades</h4><p>Watson Studio provides the best of both worlds Open Source and IBM technologies. As part of this release, we have added support for Python 3.10, R 4.2, and Spark 3.3 versions with several updates on the library level. These updates make it easy for users to keep up with of latest and greatest while keeping their machine learning workflows secure.</p><h4>Resources</h4><ul><li>IBM Cloud Pak for Data <a href="https://www.ibm.com/docs/en/cloud-paks/cp-data">Documentation</a></li><li>Learn more about <a href="https://www.ibm.com/products/cloud-pak-for-data/ai-governance">AI Governance</a></li></ul><p>Want to learn more? <a href="https://www.ibm.com/products/cloud-pak-for-data/ai-governance?schedulerform=">Schedule a session</a> with an IBM expert today!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de69f56f453b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Threads: Introducing IBM Watson Studio Pipelines Beta on Cloud Pak for Data 4.5]]></title>
            <link>https://medium.com/ibm-data-ai/introducing-ibm-watson-studio-pipelines-beta-on-cloud-pak-for-data-4-5-eda0325b4429?source=rss-c3235dc190fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/eda0325b4429</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ibm]]></category>
            <category><![CDATA[watson-studio]]></category>
            <dc:creator><![CDATA[Snehal Gawas]]></dc:creator>
            <pubDate>Fri, 08 Jul 2022 20:44:57 GMT</pubDate>
            <atom:updated>2022-07-08T20:44:57.682Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WcpuSfHMfEL3pGGAK7nlHQ.png" /><figcaption>Orchestration of ML workflows with Watson Studio Pipelines</figcaption></figure><p>There’s been increased adoption of machine learning as more enterprises see it as a critical part of making data-driven business decisions. As more models move from prototyping to production, creating automated and governed machine learning workflows remains a big pain point.</p><p>A <a href="https://www.ibm.com/analytics/data-fabric">data fabric</a> architecture addresses this issue by helping you operationalize models and integrate data governance for trusted outcomes using <a href="https://www.ibm.com/products/cloud-pak-for-data/scale-trustworthy-ai">MLOps and Trustworthy AI</a>.</p><p>This solution is composed of three key pillars:</p><ul><li>Data: Quality data that is private, self-served, and ready for analysis by multiple personas</li><li>Model: MLOps infused with fairness, explainability, and robustness</li><li>Process: Automation to drive consistency, efficiency, and transparency for AI at scale.</li></ul><p>In the recently announced beta release of IBM Watson Studio Pipelines on Cloud Pak for Data 4.5, you can use IBM Watson Studio Pipelines to enable end-to-end orchestration of machine learning workflows in a consistent, repeatable, and automated way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9vkTvaxPXLWRRwzMfFWA2g.png" /></figure><h3>Benefits of IBM Watson Studio Pipelines</h3><ol><li><strong>Easy end-to-end orchestration</strong></li></ol><p>Machine learning is a team sport. This requires effective collaboration between different users with different levels of skills. <a href="https://www.ibm.com/cloud/watson-studio">Watson Studio</a> provides state-of-the-art capabilities to address all your data science needs. With the help of Watson Studio Pipelines, you can graphically orchestrate your end-to-end machine learning workflows.</p><p><strong>2. Fast experimentation</strong></p><p>Boost your team’s productivity by reducing experimentation time, so that you can deploy the best solution quickly.</p><p><strong>3. Automated machine learning workflows</strong></p><p>Operationalizing models manually requires significant effort to frequently monitor, compare, retrain, and deploy your models. Automate your workflows by creating consistent, repeatable flows with Watson Studio Pipelines.</p><h3>Watson Studio Pipelines Features</h3><p>Watson Studio Pipelines is built off of <a href="https://developer.ibm.com/blogs/kubeflow-pipelines-and-tekton-advances-data-workloads/">Kubeflow pipelines on Tekton</a> runtime and is fully integrated into the <a href="https://www.ibm.com/products/cloud-pak-for-data">IBM Cloud Pak for Data</a> platform. Users can leverage different tools on the platform and combine them to orchestrate their workflow with Watson Studio Pipelines including</p><ul><li>Jupyter Notebooks</li><li>Data Refinery flows</li><li>AutoAI Experiment</li><li>DataStage flows</li><li>Batch and online deployments</li></ul><p>Watson Studio Pipelines provides 30+ different out-of-the-box built-in components which you can drag and drop on the pipeline canvas to build your flow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WzzXlEvBmCvaRxFpfUu6Lw.png" /></figure><p>Once you are done orchestrating your flow, you can run the pipeline on-demand or schedule pipelines for future execution. You can also configure frequency(monthly, weekly, and so on) for your pipeline execution.</p><p>Visualize pipeline execution progress with the ability to inspect each node for logs, input, and outputs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zWIGRrP_wrSKZkTS7Jwl7A.png" /></figure><p>Watson Studio Pipelines gives you different options and flexibility to effectively orchestrate your Machine Learning flows end-to-end, including features like conditional execution, parallel and sequential execution with loops, an error handling mechanism, and the ability to trigger notifications.</p><h3>Join the beta</h3><ul><li>Click <a href="https://dataplatform.cloud.ibm.com/registration/stepone?context=cpdaas&amp;apps=data_science_experience%2Cwatson_machine_learning%2Ccos">here</a> to join Watson Studio Pipelines beta on Cloud Pak for Data as a Service.</li><li>If you have Cloud Pak for Data 4.5 installed, click <a href="https://epwt-www.mybluemix.net/software/support/trial/cst/welcomepage.wss?siteId=1691&amp;tabId=4641&amp;w=1">here</a> to try IBM Watson Studio Pipelines beta feature.</li></ul><p>Give us your feedback to help us improve our products by signing up for our research studies <a href="https://ibm.biz/DSResearch">here</a>.</p><h3>Resources</h3><ul><li><a href="https://epwt-www.mybluemix.net/software/support/trial/cst/welcomepage.wss?siteId=1691&amp;tabId=4641&amp;w=1">Access Watson Studio Pipelines beta release</a></li><li><a href="https://ibm.box.com/shared/static/ojveqcvz5i8q22j6kn32h7l81eug2m3i.mov">Watson Studio Pipelines Demo</a></li><li><a href="https://www.ibm.com/docs/SSQNUZ_4.5.x/wsj/analyze-data/ml-orchestration-overview.html">Learn more about Watson Studio Pipelines</a></li></ul><blockquote>A data fabric is an architectural approach to simplify data access in an organization to facilitate self-service data consumption. IBM Cloud Pak for Data provides a <a href="https://www.ibm.com/analytics/data-fabric">data fabric solution</a> for faster, trusted AI outcomes by connecting the right data, at the right time, to the right people, from anywhere it’s needed. This blog is related to data fabric use case <a href="https://www.ibm.com/products/cloud-pak-for-data/scale-trustworthy-ai">MLOps and Trustworthy AI</a>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eda0325b4429" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ibm-data-ai/introducing-ibm-watson-studio-pipelines-beta-on-cloud-pak-for-data-4-5-eda0325b4429">Data Threads: Introducing IBM Watson Studio Pipelines Beta on Cloud Pak for Data 4.5</a> was originally published in <a href="https://medium.com/ibm-data-ai">IBM Data Science in Practice</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing IBM Watson Studio and Watson Machine Learning 2.1]]></title>
            <link>https://medium.com/@snehalgawas/introducing-ibm-watson-studio-and-watson-machine-learning-2-1-3f897733802d?source=rss-c3235dc190fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f897733802d</guid>
            <category><![CDATA[automl]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[cloud-pak-for-data]]></category>
            <category><![CDATA[announcements]]></category>
            <category><![CDATA[watson-studio]]></category>
            <dc:creator><![CDATA[Snehal Gawas]]></dc:creator>
            <pubDate>Fri, 20 Dec 2019 20:05:44 GMT</pubDate>
            <atom:updated>2019-12-20T20:05:44.544Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://www.ibm.com/cloud/watson-studio">IBM Watson Studio</a> along with <a href="https://www.ibm.com/cloud/machine-learning">IBM Watson Machine Learning</a> empower organizations to tap into AI, inject predictions into business processes and optimize business value.</p><p>Today, we are pleased to announce the general availability of IBM Watson Studio and IBM Watson Machine Learning 2.1. These same Watson Studio features are also available on <a href="https://www.ibm.com/products/cloud-pak-for-data">Cloud Pak for Data</a> 2.5.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M7ZpyTlebtyzMhEacJbY4Q.png" /><figcaption>IBM Watson Studio and IBM Watson Machine Learning 2.1</figcaption></figure><h3>What’s New in Watson Studio 2.1</h3><p>Watson Studio 2.1 is the next step towards automating your end to end AI workflow. We understand automating data science is tough. There is no one size fits all approach; different enterprises seek to combine varying degrees of automation in order to accelerate the time to value. Watson Studio provides you these different options to accelerate your AI workflows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*foyPsIuSWXNdIlCUZZUOWg.png" /><figcaption>Data Science Automation-Productivity Spectrum</figcaption></figure><h3>AutoAI</h3><p>Take advantage of <a href="https://www.ibm.com/cloud/watson-studio/autoai">AutoAI</a> to automate and optimize end to end AI pipelines for tasks like data preparation, feature engineering, and hyperparameter optimization. AutoAI is part of Watson Studio, integrated with Watson Machine Learning add-on.</p><p><strong>Benefits of AutoAI</strong></p><ul><li>Get started with a couple of clicks without coding.</li><li>Extract more predictive power from your data with auto-feature engineering.</li><li>Easily compare candidate pipelines to find the best model for the job.</li><li>One-click deployment to IBM Watson Machine Learning.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7_kxCPf6pqExRx9ygEXAxg.png" /><figcaption>AutoAI Experiment</figcaption></figure><h3>SPSS Modeler — Text Analytics</h3><p>With Text Analytics in SPSS Modeler add-on, generate insights from unstructured data in minutes without having to code.</p><p><strong>Text Analytics feature in SPSS Modeler provides</strong></p><ul><li>Set of linguistic resources, such as dictionaries for terms and synonyms, libraries, and templates. You can further refine these linguistic resources to your context.</li><li>Interactive workbench includes categories, conceptions, and text link analysis.</li><li>Includes built-in support for English, French, German, Spanish, Dutch, Italian, and Portuguese.</li><li>Backward compatible with all SPSS Modeler text analysis streams.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yghf3lPE-yZjwVA4VmNsMg.png" /><figcaption>Text Analytics Interactive Bench</figcaption></figure><p>Other than Text Analytics, SPSS Modeler is packed with many new features like new interactive charting node to exact insights quickly and more interactively, a SQL Pushback feature to push data preparation operations directly in your database and a migration wizard to easily migrate input and export nodes.</p><h3>JupyterLab</h3><p>Now, Watson Studio users can use JupyterLab IDE to work with Jupyter notebooks, text editors, and terminals side by side in a tabbed work area. JupyterLab provides a flexible and more powerful user interface. Launch JupyterLab IDE from the Watson Studio project to directly access data assets available in the project. Using a Git extension, users can push changes from JupyterLab to their Git repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*arhkGoPcHNmVKypctrsPZQ.png" /><figcaption>JupyterLab IDE in Watson Studio Project</figcaption></figure><h3>Git Integration</h3><p>Watson Studio user can integrate Git with a project and use this integration to export a project to a Git repository, to work with documents and notebooks in JupyterLab, or as a method to backup a project for source code management purposes. We support GitHub, GitHub Enterprise, GitLab and Bitbucket integrations at this point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FOhxY6QCv2IaCQyZXRpX4Q.png" /><figcaption>Sync Watson Studio Project with Git Repo</figcaption></figure><h3>Decision Optimization</h3><p><a href="https://www.ibm.com/us-en/marketplace/decision-optimization-for-watson-studio">Decision Optimization in Watson Studio</a> enables organizations to solve prescriptive problems to make better decisions and deliver improved business outcomes. In Watson Studio, you can build Decision Optimization models either with Jupyter notebooks or by using the powerful and easy to use model builder interface. You can solve multiple scenarios and deploy your model to Watson Machine Learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AT-_mFuki7qW88TGz4MsdA.png" /><figcaption>Save Decision Optimization Model for Deployment</figcaption></figure><h3>Online and Batch deployments</h3><p>IBM Watson Machine Learning 2.1 accelerates the process of moving to deployment and integrating AI into business applications. Watson Studio along with Watson Machine Learning offers a single interface to manage the entire analytics lifecycle, from discovery to production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Eno2LL2Ev0UBCJX0o44i3w.png" /><figcaption>Deploy Model to Watson Machine Learning</figcaption></figure><p>Once the model is ready, a user can save their model to a <em>deployment space</em> to deploy the model and manage deployment. Based on the model framework, the user can deploy models in three ways: <em>Online, Batch </em>or<em> Virtual</em>.</p><p>IBM Watson Machine Learning supports different IBM and Open Source frameworks for model deployment :</p><ul><li>SPSS Modeler, Decision Optimization</li><li>Spark</li><li>Scikit-learn &amp; XGBoost</li><li>PMML</li><li>Tensorflow, PyTorch, Keras, Caffe</li><li>Python functions</li></ul><h3>Learn More</h3><p>Want to learn more? Schedule a free <a href="https://www.ibm.com/analytics/common/scheduler/expert-advice/?focusArea=Cloud%20-%20Hybrid%20-%20Managed%20Services&amp;amp;languageCode=en&amp;amp;regionCode=NA%20NORTH%20AMERICA%20IOT&amp;languageCode=en&amp;regionCode=NA%20NORTH%20AMERICA%20IOT">session</a> with an IBM expert today!</p><p><strong>Additional Resources</strong></p><ul><li>Watson Studio and Watson Machine Learning <a href="https://www.ibm.com/support/knowledgecenter/SSHGWL_2.1.0/">documentation</a></li><li><a href="https://medium.com/ibm-watson/from-data-to-predictions-to-actions-with-watson-studio-in-cpd-2-5-b426bbeef5fe">Learn</a> how can decisions and actions in your business be optimized <br>by intelligent predictions based on trusted data</li><li><a href="https://medium.com/ibm-watson/introducing-autoai-for-watson-studio-241675fb0454">AutoAI</a></li><li><a href="https://medium.com/ibm-watson/decision-optimization-autoai-perfect-blend-available-in-ibm-cloud-pak-for-data-95ec0ba0aed7">Combining Decision Optimization and Intelligent Automation in a Unified Environment</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f897733802d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Announcing Data Refinery Spark Environments Beta]]></title>
            <link>https://medium.com/@snehalgawas/announcing-data-refinery-spark-environments-beta-2428d4352a9c?source=rss-c3235dc190fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/2428d4352a9c</guid>
            <category><![CDATA[announcements]]></category>
            <category><![CDATA[data-refinery]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ibm]]></category>
            <category><![CDATA[watson-studio]]></category>
            <dc:creator><![CDATA[Snehal Gawas]]></dc:creator>
            <pubDate>Sun, 10 Feb 2019 23:11:07 GMT</pubDate>
            <atom:updated>2019-02-10T23:11:07.357Z</atom:updated>
            <content:encoded><![CDATA[<p>Today we are excited to announce that <strong>Spark environments are available in beta for Data Refinery Flows </strong>in <a href="https://www.ibm.com/cloud/watson-studio">Watson Studio</a> &amp; <a href="https://www.ibm.com/cloud/watson-knowledge-catalog">Watson Knowledge Catalog</a>.</p><p><a href="https://www.ibm.com/cloud/data-refinery">Data Refinery</a> is a self-service data preparation tool available in Watson Studio and Watson Knowledge Catalog that provides data preparation and visualization capabilities. Data Refinery enables quick transformation of large amounts of raw data into consumable and quality information that’s ready for analytics. Currently, Data Refinery uses a default non-customizable Spark cluster to run all data flows.</p><h3>What is Spark Environment?</h3><p>Now, Spark environments users will get flexible and customizable options to run data flows based on the size of the data. Each Spark environment consists of one SparkR kernel as a service. The kernel has a dedicated Spark cluster and Spark executors. When you select to run a data flow in a Spark environment, it runs in its own cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*OHAq-F12ZmGQ_hq-qH1vYg.png" /></figure><h3>Get started</h3><ol><li>Access Data Refinery from within a project. Click Add to project and then choose Data Refinery flow. Then <a href="https://dataplatform.cloud.ibm.com/docs/content/refinery/asset_browser.html">select the data</a> that you want to work with.</li><li>Apply operations to cleanse, shape and enrich your data</li><li><a href="https://dataplatform.cloud.ibm.com/docs/content/refinery/metrics.html">Validate your data</a> throughout the data refinement process.</li><li><a href="https://dataplatform.cloud.ibm.com/docs/content/refinery/visualizations.html">Use visualizations</a> to gain insights into your data and to uncover patterns, trends and correlations within your data.</li><li>When you’ve refined the sample, data set to suit your needs, select Spark environment -<em>Default Spark R 3.4(Beta)</em> from drop down. Click the <strong>Save and</strong> <strong>Run flow</strong> button in the toolbar to run the Data Refinery flow on the entire data set.</li></ol><p>During beta phase, users will have the option to select between default or custom Spark<em> </em>R (Beta) environment and <em>Data Refinery default </em>runtime.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aU7jS6n40vbJuLQLjNqHTQ.png" /><figcaption>Save and Run Flow</figcaption></figure><p>Users can create their own custom Spark environments using <strong>Environments</strong> tab in the project. Those custom environments will be available for use in this environment drop-down list.</p><h3>Create Custom Environments</h3><ol><li>To create a custom environment, go to <strong>Environments </strong>tab in the project and click New environment definition.</li><li>Enter a name and a description. Select the environment type Spark to see the environment runtime configuration options and choose the hardware and software configuration.</li><li>Select the software version R 3.4.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kISwxi0RtLcHvXVvGOlWYQ.png" /><figcaption>Define Custom Environment</figcaption></figure><p>The current Data Refinery default runtime will continue to consume 6 capacity unit per hour. Capacity units required for a Spark environment is calculated based on the hardware configuration, size of the driver, the number of executors and their size. <a href="https://dataplatform.cloud.ibm.com/docs/content/analyze-data/track-runtime-usage.html?context=analytics#spark">Learn more</a> about capacity unit calculation for Spark environment.</p><p>These Spark Environments will make it easier for Data Refinery users to easily scale their compute based on data workloads.</p><p>For additional information on Data Refinery Spark Environments, please visit our <a href="https://dataplatform.cloud.ibm.com/docs/content/analyze-data/spark-dr-envs.html">official documentation</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2428d4352a9c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Watson Studio Desktop is now Generally Available]]></title>
            <link>https://medium.com/@snehalgawas/watson-studio-desktop-is-now-generally-available-750c1d6a28aa?source=rss-c3235dc190fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/750c1d6a28aa</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ibm-watson]]></category>
            <category><![CDATA[ibm]]></category>
            <category><![CDATA[announcements]]></category>
            <dc:creator><![CDATA[Snehal Gawas]]></dc:creator>
            <pubDate>Tue, 11 Dec 2018 17:32:14 GMT</pubDate>
            <atom:updated>2018-12-11T20:00:16.843Z</atom:updated>
            <content:encoded><![CDATA[<h3>Watson Studio Desktop is Now Generally Available!</h3><p>Today we are very excited to announce that Watson Studio Desktop is now Generally Available to try and to buy!</p><p><a href="https://www.ibm.com/account/reg/us-en/signup?formid=urx-19947&amp;cm_mmc=OSocial_Blog-_-Hybrid+Cloud_Data+Science-_-WW_WW-_-ga+watson+studio+desktop&amp;cm_mmca1=000000RE&amp;cm_mmca2=10000668"><strong>Watson Studio Desktop</strong></a> is a new offering within the Watson Studio family. It brings all the amazing features of the Watson Studio platform to your desktop for fast and productive use. Now, it’s really easy to keep your data on your desktop and to take advantage of powerful Watson Studio features while you are offline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*gHL5JRDwV3PNoZQldo-vIw.png" /><figcaption>Watson Studio features on your desktop</figcaption></figure><p><a href="https://medium.com/ibm-watson/introducing-ibm-watson-studio-e93638f0bb47">Watson Studio</a> offers a powerful and flexible environment to accelerate your machine and deep learning workflows and collaboration. You can access a broad range of data science tools for preparing and analyzing your data, building, training and deploying models.</p><p>You can deploy Watson Studio in a few ways:</p><ol><li><strong>Cloud</strong>: Watson Studio in a flexible, hybrid cloud environment</li><li><strong>Local</strong>: Watson Studio deployed behind the firewalls for enterprise deployment</li><li><strong>Desktop</strong>: Now with this new offering Watson Studio is also available on your Mac or PC</li></ol><h3>Features</h3><p>Watson Studio Desktop contains many of the most popular tools and features from the Watson Studio platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*70jNPBA9fO7XlEx2Jg48iQ.png" /><figcaption>SPSS Modeler</figcaption></figure><p>Here are the initial main features of Watson Studio Desktop</p><ul><li><strong>Projects</strong>: Easily organize and manage your resources and data.</li><li><strong>SPSS Modeler</strong>: Take advantage of this powerful and flexible drag and drop visual interface for data exploration, preparation, feature engineering, and modeling. No coding is required.</li><li><strong>Data Refinery</strong>: Quickly transform your data into consumable quality information that’s ready for analytics.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rXlZJZVvRnP1ET4X052hIQ.png" /><figcaption>Profiling in Data Refinery</figcaption></figure><p>This is just the beginning. We are working on adding more cool features in Waston Studio Desktop. Here’s a glimpse of some of the upcoming features.</p><ul><li>Python 3 Jupyter Notebooks</li><li>Watson Studio Cloud deployments</li></ul><h3>Benefits</h3><ul><li>Increase productivity with an easy and fast locally installed client.</li><li>Work offline. No Wi-Fi connection is required.</li><li>All data and models remain on your own machine so you don’t have to worry about security and governance policies.</li><li>Build models iteratively without any overage cost. When you’re ready, move them to the cloud.</li></ul><p>Are you ready to start your journey with Watson Studio Desktop? Here is a quick introductory <a href="https://youtu.be/7NFB_Hzwtzc">demo</a> video for you.</p><p>Take advantage of 30 day trial period (no card is required) to get familiar with Watson Studio Desktop.</p><blockquote><strong>Start your 30 day </strong><a href="https://www.ibm.com/account/reg/us-en/signup?formid=urx-19947&amp;cm_mmc=OSocial_Blog-_-Hybrid+Cloud_Data+Science-_-WW_WW-_-ga+watson+studio+desktop&amp;cm_mmca1=000000RE&amp;cm_mmca2=10000668"><strong>free trial</strong></a> <strong>today.</strong></blockquote><p>We would love to meet you in person. Come visit us at <a href="https://www.ibm.com/events/think/register/?cm_mmc=OSocial_Blog-_-Hybrid+Cloud_Data+Science-_-WW_WW-_-general+availability+of+Watson+Studio+Desktop&amp;cm_mmca1=000000RE&amp;cm_mmca2=10007925&amp;cm_mmca3=M00000000">IBM Think</a> in San Francisco for <a href="https://myibm.ibm.com/events/think/all-sessions/session/6962A?cm_mmc=OSocial_Blog-_-Hybrid+Cloud_Data+Science-_-WW_WW-_-watson+studio+desktop+think&amp;cm_mmca1=000000RE&amp;cm_mmca2=10000668">Watson Studio Desktop session</a> on Thursday February 14th, 2019.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=750c1d6a28aa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Control your DSX projects using Python]]></title>
            <link>https://medium.com/@snehalgawas/control-your-dsx-projects-using-python-c69e13880312?source=rss-c3235dc190fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/c69e13880312</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[object-storage]]></category>
            <category><![CDATA[ibm]]></category>
            <category><![CDATA[dsx]]></category>
            <dc:creator><![CDATA[Snehal Gawas]]></dc:creator>
            <pubDate>Tue, 27 Feb 2018 20:54:31 GMT</pubDate>
            <atom:updated>2018-02-27T20:55:07.324Z</atom:updated>
            <content:encoded><![CDATA[<p>In DSX, we use project to organize resources like data, notebooks, models &amp; connections. To easily interact with these assets now we have <strong>project-lib</strong> along with object storage APIs. Project-lib is programmatic interface to interact with your data stored in object storage. It allows you to easily access all your project assets including files, connections and metadata.</p><p>In this blog, we will explore this project-lib library in a python notebook.</p><p><strong>Set-Up Your Project</strong></p><p>Project-lib is pre-installed on DSX and can be imported in notebook through simple steps:</p><ol><li>Click on <strong>more (</strong>three dots) option from top right hand side panel and select <strong>insert project token</strong>.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FT9iI5Bwt9SXpJZRccosMw.png" /></figure><p>2. If this is the first time you are using a token in this project, you will receive below message.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mCV8nuKcMu7GDVjrM00gSg.png" /></figure><p>Click on <strong>project settings</strong> and create token by clicking <strong>new token</strong> option from access tokens sections and again repeat step 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*h_KUZrNbuWR9zGqY-yghuA.gif" /></figure><p>Once you insert project token, below code snippet will get added in the first cell of your notebook. This code snippet contains spark context, project id and token respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V4-60eoLUju8bT_hJwbxJA.png" /></figure><blockquote>If you are using notebook with environments then you don’t have spark context in your notebook. In this case, you will have to replace sc with None in the above code snippet.</blockquote><pre>#The project token is an authorization token that is used to access project resources like data sources, connections, and used by platform APIs.<br>from project_lib import Project<br>project = Project(None, &#39;**************&#39;, &#39;**************&#39;)<br>pc = project.project_context</pre><p><strong>Get Project Details</strong></p><p>You can fetch project details like project name, description and bucket name programmatically using different methods from project-lib.</p><p>You can use <em>get_assets</em> method to fetch list of all assets with their names and ids. Filter these assets based on asset type into data or connections.</p><p>You can also list files in your project using function <em>get_files</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sQSeLPeli1YLnmCaw0LL4A.png" /></figure><p><strong>Upload Data to Object Storage</strong></p><p>You can use <em>save_data</em> method to upload data to object storage. It takes 4 parameters.</p><ol><li><strong>file_name</strong> : Name of the file. Your data will get saved with this name in object storage.</li></ol><p>2.<strong> data</strong>: Data to upload. It should be file like object. Example — byte-buffers, string-buffers. Learn more about file like objects <a href="https://docs.python.org/3/glossary.html#term-file-object">here</a>.</p><p>3.<strong> set_project_asset</strong> : Add asset to project after successful upload (optional, default True). If this is set to False, file will only get added to the COS bucket and not to the project.</p><p><strong>4. overwrite</strong>: Overwrite the object if already exists. (optional, default False)</p><p>Suppose you are working on pandas data frame and you want to save that data frame as csv file in object storage, you can use the code below to save this data:</p><pre><em># Save dataframe as csv file to storage</em><br>project.save_data(data=df.to_csv(index=<strong>False</strong>),file_name=&#39;iris1.csv&#39;,overwrite=<strong>True</strong>)</pre><p><strong>Get Data from Object Storage</strong></p><p>To get data from object storage use method <em>get_file </em>and provide name of the file:</p><pre><em>#Getting csv file and loading as pandas datafarme</em><br>pd.read_csv(project.get_file(&#39;iris1.csv&#39;)).head(5)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cVus1vRdMisdIJc55jo2Fg.png" /></figure><p><strong>Upload Pickle File to Object Storage</strong></p><p>While working with machine learning models we can save model as pickle object in object storage.</p><pre># save pickle<br>project.save_data(data=pickle.dumps(clf),file_name=&#39;RF.pkl&#39;,overwrite=True)</pre><pre># Download and load model<br>pickle.load(project.get_file(&#39;RF.pkl&#39;))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KyiomQtw8qWdFZd0QJ046Q.png" /></figure><p><strong>Bring Python Scripts into your DSX Environment</strong></p><p>Suppose you have folder with python scripts on your local machine and you want to bring these scripts into DSX.</p><p>First zip your folder with all the files. Upload that zip file to the project using find and add data option on top panel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fbhQM_4zGZWHohvMbJs1WQ.png" /></figure><p>Then use below function to get all those files in the notebook.</p><pre>import zipfile<br>def get_zip(file_name):<br>    &#39;&#39;&#39;<br>    file_name = Name of zip file you want to download from object storage<br>    &#39;&#39;&#39;<br>    try:<br>        fobj = open(file_name, &quot;wb&quot;)<br>        fobj.write(project.get_file(file_name).read()) <br>        fobj.close()<br>        z = zipfile.ZipFile(file_name)<br>        z.extractall()<br>    except Exception as e:<br>        print(Exception,e)<br>    else:<br>        print(&#39;Files downloaded successfully&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ejGWsbBUYJxKLGQL-eyL-g.png" /></figure><p>You can use magic commands to load and run these python scripts or you can import function from python script. In below code snippet, we are importing function <em>prim_numbers </em>from<em> </em>cal_prime.py script.</p><pre># Load script <br>%load Python\ Scripts/cal_prime.py</pre><pre>#run script<br>%run Python\ Scripts/cal_prime.py</pre><pre># Import function from python script<br>import cal_prime<br>cal_prime.prim_numbers()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LF233pD2Z4xfbfM5QHHPPQ.png" /></figure><p><strong>Create Zip File from Files in Object Storage</strong></p><p>You have images stored in your object storage. You want to download these images and zip them together and again upload that zip file to object storage.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/94e433528b6e4d1c3ee2282e0c638148/href">https://medium.com/media/94e433528b6e4d1c3ee2282e0c638148/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LLhYA9qC6sXzoXdEXBo2IA.png" /></figure><p><strong>Get Connections:</strong></p><p>You can use <em>get_connections</em> method to list all the available connections in the project. To get connection use <em>get_connection</em> method and pass either connection id or name of the connection. It will return credentials required for the connection.</p><p>Here is an example of how to read data from connection. I have DB2 connection in project and I want to read one table from this database into python data frame.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0hvUGTd2PnHlBL0UdgkQ0g.png" /></figure><p><strong>Get Data Using Spark</strong></p><p>If you are using a notebook running on Spark service and want to get data into Spark data frame then <em>get_file_url</em> will help you fetch a file from the object storage using Spark.</p><pre>from pyspark.sql import SparkSession</pre><pre># <a href="http://twitter.com/hidden_cell">@hidden_cell</a><br># The project token is an authorization token that is used to access project resources like data sources, connections, and used by platform APIs.</pre><pre>from project_lib import Project<br>project = Project(sc, &#39;******************&#39;, &#39;******************&#39;)<br>pc = project.project_context</pre><pre># Get file url using file name<br>url=project.get_file_url(https://medium.com/&#39;iris.csv&#39;)</pre><pre># get data into spark dataframe<br>spark = SparkSession.builder.getOrCreate()<br>df = spark.read.format(&#39;org.apache.spark.sql.execution.datasources.csv.CSVFileFormat&#39;)\<br>  .option(&#39;header&#39;, &#39;true&#39;)\<br>  .load(url)</pre><pre>df.show(5)</pre><p>Now you can easily access all your project assets with the help of project-lib. For more details on project-lib refer this <a href="https://datascience.ibm.com/docs/content/analyze-data/publish-library-documentation.html?audience=cloud">documentation</a> and for python code refer this <a href="https://dataplatform.ibm.com/analytics/notebooks/v2/a972effc-394f-4825-af91-874cb165dcfc/view?access_token=ee2bd90bee679afc278cdb23453946a3922c454a6a7037e4bd3c4b0f90eb0924">notebook</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c69e13880312" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predict Chronic Kidney Disease Using SPSS Modeler Flows]]></title>
            <link>https://medium.com/@snehalgawas/predict-chronic-kidney-disease-using-spss-modeler-flows-cf82a16835a8?source=rss-c3235dc190fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/cf82a16835a8</guid>
            <category><![CDATA[spss]]></category>
            <category><![CDATA[ibm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Snehal Gawas]]></dc:creator>
            <pubDate>Mon, 12 Feb 2018 16:22:54 GMT</pubDate>
            <atom:updated>2018-03-19T16:19:30.178Z</atom:updated>
            <content:encoded><![CDATA[<p>In this blog post we will try to predict chronic kidney disease using various attributes collected from hospitals. Chronic kidney disease (CKD) is a condition characterized by a gradual loss of kidney function over time, which may lead to kidney failure.</p><p><strong>Data Source</strong>:</p><p>We are using the UCI Chronic Kidney Disease data set from the Data Science Experience community. You can get data into your project in two steps:</p><ol><li>Go to the <a href="https://datascience.ibm.com/community">Data Science Experience community</a> . You can also navigate to the community from DSX by clicking the Community tab on the top panel.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zmnd3s3ptuKeXzI-xURg6Q.png" /></figure><p>2. Select the data set from the community and click the add (+) icon on the community card. Select your project and click <strong>Add</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*12ZoPBRGJz65WUUzjTjhQw.png" /></figure><p>After you add the data set to your project you can find it in the <strong>Data assets</strong> section under the <strong>Assets</strong> tab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uz0KBpdc0aSXMqQORUN62A.png" /><figcaption>List of data sets In a project</figcaption></figure><p><strong>Tools:</strong></p><p>Data Science Experience offers an array of options for working with your data. To model this problem and understand the factors affecting chronic kidney disease, we will use IBM SPSS Modeler Flow in DSX. IBM SPSS Modeler Flow is a graphical interface to create different machine learning flows.</p><p><strong>Create SPSS Modeler Flow</strong>:</p><p>To create an SPSS Modeler Flow, go to the <strong>Assets</strong> tab as shown above and click the new flow icon under the <strong>SPSS Modeler Flow</strong> section. Give a name and description to the flow and select the IBM SPSS Modeler runtime.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zQkwoiOWO-0LfN9hXNTkSQ.png" /></figure><p><strong>Nodes In IBM SPSS Modeler Flow:</strong></p><p>Before starting with the analysis, let’s have a look at different node options available in SPSS Modeler Flow.</p><p>On left side panel (Nodes Palette)<strong> </strong>you can see different types of nodes available for you to use while working on your data. There are six types of node categories:</p><ol><li><strong>Record Operations:</strong> As the name suggests, you can use them to perform operations such as selecting, appending, sorting on the record (row) level.</li><li><strong>Field Operations: </strong>These nodes are helpful in the data preparation phase. You can filter data, rename features, and choose the type of your attributes.</li><li><strong>Graphs: </strong>Nodes in this section will help you with basic data exploration and understanding distribution or relationship between features.</li><li><strong>Modeling: </strong>These nodes provide different modeling algorithms for different types of problems.</li><li><strong>Outputs: </strong>These nodes are helpful in<strong> </strong>understanding your data<strong> </strong>and model. You can display results in table format or get a report on evaluation parameters of your model.</li><li><strong>Export: </strong>After processing and modeling, this node will help you export data<strong> </strong>from the flow editor to your DSX project<strong>.</strong></li></ol><p>Drag and drop the node into the canvas and right-click to take further actions such as open, preview, or run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p1s7fjouFVgUhGzEoDMygg.png" /></figure><p><strong>Data Cleaning:</strong></p><p>To start working on the problem, first we need to get data into the canvas. It is as easy as drag-and-drop. To preview the file, right-click on a node and select <strong>Preview</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x8AdhFt3mzUC9JOQ7CrngQ.gif" /></figure><p>There are a few values missing from our data. Let’s dig deeper into summary statistics of our data using the Data Audit node. Drag the Data Audit node and connect it with the data node. We have to open the node to change settings or give a custom name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XDnTEyBK7xw5YS90i9TheQ.png" /></figure><p>After running the node you can see your audit report on right side panel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9PqAN9xofsQF0-gd3DuxnA.png" /></figure><p>In the audit report, we can see that few columns have missing value (?) as mode. Let’s drop those features using the Filter node, and then we will drop rows with missing values using the Select node. In this way, we can retain the maximum number of records.</p><p>Drop Columns</p><ol><li>To drop columns drag and drop Filter node on the canvas and connect it with data node.</li><li>Right click and open the node. Select below columns under filter section [bgr,sod,pot,hemo,pcv,wbcc &amp; rbcc] make sure to check box — Filter selected fields and hit save button.</li></ol><p>Drop Rows</p><ol><li>Drag and drop Select node on the canvas, connect it with Filter node, right click and open the node.</li><li>Select discard mode and provide below condition to remove rows with missing values.</li></ol><p>(sg = ‘?’ or al= ‘?’ or su= ‘?’ or rbc= ‘?’ or pc= ‘?’ or pcc= ‘?’ or ba= ‘?’ or bu= ‘?’ or appet= ‘?’ or pe= ‘?’ or ane= ‘?’)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*swfx3gsk3QF3NtF7sk24DQ.png" /></figure><p>If you decide to impute these missing values, the Filler node will help you do that.</p><p>Once our data is clean, we can set our class variable as the target variable using the Type node. It will help our model to distinguish between input and target features.</p><ol><li>Drag and drop the Type node on the canvas, connect it with Select node, right click and open the node.</li><li>Click on configure types. Select column name <em>class</em>, change role to target hit ok and then save.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uxblb4ErZtidD83XbCPT4w.png" /><figcaption>Set class feature as target</figcaption></figure><p>Let’s take a quick look at the distribution of our target variable. Drag the Distribution node from the graphs section of the node palette and provide field information under settings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_4Rk0WoZLSnpx66dLhPtSQ.png" /></figure><p>In our data we have more non-chronic kidney disease cases than chronic kidney disease cases.</p><p>One more step before building the classification model is to divide data into train and test sets. We will use the Partition node for this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jR4Fo68THJWmeFBF7Tf_-w.png" /></figure><p>Now let’s fit the classification model. We will be using a C5.0 algorithm to build a decision tree . A C5.0 model works by splitting the data based on the field that provides the maximum information gain.You can see node C5.0 under the Modeling section of the nodes palette.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*prYIQ9ZHIcopElAvrN0EUg.png" /></figure><p>While building this model we don’t have to specify input and output variables. We have already done that in the Type node. Once you run your decision tree model you will be able to see your model in a golden color node.</p><p>Right-click on the golden color node and view the model. You can see predictor importance, tree digram, and other model information here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hm7jZPgTzp98Au1zvxYkSw.png" /></figure><p>To evaluate the performance of the model, select the Analysis node from the Output section of the node palette and connect it with the model. Similarly, use the Table node to view data in a table format with predicted labels and confidence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZfNF7ONehQEIwvnT_-qcIg.png" /></figure><p>This is the analysis report for our model.We have achieved 97% accuracy on our test data set with this model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4rWfyV1Y34_72QBjjfYfsw.png" /></figure><p>Now it’s time to save our model. Right click on a terminal node in the flow (e.g. analysis/table nodes) , click on <strong>save as a model </strong>option and provide model name to save this SPSS model to our project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JTZx5JYq1OAORu5Gj4sFyA.png" /></figure><p>A great thing about SPSS Modeler Flow is that you can build different models within the same canvas.</p><p>Assume we don’t know if a person has chronic kidney disease or not. Let’s try to build an unsupervised model and see if we can identify pattern for chronic kidney disease.</p><p>We will choose the K-Means node for this experiment and select all features except class (target feature). Set cluster size to 2. We are expecting only two clusters here (chronic kidney disease &amp; non-chronic kidney disease).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JSBvdWTWWV0lhXSHC1qG-Q.png" /></figure><p>We can view model information by right-clicking the golden K-means node and selecting <strong>View model</strong> option.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g7cMM7Xlakoh9NorJUCI8g.png" /></figure><p>Finally, let’s evaluate clustering results with respect to our original target labels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gB3asQo4mAeebaWV6-Yzfg.png" /></figure><p>Here cluster-1 represents non-chronic kidney disease patients and cluster-2 belongs to patients who have this disease. Some people with chronic kidney disease are wrongly identified by the K- Means algorithm as non-chronic kidney disease patients.</p><p>Finally, to export your results use the Data Export node. Give a name to your file under the path section in the settings. Data will be exported to your project storage and you can see it under the <strong>Data Asset</strong> tab in the project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-JNLJMtD6BI4kGpdqlJiLQ.png" /></figure><p>You can choose to run all flows on the canvas by using the run icon on the top panel. You can also take advantage of shortcuts such as copy, cut, and paste. To download your flow, click on the download icon from upper right side panel. This will download a SPSS Modeler .str file that can be opened in SPSS Modeler.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cHgXJivaDXhGqxwvivQOdw.png" /></figure><p>You can work on different problems by experimenting with different algorithms and modeling techniques in SPSS Modeler Flow.</p><p>To get started, go to <a href="https://datascience.ibm.com/">Data Science Experience</a> and create your own SPSS Modeler flow!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf82a16835a8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Working With IBM Cloud Object Storage In Python]]></title>
            <link>https://medium.com/@snehalgawas/working-with-ibm-cloud-object-storage-in-python-fe0ba8667d5f?source=rss-c3235dc190fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe0ba8667d5f</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Snehal Gawas]]></dc:creator>
            <pubDate>Wed, 31 Jan 2018 19:01:00 GMT</pubDate>
            <atom:updated>2023-03-02T19:57:39.377Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>IBM Cloud Object Storage</strong></p><p>Working with <a href="https://www.ibm.com/cloud/watson-studio">IBM Watson Studio</a> comes with a flexible storage option of IBM Cloud Object Storage. When you create a project in Watson Studio, integration with Cloud Object Storage for storing project assets is automatically created for you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Wp2kkPQArt8IIB4Ii0MwQ.png" /></figure><p>This blog is focused on how to use IBM Cloud Object Storage in Python, but you can also easily load data in Watson Studio using UI.</p><p><strong>Import Credentials</strong></p><p>To access IBM Cloud Object Storage you need credentials. You can get these credentials using insert to codeoption in the notebook. To insert credentials you need to first upload some data to Watson Studio using browse functionality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*5hB2LYBPzF94F5lqmwyoZw.png" /><figcaption>Insert credentials in the notebook</figcaption></figure><pre># <a href="http://twitter.com/hidden_cell">@hidden_cell</a><br># The following code contains the credentials for a file in your IBM Cloud Object Storage.<br># You might want to remove those credentials before you share your notebook.<br>credentials = {<br>    &#39;IBM_API_KEY_ID&#39;: &#39;*******************************&#39;,<br>    &#39;IAM_SERVICE_ID&#39;: &#39;*******************************&#39;,<br>    &#39;ENDPOINT&#39;: &#39;*******************************&#39;,<br>    &#39;IBM_AUTH_ENDPOINT&#39;: &#39;*******************************&#39;,<br>    &#39;BUCKET&#39;: &#39;*******************************&#39;,<br>    &#39;FILE&#39;: &#39;*******************************&#39;<br>}</pre><p>ibm_boto3 library provides complete access to the IBM® Cloud Object Storage API. We need to create a low-level client using above credentials.</p><pre>from ibm_botocore.client import Config<br>import ibm_boto3</pre><pre>cos = ibm_boto3.client(service_name=&#39;s3&#39;,<br>    ibm_api_key_id=credentials[&#39;IBM_API_KEY_ID&#39;],<br>    ibm_service_instance_id=credentials[&#39;IAM_SERVICE_ID&#39;],<br>    ibm_auth_endpoint=credentials[&#39;IBM_AUTH_ENDPOINT&#39;],<br>    config=Config(signature_version=&#39;oauth&#39;),<br>    endpoint_url=credentials[&#39;ENDPOINT&#39;])</pre><p><strong>Upload Files</strong></p><p>To upload file in COS, we will be using upload_file function. It takes three parameters — local file name(along with path), bucket name and key. Key can be different from local file name. Your file will be identified by this name within the bucket. When you create a project with IBM Cloud Object Storage option, Watson Studio creates bucket for your project . You can find bucket corresponding to your project in credentials.</p><pre>cos.upload_file(Filename=&#39;wine/wine.csv&#39;,Bucket=credentials[&#39;BUCKET&#39;],Key=&#39;wine_data.csv&#39;)</pre><p>We can use this function for uploading zip-files or pickle objects. Here I have Gradient Boosting Classifier as pickle object.</p><pre>#Upload zip file<br>cos.upload_file(&#39;wine.gz&#39;, credentials[&#39;BUCKET&#39;],&#39;wine.gz&#39;)</pre><pre>#upload pickle object<br>cos.upload_file(&#39;GB_Classification_model.pkl&#39;, credentials[&#39;BUCKET&#39;],&#39;GB_Classification_model.pkl&#39;)</pre><p>To help you with quickly uploading files here is the upload_file_cos function. You need to pass your credentials, local file name and key as parameters to this function.</p><pre>from ibm_botocore.client import Config<br>import ibm_boto3</pre><pre>def upload_file_cos(credentials,local_file_name,key):  <br>    cos = ibm_boto3.client(service_name=&#39;s3&#39;,<br>    ibm_api_key_id=credentials[&#39;IBM_API_KEY_ID&#39;],<br>    ibm_service_instance_id=credentials[&#39;IAM_SERVICE_ID&#39;],<br>    ibm_auth_endpoint=credentials[&#39;IBM_AUTH_ENDPOINT&#39;],<br>    config=Config(signature_version=&#39;oauth&#39;),<br>    endpoint_url=credentials[&#39;ENDPOINT&#39;])<br>    try:<br>        res=cos.upload_file(Filename=local_file_name, Bucket=credentials[&#39;BUCKET&#39;],Key=key)<br>    except Exception as e:<br>        print(Exception, e)<br>    else:<br>        print(&#39;File Uploaded&#39;)</pre><p><strong>Download Files To Local Machine</strong></p><p>Once you have your file in IBM Cloud Object Storage , now you can download it on your local machine.</p><ol><li>Click on your project</li><li>Click find and add data icon from upper right hand side panel.</li><li>Select the file and click download.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-6zaMSCQKUlIXM9vw2YCwA.png" /></figure><p><strong>Get Data From COS Into Notebook</strong></p><p>To download file from COS into notebook, we will be using download_file function. It takes same parameters as above. Here I am downloading file wine.csv to saving it with name wine1.csv</p><pre>cos.download_file(Bucket=credentials[&#39;BUCKET&#39;],Key=&#39;wine.csv&#39;,Filename=&#39;wine1.csv&#39;)</pre><p>Here is download_file_cos function for quick use.</p><pre>from ibm_botocore.client import Config<br>import ibm_boto3</pre><pre>def download_file_cos(credentials,local_file_name,key):  <br>    cos = ibm_boto3.client(service_name=&#39;s3&#39;,<br>    ibm_api_key_id=credentials[&#39;IBM_API_KEY_ID&#39;],<br>    ibm_service_instance_id=credentials[&#39;IAM_SERVICE_ID&#39;],<br>    ibm_auth_endpoint=credentials[&#39;IBM_AUTH_ENDPOINT&#39;],<br>    config=Config(signature_version=&#39;oauth&#39;),<br>    endpoint_url=credentials[&#39;ENDPOINT&#39;])<br>    try:<br>        res=cos.download_file(Bucket=credentials[&#39;BUCKET&#39;],Key=key,Filename=local_file_name)<br>    except Exception as e:<br>        print(Exception, e)<br>    else:<br>        print(&#39;File Downloaded&#39;)</pre><p>Instead of file if you want to upload/download file-like object you can use upload_fileobj and download_fileobj. Object must implement the read method, and return/accept bytes respectively.</p><pre>with open(&#39;wine.csv&#39;, &#39;rb&#39;) as data:<br>    cos.upload_fileobj(data,  credentials[&#39;BUCKET&#39;], &#39;wine_bytes&#39;)</pre><pre>with open(&#39;wine_copy.csv&#39;, &#39;wb&#39;) as data:<br>    cos.download_fileobj(credentials[&#39;BUCKET&#39;], &#39;wine_bytes&#39;, data)</pre><p>Credentials you get in Watson Studio using insert to codeoption are scoped to one bucket access permission i.e. they allow you to only interact with your project’s bucket. If you want to interact with other buckets then you will have to create new credentials with appropriate access permissions.</p><p><strong>Create New Service Credentials</strong></p><p>To create new credentials, you need to follow these steps.</p><ol><li>Go to <a href="https://console.bluemix.net/">IBM Cloud</a> and click on IBM Cloud Object Storage service under services section.</li><li>Select Service credentials from left hand panel and click on New credential button.</li><li>Give name and choose appropriate role based on your requirements and hit add button.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3iqGA13bls-oR5J5uCjcTw.png" /></figure><p>You should be able to see this credential under service credentials.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nFHUitv5f3FXp8uDIXlKWA.png" /></figure><p>Just copy these credentials from view credentials option and create cos object.</p><pre>cos_credentials={<br>  &quot;apikey&quot;: &quot;***********************&quot;,<br>  &quot;endpoints&quot;: &quot;***********************&quot;,<br>  &quot;iam_apikey_description&quot;: &quot;***********************&quot;,<br>  &quot;iam_apikey_name&quot;: &quot;***********************&quot;,<br>  &quot;iam_role_crn&quot;: &quot;***********************&quot;,<br>  &quot;iam_serviceid_crn&quot;: &quot;***********************&quot;,<br>  &quot;resource_instance_id&quot;: &quot;***********************&quot;<br>}</pre><pre>auth_endpoint = &#39;<a href="https://iam.bluemix.net/oidc/token&#39;">https://iam.bluemix.net/oidc/token&#39;</a><br>service_endpoint = &#39;<a href="https://s3-api.us-geo.objectstorage.softlayer.net&/#39;">https://s3-api.us-geo.objectstorage.softlayer.net&#39;</a></pre><pre>cos = ibm_boto3.client(&#39;s3&#39;,<br>                         ibm_api_key_id=cos_credentials[&#39;apikey&#39;],<br>                        ibm_service_instance_id=cos_credentials[&#39;resource_instance_id&#39;],<br>                         ibm_auth_endpoint=auth_endpoint,<br>                         config=Config(signature_version=&#39;oauth&#39;),<br>                         endpoint_url=service_endpoint)</pre><p><strong>List Buckets</strong></p><p>Using list_buckets function we can list all buckets.</p><pre>for bucket in cos.list_buckets()[&#39;Buckets&#39;]:<br>    print(bucket[&#39;Name&#39;])</pre><p><strong>Create &amp; Delete Buckets</strong></p><p>create_bucket and delete_bucket function will help you in creating and deleting buckets.</p><pre>cos.create_bucket(Bucket=&#39;bucket1-test&#39;)<br>cos.delete_bucket(Bucket=&#39;bucket1-test&#39;)</pre><p>There are many functions you can use to manage your IBM Cloud Object Storage . In this blog,we have covered basic functions to make your job easy while working with IBM Cloud Object Storage in Watson Studio using Python.</p><p>For more information you can refer this <a href="https://console.bluemix.net/docs/services/cloud-object-storage/about-cos.html#about-ibm-cloud-object-storage">documentation</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe0ba8667d5f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Spatial Data Analysis in R using RGEOS]]></title>
            <link>https://medium.com/@snehalgawas/introduction-to-spatial-data-analysis-in-r-using-rgeos-ea69059c3b90?source=rss-c3235dc190fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/ea69059c3b90</guid>
            <category><![CDATA[rstudio]]></category>
            <category><![CDATA[rstats]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Snehal Gawas]]></dc:creator>
            <pubDate>Wed, 24 Jan 2018 17:44:52 GMT</pubDate>
            <atom:updated>2018-01-24T17:44:52.230Z</atom:updated>
            <content:encoded><![CDATA[<p>Working with spatial data is easy in R. In this blog, we will go through some of the R — libraries that are useful for spatial data analysis and their basic usage.</p><p><strong>Data &amp; Use Case</strong></p><p>Today we will be using median household income data for the United States from the 2006 to 2010 census. You can download data <a href="https://www.psc.isr.umich.edu/dis/census/Features/tract2zip/">here</a>.</p><p>Shape file: There are various online resources to get shape files . You can download different shape files <a href="https://www.census.gov/geo/maps-data/data/tiger-cart-boundary.html">here</a>.</p><p>We will learn about how to read and plot shape files in R and we will try to explore household income data for the United States using different libraries mentioned below.</p><p><strong>Libraries</strong></p><ol><li><strong>rgeos</strong>-rgeos implements functionality for the manipulation and querying of spatial geometries using the Geometry Engine — Open Source (GEOS)C library. In this blog, our focus will be to demonstrate basic functionalities of rgeos.</li><li><strong>maptools</strong>- It contains different functions to read and manipulate the geographic data. We will be using maptools to read our shape file. A shapefile is a format for storing the geometric locations and attribute information of geographic features. The shapefile format spatially describes vector features like points, lines, and polygons.</li><li><strong>ggmap</strong> -This package has a collection of functions to visualize spatial data and models on top of static maps from various online sources.</li><li><strong>sp</strong> -This is an important package for plotting data as maps, spatial selection, retrieving coordinates, subsetting etc.</li><li><strong>zipcodes</strong>-This package contains a database of city, state, latitude, and longitude information for U.S. ZIP codes. We will be using the package to extract latitude and longitude attributes of available zip codes in the data.</li></ol><p><strong>Required Libraries</strong></p><p>First, let’s import all required libraries and their dependencies.</p><pre>library(ggmap)<br>library(zipcode)<br>library(rgeos)<br>library(sp)<br>library(maptools)</pre><p><strong>Data Import and Cleaning</strong></p><p>Let’s now read data in R. This data is in csv format.</p><pre>DF&lt;-data.table::fread(&#39;/home/rstudio/mean houshold income.csv&#39;)<br>head(DF)<br>     <strong>Zip Median $    Pop<br>1: 1001   56,663 16,445<br>2: 1002   49,853 28,069<br>3: 1003   28,462  8,491<br>4: 1005   75,423  4,798<br>5: 1007   79,076 12,962<br>6: 1008   63,980  1,244</strong></pre><p>Before we start working on this data we need to clean it a bit and rename the columns for convenient use. We are going to remove $ and, symbols from income and population columns and convert them to numeric type.</p><pre># Clean Data <br>names(DF) &lt;- c(&#39;Zip&#39;,&#39;Median_income&#39;,&#39;Population&#39;)<br>DF$Zip&lt;-as.factor(DF$Zip)<br>DF$Median_income &lt;- gsub(pattern = &#39;,&#39;, replacement = &#39;&#39;,x =  DF$Median_income)<br>DF$Median_income &lt;- as.numeric(DF$Median_income)<br>DF$Population &lt;- gsub(pattern = &#39;,&#39;, replacement = &#39;&#39;,x =  DF$Population)<br>DF$Population &lt;- as.numeric(DF$Population)</pre><p><strong>Extract of Latitude &amp; Longitude</strong></p><p>Next step is to get latitude and longitude attributes for our data using package zipcode. zipcode data contains zip codes, cities, states, latitudes and longitudes. We will be joining our data to zipcode data on zip column.</p><pre># Extract longitude and latitude from available zipcodes<br>data(zipcode)<br>DF=merge(DF,zipcode,by.x=&#39;Zip&#39;,by.y=&#39;zip&#39;)<br>head(DF)</pre><pre><strong>Zip Median_income Population     city state latitude longitude<br>1: 10001         71245      17678 New York    NY 40.75074 -73.99653<br>2: 10002         30844      70878 New York    NY 40.71704 -73.98700<br>3: 10003         89999      53609 New York    NY 40.73251 -73.98935<br>4: 10004        110184       1271 New York    NJ 40.69923 -74.04118<br>5: 10005        115133       1517 New York    NY 40.70602 -74.00858<br>6: 10006        111220        972 New York    NY 40.70790 -74.01342</strong></pre><p><strong>Plot Data On Map</strong></p><p>Now, we have everything to get started. let’s start by plotting this data on a map using ggmap . Here we will be using get_map wrapper to query Google Maps. ggmap will plot the base map of united states for us and then using geom_point we will be plotting latitude and longitude points on this map. scale_color_gradient will help us to creates color gradient based on median household income (low to high).</p><pre>map&lt;- get_map(location = &#39;united_states&#39;,zoom=4,maptype = &#39;terrain&#39;,source = &#39;google&#39;,color = &#39;color&#39;)</pre><pre>ggmap(map) +<br>  geom_point(aes(x=longitude,y=latitude<br>                 ,color = Median_income)<br>                ,data=DF<br>                ,na.rm = T<br>                ,size = .5<br>  ) +scale_color_gradient(low=&quot;coral&quot;, high=&quot;blue&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*kXFRgLTHGLRdlFf1jI2upQ.png" /></figure><p>We are interested in highest and lowest income points.</p><pre>x_max=DF[DF$Median_income==max(DF$Median_income),longitude]<br>y_max=DF[DF$Median_income==max(DF$Median_income),latitude]</pre><pre>x_min=DF[DF$Median_income==min(DF$Median_income),longitude]<br>y_min=DF[DF$Median_income==min(DF$Median_income),latitude]</pre><p><strong>Read &amp; Plot Shape File</strong></p><p>Now let’s read shape file of USA states. Here readShapePoly function from maptools will help us to read data from a polygon shapefile into a SpatialPolygonsDataFrame object.</p><p>The CRS holds the projection definition and provides a standardized way of describing locations. Different CRS are used to describe geographic data.</p><pre>#Read shape file<br>crswgs84=CRS(&quot;+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs&quot;)</pre><pre>states_shape &lt;- readShapePoly(&quot;/home/rstudio/states_21basic/states&quot;,proj4string=crswgs84,verbose=TRUE)</pre><pre>plot(states_shape,border=&quot;wheat3&quot;, col=&quot;wheat1&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*UStjAZTkARsBOOXIxBtJ7Q.png" /></figure><p><strong>Subsetting Shape File</strong></p><p>We can take subset of this SpatialPolygonsDataFrame object and play around it. Here we have selected and East-North-Central region of USA.</p><pre>East_North_Central&lt;-subset(states_shape,STATE_NAME==&quot;Illinois&quot; | STATE_NAME==&quot;Indiana&quot;|STATE_NAME==&#39;Michigan&#39;|STATE_NAME==&#39;Ohio&#39;|STATE_NAME==&#39;Wisconsin&#39;)</pre><pre>plot(East_North_Central,border=&quot;wheat3&quot;, col=&quot;wheat1&quot;)<br>text(East_North_Central,East_North_Central$STATE_ABBR,cex=0.8)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*k3fYrl-zG0-Po0Tej53WMg.png" /></figure><p>Now let’s try to find which states have highest and lowest income points by plotting them on the shape file.</p><pre># plot high and low income points <br>plot(states_shape,border=&quot;wheat3&quot;, col=&quot;wheat1&quot;)<br>text(states_shape,states_shape$STATE_ABBR,cex=0.4)<br>points(x_max,y_max,col=&#39;red&#39;,pch=16)<br>points(x_min,y_min,col=&#39;green&#39;,pch=16)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*CrlPzhgWx50dMw6ByZLvaA.png" /></figure><p>Looking at the above graph it is easy identify Illinois with highest income point while Mississippi with lowest income point but can we do it without looking at these names?</p><p>Using gContains function in rgeos we can search for these low and high income points in shapefile. gContains will test whether one geometry contains or is contained within another geometry</p><p>To use gContains first we have to convert our latitude and longitude into SpatialPoint objects.</p><pre># Convert Latitude and longitude into SpatialPoints objects</pre><pre>co_max = cbind(x_max,y_max)<br>co_min= cbind(x_min,y_min)</pre><pre>pt_max_income = SpatialPoints(co_max,proj4string = crswgs84)<br>pt_min_income = SpatialPoints(co_min,proj4string = crswgs84)</pre><p>Then we can use the function to search which state have these points.</p><pre>res_high &lt;- colSums(gContains(states_shape, pt_max_income, byid = TRUE)) <br>HIgh_median_Income=setNames(res_high, states_shape@data$STATE_NAME)<br>HIgh_median_Income[HIgh_median_Income&gt;0]</pre><pre><strong>Illinois <br>       1</strong></pre><pre>res_low &lt;- colSums(gContains(states_shape, pt_min_income, byid = TRUE)) <br>Low_median_Income=setNames(res_low, states_shape@data$STATE_NAME)<br>Low_median_Income[Low_median_Income&gt;0]</pre><pre><strong>Mississippi <br>          1</strong></pre><p>gDistance function is helpful in calculating distance between two geometries. It returns the cartesian minimum distance between the two geometries in the units of the current projection. Here we are interested in distance between high and low income points.</p><pre>&gt; gDistance(pt_max_income,pt_min_income)<br>  <strong>9.281726</strong></pre><p>Now let’s divide data into high and low income group and create SpatialPointsDataFrame.</p><pre># divide data into high and low groups</pre><pre>High_income=DF[(DF$Median_income&gt;summary(DF$Median_income)[5])]<br>low_income=DF[(DF$Median_income&lt;summary(DF$Median_income)[2])]</pre><pre>#convert dataframe to SpatialPointsDataFrame</pre><pre>xy_high &lt;- High_income[,c(6,7)]<br>spdf_high &lt;- SpatialPointsDataFrame(coords = xy_high, data = High_income,proj4string = crswgs84)</pre><pre>xy_low &lt;- low_income[,c(6,7)]<br>spdf_low &lt;- SpatialPointsDataFrame(coords = xy_low, data = low_income,proj4string = crswgs84)</pre><p>Using function gConvexHull we will create smallest convex polygon that contains all sub geometries of high and low income groups respectively.</p><pre>high_poly=gConvexHull(spdf_high)<br>low_poly=gConvexHull(spdf_low)</pre><p>Let’s check whether geometries that we have created are valid or not. gIsValid function tests if the given geometry is valid and returns TRUE if the given geometry is well formed, FALSE otherwise. If reason is set to TRUE then a character string is returned describing the geometry, “Valid Geometry” if it is valid or details of the specific issue</p><pre>gIsValid(high_poly,reason=TRUE)<br><strong>&quot;Valid Geometry&quot;<br></strong>gIsValid(low_poly,reason=TRUE)<br><strong>&quot;Valid Geometry&quot;</strong></pre><p>Once our geometry is well formed, we can calculate centroid of these geometries by using gCentroid function. It will return a SpatialPoints object of the geometry .gIntersection function determines the intersection between the two given geometries and returns spatial intersections as sp objects.</p><pre>plot(high_poly,border=&#39;red&#39;)<br>plot(gIntersection(low_poly,high_poly),add=TRUE,col=&#39;beige&#39;)<br>plot(gCentroid(high_poly),col=&#39;red&#39;,add=TRUE)<br>plot(gCentroid(low_poly),col=&#39;blue&#39;,add=TRUE)</pre><p>In the below image, colored part is the intersection between high and low income regions with their centroids.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*ua7ptv67177_YylVW8CLpw.png" /></figure><p>There are different other functions to experiment with in rgeos package.</p><p>gLength : Calculate length of the geometry in the units of the current projection.</p><p>gBoundary: Determine the Boundary of the given geometry.</p><p>gArea : calculates area of the geometry in the units of the current projection.</p><pre>gArea(high_poly)<br><strong>3586.777<br></strong>gLength(high_poly)<br><strong>253.1442<br></strong>gBoundary(high_poly)<br><strong>class       : SpatialLines <br>features    : 1 <br>extent      : 19.2783, 71.29953, -168.0218, -71.9506  (xmin, xmax, ymin, ymax)<br>coord. ref. : +proj=longlat +ellps=WGS84</strong></pre><p>In the above example, we worked with only one shape file. Now let’s see how to use multiple shapefiles. Let’s choose western region of the USA and try to plot counties and their centroids.</p><p>Here we will be using readShapePoly to read counties shape file and readShapePoints to read centroids shape file.</p><pre>#Read counties and centroids shape file<br>counties &lt;- readShapePoly(&quot;counties&quot;)<br>centroids &lt;- readShapePoints(&quot;centroids&quot;)</pre><pre>plot(counties, border=&quot;wheat3&quot;, col=&quot;wheat1&quot;)<br>points(centroids,col=&#39;red&#39;,pch=16,cex=0.3)<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*H0GqZzzGqJFkgHQVxarVYA.png" /></figure><p>This is just high level introduction to rgeos and other libraries useful in spatial data analysis and plotting. There are many libraries in R to play with spatial data.</p><p>You can find the full R script <a href="https://gist.github.com/SnehalGawas/e2e078dc53f755d2d79878e5d60ad81b">here</a>. To get started, create a free account at <a href="https://datascience.ibm.com/">Data Science Experience</a> and start experimenting with spatial data in RStudio.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea69059c3b90" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>