<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Veda Shankar on Medium]]></title>
        <description><![CDATA[Stories by Veda Shankar on Medium]]></description>
        <link>https://medium.com/@veda.shankar?source=rss-ea8d1256acec------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*d5qNPqdBeUo6WwgRupXNRQ.jpeg</url>
            <title>Stories by Veda Shankar on Medium</title>
            <link>https://medium.com/@veda.shankar?source=rss-ea8d1256acec------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 21:03:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@veda.shankar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Using OmniSci as Data Source for Power BI]]></title>
            <link>https://medium.com/@veda.shankar/using-omnisci-as-data-source-for-power-bi-969bda62737d?source=rss-ea8d1256acec------2</link>
            <guid isPermaLink="false">https://medium.com/p/969bda62737d</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[gpu-computing]]></category>
            <category><![CDATA[visualization]]></category>
            <dc:creator><![CDATA[Veda Shankar]]></dc:creator>
            <pubDate>Thu, 20 Jun 2019 06:07:54 GMT</pubDate>
            <atom:updated>2019-06-27T20:00:54.214Z</atom:updated>
            <content:encoded><![CDATA[<p>Microsoft Power BI is a popular business analytics solution that supports a wide range of data sources from SQL databases, web platforms such as Google Analytics and Salesforce, and Azure cloud. In this tutorial, we walk you through step-by-step instructions on how to use the world’s fastest GPU-powered database — OmniSci — as a data source for Power BI.</p><p>OmniSci natively supports ODBC (Open Database Connectivity), and Power BI does as well, allowing communication with OmniSci through the Windows platform ODBC Driver Manager. The OmniSci-specific ODBC driver processes ODBC function calls, submits the SQL requests to OmniSci and return the results to Power BI. Note: The OmniSci ODBC driver is provided as part of OmniSci Enterprise Edition; contact your OmniSci Sales Representative for more details. . To see all of the Tier 1 (Core) ODBC functions supported by OmniSci, refer to our <a href="https://www.omnisci.com/docs/latest/6_odbc.html#supported-odbc-functions-1">documentation</a>..</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/0*bIeM9d4wq4c-FCon" /></figure><h3>Installing the OmniSci ODBC Software</h3><p>Download the OmniSci ODBC Software using the URL provided by your OmniSci Sales Representative. The downloaded installation package has a filename format as OmniSciInstall_&lt;major_version&gt;.&lt;minor_version&gt; as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0kPY6-aaFcsEgmYO" /></figure><p>Run the installer program as a Windows <strong>Administrator</strong> and confirm to proceed with the installation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*A1ccHX2ZoJjbdLEB" /></figure><p>By default the software will be installed under <strong>C:\Program Files\OmniSci\ODBC</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/0*7Al2Hkzcp4-at2LN" /></figure><p>At the end of a successful installation, you get the following message:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/0*Kig8pyhD8tGBRV0X" /></figure><h3>ODBC Driver DSN Setup</h3><p>Open the <strong>Administrator Tool</strong> and open the<strong> ODBC Data Sources (64-bit).</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Gjx650sUYwc_peYp" /></figure><p>Select <strong>Add</strong> on the ODBC Data Source Administrator (64-bit):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yVVK97w_b1CHM0tD" /></figure><p>Select the OmniSci Driver:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e62AAVcP5LH3aKra" /></figure><p>This will bring up the form to setup the Data Source Name (DSN) for OmniSci database. The form shown below is for the default settings for OmniSci <strong>version 4.6</strong>. Please make sure that the name of the database, the user name and password are correct for your environment. Use port <strong>6274</strong> for connecting to the database. After the form is filled, select <strong>Test</strong> to make sure you can connect to the OmniSci database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6rHEMlopTcZiwoXd" /></figure><p>If the settings in the DSN form are correct then you should be able to connect to OmniSci database and see the following success message:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/0*w2BUyMtMXBGq-rlL" /></figure><h3>Querying an OmniSci Table from Power BI</h3><p>Open the Power BI application and select <strong>Get data</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mI-He8gssVSH46jd" /></figure><p>From the list of connectors, choose <strong>Other</strong> -&gt; <strong>ODBC</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*evs-_E9C_dGLteAI" /></figure><p>You should be able to see the OmniSci DSN that was created in the previous step, select <strong>OK</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*x8gXGu5W46vnxxgg" /></figure><p>Enter a valid user name and password for the database, and press <strong>Connect</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*80bf--2VFcRMpe2D" /></figure><p>The navigator window now shows the list of tables under the selected <strong>mapd</strong> database. Select the table <strong>flights_2008_7M</strong> which is usually added to the database as part of OmniSci installation. The Navigator window displays a preview of the flights table. At this point, you can edit the query before loading the table, by selecting <strong>Edit</strong> or you can <strong>Load</strong> the table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BLm7RudICZGOA0-2" /></figure><p>You now have access to all the fields of the flights table, and you can create a visualization by selecting the chart type and dragging the desired fields from the <strong>Fields</strong> list.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mLctxQnaTM72sHNW" /></figure><p>Using the flights dataset, I created a dashboard with a bar chart showing the number of records for each airline and a scatter plot showing the obvious correlation between the departure delay along the X axis (<strong>depdelay</strong>) and arrival delay along the Y axis (<strong>arrdelay</strong>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RTE_qp7KUahu-ilS" /></figure><p>When you select <strong>Get Data -&gt; Other -&gt; ODBC</strong>, instead of getting the entire table, you can also get the desired columns using a SQL query as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*XyjKB8Mww538zcRPLduPzA.png" /></figure><p>The SQL query executes on OmniSci, and a preview of the results is shown:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gRqLMUUe2E6DiBJhw8BEWg.png" /></figure><p>Upon selecting <strong>LOAD</strong>, all of the data defined by the query is imported into the Power BI cache. Now you are ready to build charts on the loaded data. If the data changes on the underlying tables on OmniSci, the changes are not reflected in the charts. You have to select <strong>Refresh</strong> to trigger a reimport of the data to reflect changes on the Power BI charts.</p><p>This basic tutorial shows how to work with OmniSci using Power BI, so I am not diving into the advanced integration points that both platforms offer. For example, I am thinking of creating an interactive Power BI dashboard alongside OmniSci’s Choropleth chart that renders millions of polygons using GPU technology. If you have interesting ideas on how best to leverage OmniSci’s fast SQL database and extreme rendering capabilities with Power BI visualization, then stop by our <a href="https://community.omnisci.com/communities/community-home/digestviewer?CommunityKey=d06df790-8ca4-4e54-91a0-244af0228ddc&amp;tab=digestviewer"><strong>community forum</strong></a> to discuss.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=969bda62737d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[OmniSci — Importing Data From Kinesis Stream]]></title>
            <link>https://medium.com/@veda.shankar/omnisci-importing-data-from-kinesis-stream-5b87dc2324c3?source=rss-ea8d1256acec------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b87dc2324c3</guid>
            <category><![CDATA[visual-analytics]]></category>
            <category><![CDATA[omnisci]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[kafka-streams]]></category>
            <category><![CDATA[big-data-analytics]]></category>
            <dc:creator><![CDATA[Veda Shankar]]></dc:creator>
            <pubDate>Tue, 21 May 2019 20:31:36 GMT</pubDate>
            <atom:updated>2019-05-21T20:31:36.638Z</atom:updated>
            <content:encoded><![CDATA[<h3>OmniSci — Importing Data From Kinesis Stream</h3><h3>OmniSci — Importing Data From Kinesis Stream</h3><p><a href="https://kafka.apache.org/">Apache Kafka</a> is a very popular open-source, distributed messaging solution that is used by companies to build real-time data pipelines and streaming apps. However, implementing a scalable, fault-tolerant and performant Kafka deployment is quite involved with many resource considerations. Amazon Kinesis is a managed platform for streaming data on Amazon Web Services (AWS). Kinesis provides built-in features like scaling, automatic replication for high availability, encryption etc that are a must in a production data pipeline. This makes it a very compelling alternative to Kafka especially if you are already using AWS for your workloads.</p><p>You can use Amazon Kinesis Data Streams to collect and process large<a href="https://aws.amazon.com/streaming-data/"> streams</a> of data records in real time. A typical use case is real-time aggregation of data followed by loading the aggregate data into a data warehouse for analysis. In this tutorial, I provide a step-by-step procedure for creating a simple Kinesis Data Stream application, reading data from a Kinesis stream as data records and writing it to an OmniSci database table for querying and visualization.This tutorial uses an <strong>Amazon Linux 2 AMI</strong> instance for testing with IAM policies <strong>AmazonKinesisFullAccess</strong>, <strong>AmazonDynamoDBFullAccess</strong>, and <strong>CloudWatchFullAccess</strong> attached to it.</p><h3>Kinesis Streams Concepts</h3><p>Here is a brief description of the key concepts in Amazon Kinesis Data Streams;for more details, refer to <a href="https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html">AWS Documentation</a>.</p><h4>Producer</h4><p>Producers put records into Amazon Kinesis Data Streams. For example, a web server sending log data to a stream is a producer.</p><h4>Consumer</h4><p>Consumers get records from Amazon Kinesis Data Streams and process them. These consumers are known as<a href="https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html#enabled-application"> Amazon Kinesis Data Streams Application</a>.</p><h4>Kinesis Data Stream</h4><p>A Kinesis data stream is a set of<a href="https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html#shard"> shards</a>. Each shard has a sequence of data records. Each data record has a<a href="https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html#sequence-number"> sequence number</a> that is assigned by Kinesis Data Streams. Each shard can support up to 5 transactions per second for reads, up to a maximum total data read rate of 2 MB per second and up to 1,000 records per second for writes, up to a maximum total data write rate of 1 MB per second</p><h4>Data Record</h4><p>A data record is the unit of data stored in a<a href="https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html#stream"> Kinesis data stream</a>. Data records are composed of a<a href="https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html#sequence-number"> sequence number</a>, a<a href="https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html#partition-key"> partition key</a>, and a data blob, which is an immutable sequence of bytes. A data blob can be up to 1 MB.</p><h4>Retention Period</h4><p>The retention period is the length of time that data records are accessible after they are added to the stream. A stream’s retention period is set to a default of 24 hours after creation which can be increased to 168 hours (7 days).</p><p>The following diagram depicts the different components in the discussed Kinesis Data Stream application. In this example, the producer will write mock natural gas production data records to the Kinesis stream.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*3YwTroM3qRbDPhfgmfEUFA.jpeg" /></figure><h3>Create a Kinesis Stream</h3><p>The first step is to create a stream and verify that it was successfully created. Use the following command to create a stream named <strong>gas-production</strong> with a single shard.</p><blockquote><strong>aws kinesis create-stream — stream-name gas-production — shard-count 1</strong></blockquote><p>The stream name <strong>gas-production</strong> identifies the stream. The name is scoped to the AWS account used and is also scoped by the AWS Region. That is, two streams in two different accounts can have the same name, and two streams in the same account, but in two different Regions, can have the same name.</p><p>Check if the stream was created:</p><blockquote><strong>aws kinesis list-streams</strong></blockquote><pre>{<br>  &quot;StreamNames&quot;: [<br>    &quot;gas-production&quot;<br>  ]<br>}</pre><p>Check the status of the stream using:</p><blockquote><strong>aws kinesis describe-stream — stream-name gas-production</strong></blockquote><pre>{<br>  &quot;StreamDescription&quot;: {<br>    &quot;KeyId&quot;: null,<br>    &quot;EncryptionType&quot;: &quot;NONE&quot;,<br>    &quot;StreamStatus&quot;: &quot;ACTIVE&quot;,<br>    &quot;StreamName&quot;: &quot;gas-production&quot;,<br>    &quot;Shards&quot;: [<br>      {<br>        &quot;ShardId&quot;: &quot;shardId-000000000000&quot;,<br>        &quot;HashKeyRange&quot;: {<br>          &quot;EndingHashKey&quot;: &quot;340282366920938463463374607431768211455&quot;,<br>          &quot;StartingHashKey&quot;: &quot;0&quot;<br>        },<br>        &quot;SequenceNumberRange&quot;: {<br>          &quot;StartingSequenceNumber&quot;: &quot;49595724303092122779582799139640537568816759595765596162&quot; <br>        }<br>      }<br>    ],<br>    &quot;StreamARN&quot;: &quot;arn:aws:kinesis:us-east-1:041129453290:stream/gas-production&quot;,<br>    &quot;EnhancedMonitoring&quot;: [<br>      {<br>        &quot;ShardLevelMetrics&quot;: []<br>      }<br>    ],<br>    &quot;StreamCreationTimestamp&quot;: 1557871137.0,<br>    &quot;RetentionPeriodHours&quot;: 24<br>  }<br>}</pre><p>You can also use the AWS Console Kinesis Dashboard to monitor the status of the service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x5vtgJFRM8BAWgXBxMNmRQ.png" /></figure><p>The StreamStatus is initially in state <strong>CREATING</strong>, and after some time it transitions to <strong>ACTIVE</strong> state. The read and write operations can only be performed on an ACTIVE stream.</p><h3>Stream Data To Producer Using Kinesis Agent</h3><p>Kinesis Agent is a stand-alone Java software application that offers an easy way to collect and send data to Kinesis Data Streams. The agent continuously monitors a set of files and sends new data to a stream. The agent handles file rotation, checkpointing, and retry on failure. It delivers all of your data in a reliable, timely, and simple manner. It also emits Amazon CloudWatch metrics to help you better monitor and troubleshoot the streaming process. <a href="https://docs.aws.amazon.com/streams/latest/dev/writing-with-agents.html">Reference AWS documentation</a>.</p><p>Use the following command to download and install the agent:</p><blockquote><strong>sudo yum install –y</strong><a href="https://s3.amazonaws.com/streaming-data-agent/aws-kinesis-agent-latest.amzn1.noarch.rpm"><strong> https://s3.amazonaws.com/streaming-data-agent/aws-kinesis-agent-latest.amzn1.noarch.rpm</strong></a></blockquote><p>As superuser, edit the configuration file<strong> /etc/aws-kinesis/agent.json</strong>. In this configuration file, specify the files ( “filePattern” ) from which the agent collects data, and the name of the stream ( “kinesisStream” ) to which the agent sends data. Note that the file name is a pattern, and the agent recognizes file rotations. You can rotate files or create new files no more than once per second. The agent uses the file creation timestamp to determine which files to track and tail into your stream; creating new files or rotating files more frequently than once per second does not allow the agent to differentiate properly between them.</p><pre>{<br>  &quot;cloudwatch.emitMetrics&quot;: true,<br>  &quot;kinesis.endpoint&quot;: &quot;&quot;,<br>  &quot;firehose.endpoint&quot;: &quot;&quot;,<br>  &quot;flows&quot;: [<br>    {<br>      &quot;filePattern&quot;: &quot;<strong>/tmp/kinesis/production-data*</strong>&quot;,<br>      &quot;kinesisStream&quot;: &quot;<strong>gas-production</strong>&quot;,<br>      &quot;partitionKeyOption&quot;: &quot;RANDOM&quot;<br>    }<br>  ]<br>}</pre><p>Start the agent manually</p><blockquote><strong>sudo service aws-kinesis-agent start</strong></blockquote><p>Configure the agent to start on system startup:</p><blockquote><strong>sudo chkconfig aws-kinesis-agent on</strong></blockquote><p>The agent is now running as a system service in the background. It continuously monitors the specified files and sends data to the specified stream. The Kinesis agent runs in the background continuously monitoring the directory <strong>/tmp/kinesis</strong> for changes to files that fit the pattern <strong>production-data*</strong>. New records added to the files are sent to the stream <strong>gas-production</strong>. The activities of the agent is logged in <strong>/var/log/aws-kinesis-agent/aws-kinesis-agent.log</strong>.</p><p>To simulate the production of real time data, I ran a script in a loop adding records with the following format:</p><blockquote><strong>flow_date, flow_value, state, county, region, display name, latitude, longitude</strong></blockquote><blockquote><strong>2018–02–08 16:00:00, 10.008, Pennsylvania, Warren, Northeast, National Fuel Gas Supply, 41.837835, -79.148063</strong></blockquote><p>You can use <strong>AWS CloudWatch</strong> -&gt; <strong>Kinesis Data Stream</strong> to visually monitor the records that are written and read from the stream.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gmMnGm5JIjjGn-hhE6LuqA.png" /></figure><h3>Creating a Kinesis Consumer</h3><p>I used the <strong>boto</strong> python library to implement the Kinesis data stream consumer application. By setting the AWS keys as environment variables in your terminal, you can use the boto library without having to specify the AWS credentials in the python script.</p><blockquote><strong>export AWS_ACCESS_KEY_ID=XXXXXX</strong></blockquote><blockquote><strong>export AWS_SECRET_ACCESS_KEY=XXXXXX</strong></blockquote><p>Get a low-level client handle representing Amazon Kinesis for a specific region:</p><pre><strong>kinesis_client = boto3.client(&#39;kinesis&#39;, region_name=&#39;us-east-1&#39;)</strong></pre><p>Use <strong>describe_stream</strong> to get the details for the stream <strong>gas-production</strong>. The returned information includes the stream name, Amazon Resource Name (ARN), creation time, enhanced metric configuration, and shard map.</p><pre><strong>my_stream_name = ‘gas-production&#39;<br>response = kinesis_client.describe_stream(StreamName=my_stream_name)</strong></pre><p>You can see the description of the returned JSON structure in <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/kinesis.html#Kinesis.Client.get_records"><strong>boto documentation</strong></a>.</p><p>Get the <strong>shard id</strong> which uniquely identifies the shard in the stream. As <strong>gas-production</strong> was created with only one shard, use <strong>index 0</strong> in the <strong>Shards</strong> dictionary structure to get the id.</p><pre><strong>my_shard_id = response[‘StreamDescription&#39;][&#39;Shards&#39;][0][&#39;ShardId&#39;]</strong></pre><p>Use<strong> get_shard_iterator</strong> to get the shard iterator for gas_production stream. The shard iterator specifies the shard position from which to start reading data records sequentially. The position is specified using the sequence number of a data record in a shard. I use <strong>LATEST</strong> for the <strong>ShardIteratorType</strong> to start reading just after the most recent record in the shard, so that you always read the most recent data.</p><pre><strong>shard_iterator = kinesis_client.get_shard_iterator(StreamName=my_stream_name, ShardId=my_shard_id, ShardIteratorType=&#39;LATEST&#39;)<br>my_shard_iterator = shard_iterator[‘ShardIterator&#39;]</strong></pre><p>Use the shard iterator with the<strong> get_records</strong> API to start reading data records sequentially. To read from a stream continually, call <strong>get_records</strong> in a <strong>loop</strong>. Use the following loop to read one record at a time from the stream; the Data element in the Records dictionary structure contains the payload data in base64-encoding. The data blob is decoded and finally appended to a Pandas dataframe.</p><pre><strong>df = pd.DataFrame(columns=[&#39;flow_date&#39;, &#39;flow_value&#39;, &#39;state&#39;, &#39;county&#39;, &#39;region&#39;, &#39;display_name&#39;, &#39;latitude&#39;, &#39;longitude&#39;])<br>row = 0<br>while 1==1:<br>  record_response = kinesis_client.get_records(ShardIterator=my_shard_iterator, Limit=1)<br>  for item in record_response[&quot;Records&quot;]:<br>    record_data = item[&quot;Data&quot;].decode(&quot;utf-8&quot;)<br>    record_data = record_data.rstrip(&#39;\n&#39;)<br>    list = record_data.split(&#39;,&#39;)<br>    df.loc[row] = list<br>    row = row + 1<br>  my_shard_iterator = record_response[&#39;NextShardIterator&#39;]</strong></pre><p>When you read repeatedly from a stream, use a <strong>get_shard_iterator</strong> request to get the first shard iterator for use in your first <strong>get_records</strong> request. For subsequent reads use the shard iterator returned by the <strong>get_records</strong> request in <strong>NextShardIterator</strong> . A new shard iterator is returned by every get_records request in NextShardIterator, which you use in the <strong>ShardIterator</strong> parameter of the next get_records request.</p><h3>Using Pymapd to insert data into OmniSci</h3><p>The pymapd client interface provides a Python DB API 2.0-compliant interface to OmniSci. Using pymapd,you can perform typical database operations like creating a table, appending data and running SQL queries on OmniSci. In addition, pymapd provides the ability to return results in the Apache Arrow-based <a href="https://rapids.ai/">GDF format</a> for efficient data interchange as part of a GPU-based machine learning workflow.</p><p>You can install pymapd from <strong>conda-forge</strong></p><blockquote><strong>conda install -c conda-forge pymapd</strong></blockquote><p>Or with<strong> pip</strong></p><blockquote><strong>pip install pymapd</strong></blockquote><p>In the sample application, we use <a href="https://www.omnisci.com/cloud">OmniSci Cloud</a> to store the gas production data read from the Kinesis data stream as discussed in the above section.</p><p>First, create a connection to the OmniSci cloud instance. Refer to <a href="https://www.omnisci.com/blog/using-pymapd-to-load-data-to-omnisci-cloud/">Using pymapd to Load Data to OmniSci Cloud</a> for details on generating API keys needed for the connection.</p><pre><strong>from pymapd import connect</strong></pre><pre><strong>user_str = &#39;API Key Name&#39;<br>password_str = &#39;API Key Secret&#39;<br>host_str = &#39;use2-api.mapd.cloud&#39;<br>dbname_str = &#39;mapd&#39;<br>connection = connect(user=user_str, password=password_str, host=host_str, dbname=dbname_str, port=443, protocol=&#39;https&#39;)</strong></pre><p>Use the database connection handle with the <strong>load_table</strong> API to load the Pandas dataframe into a database table. The default behavior of the pymapd load_table API is to create the database table if it does not exist, and then load the data into the table.</p><pre><strong>table_name = &#39;natural-gas_production&#39;<br>connection.load_table(table_name, df, preserve_index=False</strong>)</pre><p>Using OmniSci <a href="https://www.omnisci.com/docs/latest/3_immerse_intro.html"><strong>Immerse</strong></a> interface confirm that the table got created and the records got loaded.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OvtwMaYYcjcCxIoEI5zILQ.png" /></figure><p>You can find the Jupyter Notebook with the sample code for the Kinesis data stream application <a href="https://github.com/omnisci/pymapd-workshop/blob/master/pymapd_kinesis_consumer.ipynb"><strong>here</strong></a>. The last code block in the sample Jupyter Notebook continuously loops to read records from the Kinesis Data Stream, processes the records and loads it to the OmniSci table as a Pandas dataframe.</p><p>Happy Streaming!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b87dc2324c3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Launching OmniSci Enterprise on Google Cloud Platform]]></title>
            <link>https://medium.com/@veda.shankar/launching-omnisci-enterprise-on-google-cloud-platform-6fc3adf5b4b5?source=rss-ea8d1256acec------2</link>
            <guid isPermaLink="false">https://medium.com/p/6fc3adf5b4b5</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Veda Shankar]]></dc:creator>
            <pubDate>Thu, 28 Mar 2019 20:10:33 GMT</pubDate>
            <atom:updated>2019-03-28T20:10:33.731Z</atom:updated>
            <content:encoded><![CDATA[<h3>Launching OmniSci Enterprise on GCP</h3><p>In this tutorial I will provide a step-by-step procedure for deploying OmniSci Enterprise Edition software on Google Cloud Platform (GCP). These instructions assume that you have a Google Cloud Platform account. Check <a href="https://console.cloud.google.com/freetrial?page=0">this</a> instruction if you don’t have a GCP account.</p><h3>Creating a Virtual Machine</h3><p>On the GCP Console, select <strong>Compute Engine</strong> and select an existing project or create a new project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nns0IKKyQZ9pYhp4" /></figure><p>Select <strong>Create Instance</strong> for creating the OmniSci virtual machine (VM).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zS4IcOnNlWwpu7YX" /></figure><p>Enter a name for the VM, and select <strong>Customize</strong> under <strong>Machine Type</strong> to choose the flavor of the compute platform which will also impact the initial selection for the region &amp; zone where the VM can be deployed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t4yRYmHW9j7MnujQ" /></figure><p>This is a minimalistic configuration for running OmniSci, you can refer the <a href="http://docs-hoarder.mapd.com/latest/4_hardware_configuration_guide.html">Hardware Configuration Reference Guide</a> for guidance on your production deployment. Change the <strong>Number of GPUs</strong> to <strong>1</strong> and from the GPU Type drop down menu select <strong>NVIDIA Tesla P4</strong>. As the P4 GPUs are available in <strong>us-west2</strong> region and <strong>us-west2-c</strong> zone, make the changes accordingly. Also, increase the number of <strong>CPU cores to 4</strong> and the memory to <strong>15 GB</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XRJq9pvP0hOffnH2" /></figure><p>Change the Boot disk, select <strong>CenOS7</strong> (x86_64) and set the size of the boot disk to <strong>20GB</strong>. The supported operating systems for OmniSci are CentOS/RHEL 7.0 or later and Ubuntu 16.04 or later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jGAbew7-zMtjCPVN" /></figure><p>Enable default access, allow HTTP/HTTPS traffic and create the VM.</p><h3>Firewall Rules</h3><p>Go to the Firewall rules <a href="https://console.cloud.google.com/networking/firewalls/list?_ga=2.227668190.-273789107.1550604043">page</a> in the Google Cloud Platform Console. Click <strong>Create a firewall rule</strong> to allow access to the OmniSci <a href="https://www.omnisci.com/docs/latest/1_using_immerse.html#using-omnisci-immerse">Immerse</a> (browser) and API ports. The caption below shows the firewall settings to access OmniSci, for further details read <a href="https://cloud.google.com/vpc/docs/using-firewalls">GCP Firewall settings</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gTiVCna5r67SgOZS" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Nxh1sDgnMHP-YiK4" /></figure><h3>Installing OmniSci Enterprise</h3><p>After creating the virtual machine, it takes a couple of minutes for the VM to become available indicated by the green icon. You can access the VM using the <strong>external IP</strong> address associated with the network interface (nic0).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NmddYMBNf6buU3Fo" /></figure><p>You can SSH in a browser window using the SSH drop down menu.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/0*qj1dTi6QZilrk5vK" /></figure><p>The document <a href="http://docs-hoarder.mapd.com/latest/4_centos7-yum-gpu-ee-recipe.html">CentOS/RHEL &amp; EE GPU installation with Yum</a> has detailed instruction on installing OmniSci Enterprise using yum. Here is a summary of all the commands for the installation.</p><pre><strong>sudo yum update</strong></pre><pre><strong>sudo reboot</strong></pre><pre><strong>sudo yum install java-1.8.0-openjdk-headless</strong></pre><pre><strong>sudo yum install epel-release</strong></pre><pre><strong>sudo useradd -U -m omnisci</strong></pre><pre><strong>sudo yum install kernel-devel-$(uname -r) kernel-headers$(uname -r)</strong></pre><pre><strong>sudo reboot</strong></pre><p>Install Nvidia CUDA drivers.</p><pre><strong>sudo yum install wget</strong></pre><pre><strong>wget </strong><a href="https://developer.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda-repo-rhel7-10-1-local-10.1.105-418.39-1.0-1.x86_64.rpm"><strong>https://developer.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda-repo-rhel7-10-1-local-10.1.105-418.39-1.0-1.x86_64.rpm</strong></a></pre><pre><strong>sudo rpm — install cuda-repo-rhel7–10–1-local-10.1.105–418.39–1.0–1.x86_64.rpm</strong></pre><pre><strong>sudo yum clean expire-cache</strong></pre><pre><strong>sudo yum install cuda-drivers</strong></pre><pre><strong>sudo reboot</strong></pre><p>Run <strong>nvidia-smi</strong> to verify the drivers are installed correctly and the GPU is recognized.</p><pre><strong>nvidia-smi</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dybwpGYtB6mtt2er" /></figure><p>Firewall settings on Centos.</p><pre><strong>sudo firewall-cmd — zone=public — add-port=6273/tcp — permanent</strong></pre><pre><strong>sudo firewall-cmd — reload</strong></pre><p>Modify the yum repository file to include OmniSci specification.</p><pre><strong>sudo vi /etc/yum.repos.d/CentOS-Sources.repo</strong></pre><pre><em>[omnisci]</em></pre><pre><em>name=&#39;omnisci ee - cuda&#39;</em></pre><pre><em>baseurl=https://releases.omnisci.com/ee/yum/stable/cuda</em></pre><pre><em>enabled=1</em></pre><pre><em>gpgcheck=1</em></pre><pre><em>repo_gpgcheck=0</em></pre><pre><em>gpgkey=https://releases.omnisci.com/GPG-KEY-mapd</em></pre><pre><strong>sudo yum install omnisci</strong></pre><pre><strong>cd ~/</strong></pre><pre><strong>vi .bashrc</strong></pre><pre><em># User specific aliases and functions</em></pre><pre><em>export OMNISCI_USER=omnisci</em></pre><pre><em>export OMNISCI_GROUP=omnisci</em></pre><pre><em>export OMNISCI_STORAGE=/var/lib/omnisci</em></pre><pre><em>export OMNISCI_PATH=/opt/omnisci</em></pre><pre><em>export OMNISCI_LOG=/var/lib/omnisci/data/mapd_log</em></pre><pre><strong>source .bashrc</strong></pre><p>Run the script to create the database data directory in $OMNISCI_STORAGE, the subdirectories for catalogs, data, export and log will be created.</p><pre><strong>cd $OMNISCI_PATH/systemd</strong></pre><pre><strong>sudo ./install_omnisci_systemd.sh</strong></pre><p>Start the OmniSci daemons and also enable it at system startup.</p><pre><strong>cd $OMNISCI_PATH</strong></pre><pre><strong>sudo systemctl start omnisci_server</strong></pre><pre><strong>sudo systemctl start omnisci_web_server</strong></pre><pre><strong>sudo systemctl enable omnisci_server</strong></pre><pre><strong>sudo systemctl enable omnisci_web_server</strong></pre><h3>OmniSci Activation</h3><p>With a web browser connect to <a href="https://www.omnisci.com/docs/latest/1_using_immerse.html#using-omnisci-immerse">Immerse</a> (OmniSci visual platform) using the external IP address of the VM and port 6273 (<strong>http://&lt;VM External IP&gt;:6273</strong>). Copy your license key from the registration email message into the text box and press Apply to validate your OmniSci instance. If you don’t have a license key then register for a <a href="https://www.omnisci.com/platform/downloads/enterprise/">30-day trial</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tNW66vChAh044brf" /></figure><h3>Load and Query Sample Dataset</h3><p>Load the sample flights dataset with 7 million records (option #1) into the database<strong>.</strong></p><pre><strong>cd $OMNISCI_PATH</strong></pre><pre><strong>sudo ./insert_sample_data</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/0*2SxwGmCuXBwt4FcG" /></figure><p>Connect to OmniSci Core with the SQL command line utility (default password is <strong>HyperInteractive</strong>) and run a query against the newly created <strong>flights_2008_7M</strong> table.</p><pre><strong>$OMNISCI_PATH/bin/omnisql</strong></pre><pre><strong>omnisql&gt; </strong><em>SELECT origin_city AS “Origin”, dest_city AS “Destination”, AVG(airtime) AS “Average Airtime” FROM flights_2008_7M WHERE distance &lt; 175 GROUP BY origin_city, dest_city;</em></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*k0sjJqGSf5mskJMl" /></figure><h3>OmniSci Charts</h3><p>Go back to the web browser Immerse session, under DATA MANAGER you will see the flights_2008_7M table we just created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OuckGyLyDXdxv8hZ" /></figure><p>Using DASHBOARDS create a new <strong>choropleth</strong> chart that renders a polygon corresponding to each destination state and colors the states based on the average flight arrival delay time into an airport in that state. The <strong>dest_state</strong> column in flights table is joined with the <strong>abbr</strong> column of <strong>omnisci_states</strong> table (preloaded by default) thus providing the <strong>omnisci_geo</strong> (POLYGONS) column from omnisci_states.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PLQbFetwO8-90Epn" /></figure><p>Make sure to APPLY and SAVE the chart, and you can create other <a href="https://www.omnisci.com/docs/latest/3_creating_a_dashboard.html">charts types</a> in the dashboard and explore the dataset using the Immerse <a href="https://www.omnisci.com/docs/latest/3_creating_a_dashboard.html#cross-filtering-1">Crossfilter</a> feature.</p><p>You can visit the OmniSci <a href="https://community.omnisci.com/home">community forum</a> to learn more and share your experience.</p><p>OmniSci is also available on Google Cloud Platform <a href="https://console.cloud.google.com/marketplace/browse?q=OmniSci&amp;project=omnisci-launcher-public">Marketplace</a>, and offers three instance types based on Nvidia GPUs K80, P100 &amp; V100.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6fc3adf5b4b5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analyze Oil & Gas Big Data with OmniSci Visual Apps]]></title>
            <link>https://medium.com/@veda.shankar/analyze-oil-gas-big-data-with-omnisci-visual-apps-1b10ffac7e04?source=rss-ea8d1256acec------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b10ffac7e04</guid>
            <category><![CDATA[big-data-analytics]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[oil-and-gas-industry]]></category>
            <dc:creator><![CDATA[Veda Shankar]]></dc:creator>
            <pubDate>Fri, 01 Mar 2019 00:09:18 GMT</pubDate>
            <atom:updated>2019-03-01T00:09:18.239Z</atom:updated>
            <content:encoded><![CDATA[<h3>About Energy Market Data Analysis</h3><p>There are many reasons why analyzing energy market data is more difficult, and more important in today’s global economy than in the past. In the US, we have gone from being a net consumer of oil and gas to the largest producer in the world.</p><p><strong>Crude Production Trends</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/0*EFS0pYKPriWJjfVi" /></figure><p><strong>Natural Gas Production Trends</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CigCY2EXEN0lRPSr" /></figure><p>This change is accompanied by many changes in world-wide logistics and distribution of Energy. The developing countries are on the fast track to Electrifying their populations with globally sourced oil and gas products and renewable energy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p1Ve7i7iHGyvgfJ7" /></figure><p>These trends, new US production and the globalization of energy distribution and consumption, have created an increased need for companies and governments to analyze large, complex amounts of data in order to optimize corporate plans and national policy to meet the needs of the future. Additionally, there is an ever increasing need to measure and analyze carbon emission data related to these markets.</p><p>Below is an overview of the types of data used in energy market analysis:</p><ul><li><strong>Technical Analysis:</strong>Similar to Stock Market Analysis. Time series in nature (some Geospatial)</li><li>Futures Price Data (Daily and Intraday ‘tick’)</li><li>Volatility, Interest Rates, Exchange Rates, Options Prices and Related ‘Greeks’</li><li>Over the Counter Swap information (same attributes as above without the ‘tick’ data)</li><li>Physical (Assessed) Prices</li><li><strong>Operational Analysis:</strong>Geospatial and Time-series in nature</li><li>Production Data (Oil and Gas Wells)</li><li>Logistics Data (pipelines, railway, processing plants, refineries, vessels)</li><li>Power Generation and Distribution: power plant fuel consumption, power output</li><li>Geopolitical: National and International policy</li></ul><p>The nature of the data requires the ability to analyze and visualize large amounts of this daily, monthly, and annual time-series data in a geo-spatial context. With most traditional technology approaches, we must choose between analyzing large data-sets without robust visualization or using incomplete subsets of data in order to get rich visualizations. With <a href="https://www.omnisci.com/link/641976975a384791b4474e192510aa13.aspx">OmniSci,</a> the pioneer in GPU-accelerated analytics, we can analyze and visualize the Oil &amp; Gas spatiotemporal datasets without compromising along these dimensions. Omnisci is an excellent platform for ingesting, analyzing and visualizing this big data in its entirety without the need for any pre-aggregation or indexing.</p><h3><strong>Analyzing US Natural Gas Flows</strong></h3><p><strong><em>US Natural Distribution System </em></strong>Natural Gas Production in the United States consists of a series of Processing Plants to extract Methane (Natural Gas) from well production. Once the Natural Gas is extracted, it is injected into the pipelines to be shipped to market centers, power plants, local utilities, etc.</p><p>As you can see from the OmniSci Linemap rendering below, natural gas is transported using an interstate pipeline network. The natural gas is transported from the producing regions using pipelines at high pressure which helps to reduce the volume by almost 600 times and propels it to its destination.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/0*ULZz9_44tuZEAbJz" /></figure><p>US Natural Gas Pipeline Network — Rendered on OmniSci Map</p><h3>Initial Market Data Analysis Case — Natural Gas Pipeline Flow Data</h3><p>All Intrastate Natural Gas Pipelines in the US are required by Federal Energy Regulatory Commission (FERC) to report their metered flows through their pipelines at least 5 times/day. This information can be scraped from the websites and cleaned for analysis of the flow and market implications related to flow.</p><p>There are roughly 42,000 meter points published daily by pipelines in the US. Flow through each meter is generated at least 5 times per day.</p><h3>About the Dataset</h3><p>The gas production dataset consists of meta data and production data sets which are joined for analysis and visualization in OmniSci. Below is a description of the data.</p><p><strong><em>Flow Data Description</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*CVHwB6zI4Pj5s-J_hJp5ng.png" /></figure><p><strong><em>Meta Data Description</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*zuksSHKw2JXToEF-4kKzXg.jpeg" /></figure><p>In this blog, we will show how we used the OmniSci charting library to create a multi-layered point &amp; line map along with bar and time charts using the natural gas production dataset.</p><p>Read the gas production data and meta-data from the CSV files into their respective Pandas DataFrames:</p><pre>df_meta = pd.read_csv(&quot;meter_metadata.csv&quot;)<br>df_measure = pd.read_csv(&quot;meter_measure.csv&quot;)</pre><p>The production metadata has a column named <strong>meter_metaID</strong> which is a unique id for the natural gas production point issued by the Petroleum Institute. Each production data record also carries a column named <strong>meterID</strong> which is the same. We merge the two dataframes using these columns:</p><pre>df = pd.merge(df_measure, df_meta, left_on=meterID,<br>              right_on=meter_metaID, how=&#39;inner&#39;)</pre><p>And finally save the merged dataframe to a CSV file:</p><pre>df.to_csv(&#39;./meter_measure_with_meta.csv&#39;, index=False)</pre><h3>Bulk Insert Data</h3><p>We chose the OmniSci Cloud platform for developing our application which is the fastest way to start using the platform. You can <a href="https://www.omnisci.com/link/65cc8b02a81b456a82aa9e48678eea71.aspx">sign up for an OmniSci Cloud account,</a> and get immediate access to an instance of OmniSci running on GPU in a public cloud.</p><p>The generated CSV file with the detailed gas production data for a 4 month period has 117 million rows and 22 columns. This is a fairly large file at 26 GB uncompressed, and using OmniSci’s bulk insert <strong>COPY FROM</strong> method would be the most performant way to load the data into the database table. Here are the steps to create the table and load the data into an OmniSci Cloud instance:</p><ul><li>Get the first 100 rows from the CSV file and save it as smaller CSV file.</li></ul><pre>head -100 meter_measure_wth_meta.csv &gt; <br>              meter_measure_with_meta_small.csv</pre><ul><li>Using the <strong>DATA MANAGER</strong> -&gt; <strong>Import</strong> -&gt; <strong>Local File</strong> option on Immerse to load the smaller CSV file.</li><li>Make sure that the table is created correctly where all the columns data types are identified correctly.</li><li>Using Immerse SQL EDITOR truncate the table so that we have an empty table with the correct DDL:</li></ul><pre>sql&gt; truncate table</pre><ul><li>Gzip the large CSV file, the 26 GB file compressed to 500 MB!</li><li>Copy the gzip’d CSV to a AWS S3 bucket.</li><li>Using Immerse SQL EDITOR append the data to the empty table from S3:</li></ul><pre>sql&gt; COPY FROM ‘s3://mybucketname/meter_measure_with_meta.csv.gz’</pre><p>Confirm that the data is loaded correctly by creating some sample charts using Immerse <strong>DASHBOARD.</strong> As you gather more data, you can establish a workflow by copying the CSV file to S3 and then executing the <strong>COPY FROM</strong> command to append to the existing table.</p><h3>OmniSci Charting</h3><p>OmniSci provides <a href="https://github.com/omnisci/mapd-charting.git"><strong>mapd-charting</strong></a> — a superfast charting library that is based on <a href="https://github.com/dc-js/dc.js"><strong>dc.js</strong>,</a> and is designed to work with<strong> mapd-connector</strong> and<strong> mapd-crossfilter</strong> to create charts instantly using OmniSci’s Core SQL Database as the backend. If you are curious about the MapD prefix, you can learn more about the rebrand from <a href="https://www.omnisci.com/link/3c1dba694cb347f28e7236ff9d8d4a9a.aspx"><strong>MapD to OmniSci</strong>.</a> In mapd-charting, you will find sample code for creating frontend rendered charts like Bar, Line and Scatterplot. The sample code also shows how you can create backend rendered charts like Point, Line and Poly Maps in multiple layers using OmniSci’s powerful GPU-based engine.</p><h3>Install mapd-charting</h3><p>Get the source code:</p><pre>git clone <a href="https://github.com/omnisci/mapd-charting.git">https://github.com/omnisci/mapd-charting.git</a></pre><p>Install Dependencies:</p><pre>cd mapd-charting yarn install</pre><p>Test a simple cross-filtered chart example:</p><pre>cp example/example1.html example/index.html yarn run start</pre><p>Open a browser to <a href="http://localhost:8080,">http://localhost:8080,</a> and you should see three cross-filtered charts that use the sample flights dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kAEbBve4HxohF-ve" /></figure><p>The function init() in<strong> example1.html </strong>shows how to connect to the OmniSci backend database, here it is connecting to the host metis.mapd.com which is a OmniSci server open to the public.</p><h3>Testing mapd-charting with your Cloud instance</h3><p>To test the charting API with your OmniSci Cloud instance, use the following init() function that will work with the standard cloud instance using the preloaded <strong>flights_2008_7M</strong> table. You will need to set the user and password variables by creating the write API key under the Developer tab in the Cloud Settings interface as shown in the caption below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WqPC0wU8z-uF8Fot" /></figure><pre>function init() { <br>  /* Before doing anything we must set up a mapd connection,<br>   * specifying username, password, host, port, and database name.<br>   */ <br>  new MapdCon() <br>    .protocol(&quot;https&quot;) <br>    .host(&quot;use2-api.mapd.cloud&quot;) // Connect to OmniSci Cloud Host<br>    .port(&quot;443&quot;) <br>    .dbName(&quot;mapd&quot;) // Default database <br>    .user(&quot;API Key Name from cloud instance SETTINGS&quot;) <br>    .password(&quot;API Key Secret from cloud instance SETTINGS&quot;)<br>    .connect(function(error, con) { <br>      /* <br>       * This instantiates a new crossfilter. <br>       * Pass in mapdcon as the first argument to crossfilter, then<br>       * the table name, then a label for the data (unused in this<br>       * example). <br>       */<br>      crossfilter.crossfilter(con,&quot;flights_2008_7M&quot;).then(<br>        createCharts) <br>      /* <br>       * Pass instance of crossfilter into createCharts. <br>       */<br>    }); <br>  }</pre><p>The three charts (bar, bubble and timeline) use the columns (dest_state, carrier_name, depdelay, arrdelay, dep_timestamp) from the flights_2008_7M table. You can apply filters by clicking on a particular state on the bar chart, or selecting an airline on the scatter plot or dragging the time brush on the time chart. You will notice that all the charts will automatically redraw, applying the selected filters using the crossfilter functions.</p><h3>Oil &amp; Gas Custom App</h3><p>For monitoring the natural gas production in the US” we will create a custom Javascript application based on mapd-charting. In the custom app, we want to see the different natural gas measuring points, the gas flow over time and the gas flow based on the US region. As the US Natural Gas Pipeline information is available from the department of energy, we can layer that on top of the gas production points. The pipeline information is available as ESRI Shapefile and we can import it into OmniSci using Immerse -&gt; DATA MANAGER -&gt; Import option. The pipelines are represented as LINESTRING which is imported as the omnisci_geo column. The sample code in <a href="https://github.com/omnisci/mapd-charting/blob/master/example/exampleMultiLayerMap.js"><strong>exampleMultiLayerMap.js</strong></a> shows how to create a backend rendered map widget with multiple layers using pointmaps. The sample code in <a href="https://github.com/omnisci/mapd-charting/blob/master/example/exampleLinemap.html"><strong>exampleLinemap.html</strong></a> shows how to create a backend rendered geo linemap. For further documentation you can refer to Dimensional Charting (<strong>dc.js</strong>) <a href="https://github.com/dc-js/dc.js/blob/master/web/docs/api-latest.md">API docs</a>.</p><p>The Pointmap is rendered using the Longitude and Latitude columns from the gas production table where the size of the point is based on the flow_value and the points are colored based on the FlowDirection (Delivery, Receipt, Throughput, etc.). The Linemap is rendered using the omnisci_geo (LINESTRING) column from the US pipeline table, and the lines are colored based on the TYPEPIPE column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*47DhhbiV_p1usDTN" /></figure><p>The sample code in <a href="https://github.com/omnisci/mapd-charting/blob/master/example/example1.html"><strong>example1.html</strong></a><strong> </strong>shows how to render bar and time charts, and also shows how to use the crossfilter API. Refer to <a href="https://github.com/square/crossfilter/wiki/API-Reference"><strong>Crossfilter</strong></a> API Reference for more details. We create the time chart with the flow_date on the x-axis and the SUM of flow_value on the y-axis. The bar chart is created using the Region column and the width of the bar represents the SUM of the flow_value. You can check out my <a href="https://github.com/omveda/oil_and_gas_apps.git">code</a> and customize it for your application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JVFiVhMG00cevREX" /></figure><h3>Conclusion</h3><p>At i2 Enabled, Inc. we have a Natural Gas production monitoring and analysis platform that provides different visualizations for time-series data. By incorporating OmniSci Core database we can scale the amount of data by extending to multiple Geos and for longer time periods. At the same time by using the OmniSci Render we can also visualize the SQL query results in milliseconds and present the charts alongside our existing application. User workflow for query and visualization is reduced from minutes with alternative platforms, to immediate analysis and visualization using the OmniSci platform. If you want to see the application in action and meet us in person, stop by our booth at <a href="http://rice2019oghpc.rice.edu/"><strong>Rice University’s Oil &amp; Gas HPC Conference</strong></a> in Houston.</p><p><em>Originally published at </em><a href="https://www.omnisci.com/blog/analyze-oil-and-gas-big-data-with-omniSci-visual-apps/"><em>www.omnisci.com</em></a><em> by Alan Lipe (i2Enabled Inc.) and Veda Shankar (OmniSci Inc.).</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b10ffac7e04" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>