<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Khushboo Sharma on Medium]]></title>
        <description><![CDATA[Stories by Khushboo Sharma on Medium]]></description>
        <link>https://medium.com/@ksharma161076?source=rss-9e30734de5aa------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*1J18BtSANTvkOAFGr9Wg6A.jpeg</url>
            <title>Stories by Khushboo Sharma on Medium</title>
            <link>https://medium.com/@ksharma161076?source=rss-9e30734de5aa------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 04:57:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ksharma161076/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Proven Techniques to Pass your certification exam of Databricks Associate Developer for Apache…]]></title>
            <link>https://medium.com/@ksharma161076/proven-techniques-to-pass-your-certification-exam-of-databricks-associate-developer-for-apache-1af08be5ee94?source=rss-9e30734de5aa------2</link>
            <guid isPermaLink="false">https://medium.com/p/1af08be5ee94</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[databricks-certification]]></category>
            <category><![CDATA[pyspark]]></category>
            <dc:creator><![CDATA[Khushboo Sharma]]></dc:creator>
            <pubDate>Sat, 17 Feb 2024 04:32:35 GMT</pubDate>
            <atom:updated>2024-02-17T04:32:35.426Z</atom:updated>
            <content:encoded><![CDATA[<h2>Proven Techniques to Pass your certification exam of Databricks Associate Developer for Apache Spark</h2><p>I have recently passed my Databricks certification exam for Apache Spark. In this article I will be sharing with you the steps that I followed to crack my certification along with the courses and mock papers. FYI, I gave the exam in Python language, but with a little tweak to below plan you could prepare for exam in Scala language as well.</p><p>You may be are at two phases currently:</p><h3><strong>First Phase:</strong></h3><p>You don’t know anything about Spark. You are just starting to explore the functionality and capabilities of spark. For you guys I must tell you that at the max 2months of preparation time is needed. Start with a course on basics of Spark. For practicing, create a free account on Databricks community edition. And start practicing the methods and code samples. When you have left with 2weeks for the exam, practice the question papers (shared in my second phase).</p><p>Here is my Youtube link to give you a start:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FfrO47B3_lb4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DfrO47B3_lb4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FfrO47B3_lb4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8c6a5fdb332dca437cd0fb0a2bda95ff/href">https://medium.com/media/8c6a5fdb332dca437cd0fb0a2bda95ff/href</a></iframe><h3><strong>Second Phase:</strong></h3><p>For the ones who have at least few months of hands on experience with Spark, they could take a refresher on the spark architecture topics and directly jump to solving question paper shared in the video below.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FCOLCLpRpwrk%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DCOLCLpRpwrk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FCOLCLpRpwrk%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a0bb2b9e572510f23f2729eef67a385d/href">https://medium.com/media/a0bb2b9e572510f23f2729eef67a385d/href</a></iframe><h3>The 5 Major steps:</h3><p>Below are 5 steps that I feel are the key to any certification exam preparation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JRx-gVgcgnRQEa1l-C6X_Q.png" /></figure><h3>My Story of getting the certificate in just 2weeks:</h3><p>I was in the second phase when I registered for the Databricks Apache Spark certification exam. I had 2 weeks for my preparation before the exam date. Below are steps and methods which I have followed before the exam.</p><p><strong>Week 1:</strong></p><p>Day 1, 2, 3 : I bought this advance Udemy course. I completed this course in three days and made notes of the advance topics which were covered in this e.g. Spark Architecture, Adaptive Query Execution, Dynamic Partition Pruning etc.</p><p>Day 4, 5 : Solved the <a href="https://files.training.databricks.com/assessments/practice-exams/PracticeExam-DCADAS3-Python.pdf">practice exam pdf given at the Databricks</a> website. I scored around 70% in the first attempt. I went through my answers and I learnt the topics for which I gave the wrong answers. I revised those topics. If the questions were related to dataframe API then I opened a Databricks notebook and practiced the scenarios. It helped me see the generated output and gave more confidence. For me in the actual exam around 30–40% questions came directly or indirectly from this paper. So I highly recommend to solve this paper at least thrice before the exam date.</p><p>Day 6,7 : Solved this mock test series on <a href="https://www.udemy.com/course/databricks-certified-developer-for-apache-spark-30-practice-exams/">Udemy</a>. I highly recommend it as the questions are very much relatable with the actual exam questionnaire. In the first attempt in all three papers, I scored 93%, 93%, 66% respectively. The last paper is a bit tough.</p><p><strong>Week 2:</strong></p><p>Day 8,9,10,11 : Solved all the papers on this <a href="https://skillcertpro.com/product/databricks-spark-developer-3-0-exam-questions/">website</a>. After solving each paper I went to check the comments and links for the questions which I answered incorrectly. I don’t recommend this course as I found that almost 5% of the questions from this site were wrong and some were directly copied from the Udemy practice set that I have shared above, so this paper doesn’t worth the money that it cost to buy it. I have shared in my <a href="https://youtu.be/COLCLpRpwrk">youtube</a> video an alternate free link to a different site which is far better . Do check that out!!</p><p>Day 12,13 : Practiced the Databricks practice paper and Udemy practice paper again.</p><p>Day 14: Revised the important topics and gave the exam in the evening of day 14. In the exam I scored 91% and got my certificate withing two days.</p><p>If your preparation is good and in your mock exams (may be in the first , second or the third attempt) you are getting a score of above 90% then you will definitely pass your exam.</p><p>Hope this article helps you to get a plan and start your exam preparation.<br>All the Best!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1af08be5ee94" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GCP BigQuery Optimization Techniques: Partitioning vs Clustering]]></title>
            <link>https://medium.com/@ksharma161076/gcp-bigquery-optimization-techniques-partitioning-vs-clustering-f13c0bdf915a?source=rss-9e30734de5aa------2</link>
            <guid isPermaLink="false">https://medium.com/p/f13c0bdf915a</guid>
            <category><![CDATA[performance-tuning]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[partitioning]]></category>
            <category><![CDATA[clustering]]></category>
            <dc:creator><![CDATA[Khushboo Sharma]]></dc:creator>
            <pubDate>Sun, 15 Oct 2023 11:32:28 GMT</pubDate>
            <atom:updated>2023-10-15T11:32:28.183Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/373/1*GMtwC5hyIJhiveCS8-gy_g.png" /><figcaption>Google Cloud Platform — BigQuery</figcaption></figure><p><em>Google Cloud’s BigQuery stands out as a highly potent and cost-effective enterprise data warehouse. It truly shines when tasked with processing and analyzing enormous amounts of data, often in the petabyte range. However, it is not always about the tool that you are using, the most important part is using the tool wisely, otherwise it could incur huge loss. When you start your journey with BigQuery, it is very important to know about the optimization techniques.</em></p><p>In this article, I will be explaining the most used and highly recommended BigQuery optimization methods: <strong>Partitioning and Clustering</strong>.</p><h3>Partitioning a BigQuery Table:</h3><p><em>Partitioning a table means dividing the table into smaller segments/partitions which makes it easier to manage and query data from the table. You have to choose a column, then create the partition on the chosen column and when you write your SQL statement, remember to make use of a partitioned column in your filter condition which could reduce the bytes read by your query and thereby reduce cost of BigQuery.</em></p><p>In general, partitioning a table is a very old technique that has been around since the time of SQL Server, Oracle, and other traditional datawarehouse. If you have ever used any of these traditional datawarehouse, then you must be aware of partition tables. The meaning of the partition table in BigQuery is also the same. However, making use of partitioning on a large BigQuery table is a must, because the cost of using BigQuery depends on the number of bytes read by your query, so if you have applied proper partitioning on your large table, then reading data from that table based on the partitioned column can save a lot of money for you.</p><p>The below image, which I have extracted from <a href="https://cloud.google.com/blog/topics/developers-practitioners/bigquery-explained-storage-overview">Google Docs</a> describes how partitioning creates separate small tables from the original table. Here, partitioning is done on the column Creation_date; hence, you can see separate blocks with the same dates in a single block.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*oPslzFNKZiV2znyQfguPCQ.png" /><figcaption>How Partition Works: <a href="https://cloud.google.com/blog/topics/developers-practitioners/bigquery-explained-storage-overview">Reference</a></figcaption></figure><p>When you are going to fire any query on the above partitioned table with filter condition on Creation_date=’2018–03–02&#39;, it will not have to scan the whole table, instead, it will go to the requested Creation_date partition (20180302) and fetch the result. This will make your query execution fast and minimize query costs.</p><p>Below is the code which can be used to create a partitioned table in BigQuery:</p><pre>-- Create a partitioned table<br>CREATE OR REPLACE TABLE `your_project_id.your_dataset_id.your_table_id`<br>PARTITION BY date_column<br>AS<br>SELECT<br>  *<br>FROM<br>  `your_project_id.your_dataset_id.source_table`;</pre><p>In the code example above, we’re creating a partitioned table based on the `date_column`. This means that the table will be partitioned by date column.</p><p>Now let’s understand the second technique called <strong>Clustering</strong>:</p><h3>Clustering in BigQuery:</h3><p><strong>Clustering</strong> is another optimization technique in BigQuery that complements partitioning. In BigQuery, clustering helps in automatically organizing the data based on the clustered column/columns. Clustering is usually performed on columns with high cardinality and can be done on a partitioned table as well, to get the maximum performance increase of your queries.</p><p>A clustered column sorts the storage blocks of a BigQuery table based on the values in the clustered columns. Queries that include the clustered column in their filter or aggregation, only have to scan the relevant blocks containing the data in the clustered column. Let us understand this from the picture below which I have extracted from <a href="https://cloud.google.com/bigquery/docs/clustered-tables">google docs</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*ZO_2SotjuSD8fWr0t_uQjg.png" /><figcaption><a href="https://cloud.google.com/bigquery/docs/clustered-tables">Example for explaining clustering</a></figcaption></figure><p>The very first table in the above picture is an example of non clustered ‘Orders’ table. The second table describes how data is organized based on countries when the Order table is clustered by Country column. Each block has specific country details. So now when you query this table and you want to see data of only country ‘UK’, your query will not have to scan the entire table, instead, it will only scan the block with ‘UK’ country.</p><p>The third table of the above picture explains clustering for multiple columns (Country and Status). When you cluster a table using multiple columns, the column order determines which columns take precedence when BigQuery sorts and groups the data into storage blocks.</p><p>Here’s how you can create a clustered table in BigQuery:</p><pre>-- Create a clustered table<br>CREATE OR REPLACE TABLE `your_project_id.your_dataset_id.your_table_id`<br>CLUSTER BY product_id<br>AS<br>SELECT<br>  *<br>FROM<br>  `your_project_id.your_dataset_id.source_table`;</pre><p>In the above code, we are creating a table that is both partitioned by `date_column` and clustered by `product_id`. This combination of partitioning and clustering can significantly enhance query performance.</p><h4>Clustering on Partitioned Table (Combine Clustering and Partitioning)</h4><p>Clustering can also be done on a partitioned table to get the maximum optimization benefits. Take the table that we used for partitioning, let’s now create a cluster on the column ‘Tags’. In the below <a href="https://cloud.google.com/blog/topics/developers-practitioners/bigquery-explained-storage-overview">image</a>, you can see how the data is organized by Tags inside each Creation_date partition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*OZ9ZsX4RqvcSHPFdK2eqwg.png" /><figcaption><a href="https://cloud.google.com/blog/topics/developers-practitioners/bigquery-explained-storage-overview">Combining Partitioning and Clustering on a table</a></figcaption></figure><p>Below is the code snippet for creating a Partitioned and Clustered Table:</p><pre>--Creating a Partitioned and Clustered Table<br>CREATE OR REPLACE TABLE `your_project_id.your_dataset_id.your_table_id`<br>PARTITION BY<br> DATE(creation_date)   --column name for partition<br>CLUSTER BY<br> tags AS               --column name for clustering<br>SELECT<br> *<br>FROM<br> `your_project_id.your_dataset_id.source_table`<br>WHERE<br> creation_date BETWEEN &#39;2018-01-01&#39; AND &#39;2018-07-01&#39;;</pre><h3>When to use Partitioning:</h3><p>Think about partitioning a table in the following scenarios:</p><ol><li>The query performance can be improved by merely scanning a table section.</li><li>If your table operation exceeds the regular table storage quota, you can limit the table operations to particular partition column values to increase the partitioned table quota.</li><li>Before running a query on a table or partitioned table, BigQuery presents query cost estimates. Calculating a query cost estimate by pruning a partitioned table, then issuing a query dry run to estimate query costs.</li></ol><h3>When to use Clustering:</h3><ol><li>If your queries frequently commonly use filters or aggregation against multiple columns.</li><li>Clustering speeds up searches that <strong>filter on columns with many different values</strong> (high cardinality). Hence clustering will be best fit for columns with high cardinality.</li><li>Clustering allows your table’s underlying storage blocks to be sized adaptively based on the table’s size.</li><li>In the Clustered table, you will not find the query cost estimates before executing the query. So you can choose to cluster when you don’t need strict cost estimates before query execution.</li></ol><h3>Conclusion:</h3><p>Partitioning and clustering your BigQuery tables is critical for effective data analysis and cost control. These strategies can improve query efficiency and save expenses when working with any kind of large dataset. You can unleash the full potential of Google BigQuery by carefully selecting your partition key and clustered columns, monitoring query performance, and understanding the individual use cases for each strategy.</p><p>Read more about Partitioning and Clustering on google documentation. Experiment with partitioning and clustering as you continue to work with BigQuery to determine the best technique for your data. Remember that aligning your table structure with your query patterns is the key to good optimization. You’ll be well on your way to becoming a BigQuery optimization expert with practice and the tips shared in this article.</p><h4>References:</h4><ul><li><a href="https://cloud.google.com/blog/topics/developers-practitioners/bigquery-explained-storage-overview">BigQuery explained: Storage overview, and how to partition and cluster your data for optimal performance | Google Cloud Blog</a></li><li><a href="https://cloud.google.com/bigquery/docs/clustered-tables">Introduction to clustered tables | BigQuery | Google Cloud Documentation</a></li><li><a href="https://cloud.google.com/bigquery/docs/partitioned-tables">Introduction to partitioned tables | BigQuery | Google Cloud</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f13c0bdf915a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Convert Tabular Data to Graph Data: A Beginner’s Guide to Start GNN from Tabular Dataset]]></title>
            <link>https://medium.com/@ksharma161076/convert-tabular-data-to-graph-data-a-beginners-guide-to-start-gnn-from-tabular-dataset-39f2f518da96?source=rss-9e30734de5aa------2</link>
            <guid isPermaLink="false">https://medium.com/p/39f2f518da96</guid>
            <category><![CDATA[graph-data]]></category>
            <category><![CDATA[gnn]]></category>
            <category><![CDATA[tabular-data]]></category>
            <category><![CDATA[convert]]></category>
            <dc:creator><![CDATA[Khushboo Sharma]]></dc:creator>
            <pubDate>Sat, 27 May 2023 06:46:45 GMT</pubDate>
            <atom:updated>2023-05-27T06:46:45.597Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Graph Neural Networks (GNNs) have become increasingly popular in recent years due to their ability to model complex relationships between data points. But to start your work with GNN, you need to have your data in graph structure. And this becomes one of the challenges to apply GNN on your project as almost all the data in industries are present in simple tabular structure or as relational database. I have also faced this problem when I started with GNN. In this article, I will guide you through the process of converting relational database to graph database so that you can leverage the power of GNNs for your own project.</em></p><p>Note before starting GNN on any dataset, it is a must to <strong>have relationship between the data points</strong>. It’s not always clear in advance, and so starting with exploratory data analysis can be helpful as a first step. But if you have a relational database, it is clear that relationship is present between the data points. Now let’s move assuming you have a relational database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Xlgds-k9gwRcYMnk" /><figcaption>Photo by <a href="https://unsplash.com/@nasa?utm_source=medium&amp;utm_medium=referral">NASA</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In a Tabular dataset the data is present in the form of rows and columns. Each row represents a unique record or observation, while each column represents a variable or feature of that observation. For example, a tabular dataset of customers may include columns such as name, age, gender, and purchase history. A relational database (RDBMS) has multiple tables where related data is present in the format of rows and columns in different tables. The relationship between the tables is defined by primary and foreign keys. Let’s understand this with an example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*6iZju6HsV_DIVrWgH-lSzw.png" /><figcaption>Relational Table (image by author)</figcaption></figure><p>The above image shows an extract of tables from RDBMS database. ‘Customer’ Table has customer id (cust_id) and customer name (cust_name) whereas ‘Sales_Channel’ table has sales channel id (S_ch_id) and sales channel name (Ch_name). If we need to find from which channel a customer was onboarded, we will have to query the row representing the unique id of the customer from ‘Customer’ table (say 12 for Eva from the above example). Then join with an associative entity table (Cust_Channel) which has the foreign keys customer id (cust_id) and its respective sales channel id (S_ch_id). Finally, we would have to query the Sales_Channel table with sales channel id (118) that we have got from the cust_channel table, to get the actual names of the sales channels.</p><p>Now we want the above scenario to be in the form of Graph structure. But to convert tabular data structure to graph, we first need to know what forms a graph.</p><p>A Graph Data is defined by nodes and the relationship(edges/links) between different nodes. So, we can say a graph is formed by nodes which are the entities of a graph and the relationships between different nodes. In the context of tabular data, each row represents a node, and the edges between them represents their relationships.</p><p>Below are some general steps which we could follow to start with converting relational data. The following points also helps to compare key components of graph with relational data structure. These points are taken reference from <a href="https://neo4j.com/developer/relational-to-graph-modeling/">Model: Relational to Graph — Developer Guides (neo4j.com)</a></p><ul><li><em>Table to Node Label</em> — each entity table in the relational model becomes a label on nodes in the graph model. From our above example Customer and Sales_Channel are node label.</li><li><em>Row to Node</em> — each row in a relational entity table becomes a node in the graph. Each customer and each sales channel becomes a node e.g. Eva, Online, Website etc.</li><li><em>Column to Node Property</em> — columns (fields) on the relational tables become node properties in the graph. e.g., id, name. If there are any property of customer in the Customer table like age will also be a node property.</li><li><em>Join tables to Relationships</em> and <em>Foreign keys to Relationship’s properties</em>— join tables are transformed into relationships, columns on those tables become relationship properties. Cust_Channel table will be replaced as relationships. And cust_id , S_ch_id will be properties of the relationships.</li><li><em>No defaults</em> — remove data with default values, no need to store those.</li><li><em>Clean up data</em> — duplicate data in denormalized tables might have to be pulled out into separate nodes to get a cleaner model.</li></ul><p>Let’s apply above key components to our customer-sales channel example and see how we transform table to graph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*gpd2n-qB_4gkGDNnVgXIDw.png" /><figcaption>Graph Structure (image by author)</figcaption></figure><p>Now we have the data in graph structure ready, we could use it to preprocess for training GNN. The preprocessing steps include:</p><ol><li>Assigning a unique ID to each node: Each node in the graph should have a unique ID, which is a requirement for many GNN libraries.</li><li>Creating the adjacency matrix: The adjacency matrix is a matrix representation the graph that shows the connections between the nodes. In our example, the adjacency matrix would show the directed edges between the customer and sales channel.</li><li>Creating the feature matrix: The feature matrix contains the features or attributes of each node. In our example, if we had age and gender, these would be our feature matrix of each customer.</li></ol><p>Once we have preprocessed the graph data, we can train a GNN on it. The GNN will learn to extract the features of the nodes and the relationships between them. It can then use these features to make predictions or classify nodes based on their attributes.</p><p>In this article, we’ve explored how to create a graph data from a simple tabular dataset for working with GNN. We’ve learned that we need to define the nodes and edges to create the graph and preprocess the data to prepare it for training a GNN. Hope this article takes you one step forward towards you GNN journey.</p><p><strong>References:</strong></p><p><a href="https://neo4j.com/developer/relational-to-graph-modeling/">Model: Relational to Graph — Developer Guides (neo4j.com)</a></p><p><a href="https://www.youtube.com/watch?v=uhbj9_xOFX8">How to transform tabular data into a graph (walk-thru) — YouTube</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39f2f518da96" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Detail explanation on Node Classification using R-GCN on AIFB dataset (StellarGraph implementation)]]></title>
            <link>https://medium.com/@ksharma161076/detail-explanation-on-node-classification-using-r-gcn-on-aifb-dataset-stellargraph-implementation-8d76b44ffc6e?source=rss-9e30734de5aa------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d76b44ffc6e</guid>
            <category><![CDATA[rgcn]]></category>
            <category><![CDATA[gnn]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[gcn]]></category>
            <category><![CDATA[mls]]></category>
            <dc:creator><![CDATA[Khushboo Sharma]]></dc:creator>
            <pubDate>Thu, 20 Apr 2023 15:51:44 GMT</pubDate>
            <atom:updated>2023-04-20T15:51:44.670Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Relational Graph Convolution Network (RGCN) is an extended implementation of Graph Convolution Network (GCN). In GCN, the graph network doesn’t take into account the relationships between entities in a knowledge base which means that the edges between the nodes are not taken into account. But in real world scenarios the relationship (which are the edges in a graph) have their own embeddings which are needed to be learned in order to give state of the art results. There can also be multiple edges between a pair of nodes. R-GCN is built on this whole idea of taking into account the embeddings generated through the relationships between nodes.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CYzatM_6Sh409sRL" /><figcaption>Photo by <a href="https://unsplash.com/@deepmind?utm_source=medium&amp;utm_medium=referral">DeepMind</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this article, I will be showing you how we can utilize the concept of R-GCN to perform node/entity classification. This concept was first introduced in a paper called <a href="http://Modeling Relational Data with Graph Convolutional Networks (arxiv.org)">Modeling Relational Data with Graph Convolutional Networks</a> which was a joint effort of Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, Max Welling.</p><p><em>Node classification simply means identifying the missing attributes of a node.</em></p><p>We are going to use the dataset from “<a href="https://en.wikiversity.org/wiki/AIFB_DataSet">Institute for Applied Informatics and Formal Description Methods (AIFB)</a>” which was used in above mentioned paper. The AIFB dataset describes the AIFB research institute in terms of its staff, research group, and publications. It contains ~8k entities, ~29k edges, and 45 different relationships or edge types.</p><p>In (Bloehdorn et al 2007) the dataset was first used to predict the affiliation (i.e., research group) for people in the dataset. The dataset contains 178 members (person) of a research group with 5 different research groups (label_affiliation). The goal is to predict to which group does a member belong. Below is a screenshot of the dataset from this <a href="https://github.com/tkipf/relational-gcn/blob/master/rgcn/data/aifb/completeDataset.tsv">GitHub</a> repo.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*45RHzx156Z7tHVMxOPo22g.png" /><figcaption>Screenshot of Dataset</figcaption></figure><p>Affiliation is the relationship between researcher and research group. It means whether a researcher is affiliated with a research group. It is not included in the graph as we are going to see if we could recover the missing relationship from our model.</p><p>So, now let’s quickly start with coding part. The following part of the article will give you an explanation on how to load the dataset for performing GNN on it, model creation and training and lastly will see how our model is performing on test dataset.</p><h4>Loading the required Libraries:</h4><pre># First do All the necessary imports<br>from rdflib.extras.external_graph_libs import *<br>from rdflib import Graph, URIRef, Literal<br><br>import networkx as nx<br>from networkx.classes.function import info<br><br>import stellargraph as sg<br>from stellargraph.mapper import RelationalFullBatchNodeGenerator<br>from stellargraph.layer import RGCN<br><br>import numpy as np<br>import matplotlib.pyplot as plt<br>import os<br>import pandas as pd<br><br>import tensorflow as tf<br>from tensorflow import keras<br>from tensorflow.keras.layers import Dense<br>from tensorflow.keras.models import Model<br><br>import sklearn<br>from sklearn import model_selection<br>from collections import Counter<br>from stellargraph import datasets<br>from IPython.display import display, HTML<br>import matplotlib.pyplot as plt<br>%matplotlib inline</pre><p><em>If you get error while importing any of the above libraries, then install that library first. In my case, I had to run below code before being able to import them.</em></p><pre>!pip install rdflib<br>!pip install chardet  ##it was required for stellargraph<br>!pip install stellargraph</pre><h4>Loading the Dataset</h4><p>The AIFB dataset is already in its usable structure under a class in stellargraph.datasets. Which means you don’t have to perform EDA on the dataset to build the model, you just need to import it and call the class AIFB() in our script. If you want to understand the code for class AIFB(), you could go to GitHub page of <a href="https://github.com/stellargraph/stellargraph/tree/master/stellargraph/datasets">stellargraph</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*9Q21nny4jxkvGb3McO1Wiw.png" /></figure><pre>dataset = datasets.AIFB()<br>display(HTML(dataset.description))<br>## Here the load method will produce the graph and one hot encoded affiliation<br>G, affiliation = dataset.load()<br>## print and see the structure of your graph<br>print(G.info())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s5KLu-qvG1tzr-IJFDu_3g.png" /><figcaption>Graph Structure Information</figcaption></figure><p>Above code will load the AIFB dataset and create graph G. If you read the graph information you will see the graph is a Directed multigraph with 8285 nodes, 29043 edges, 8285 features with multiple directed edge types.</p><h4>Input Preparation</h4><p>The nodes don’t natively have features, so they’ve been replaced with one-hot indicators to allow the model to learn from the graph structure. We’re only training on the people with affiliations, so we split that into train and test splits.</p><pre>## splitting dataset into train(80%) and test(20%)<br>train_targets, test_targets = model_selection.train_test_split(<br>    affiliation, train_size=0.8, test_size=None<br>)<br><br>## class for a data generator for use with full-batch models on relational graphs e.g. RGCN.<br>generator = RelationalFullBatchNodeGenerator(G, sparse=True)<br><br>train_gen = generator.flow(train_targets.index, targets=train_targets)<br>test_gen = generator.flow(test_targets.index, targets=test_targets)</pre><p>“RelationalFullBatchNodeGenerator” class is a data generator used when there are multiple edge types, or it is a relational graph. The input graph (G) for this class should be a StellarGraph or StellarDiGraph object with node features. The “flow” method requires nodes and optionally targets to get an object that can be used to feed into a Keras model. We have generated two objects ‘train_gen’ and ‘test_gen’ which will be used during model training.</p><h4>Model Creation and Training</h4><pre>rgcn = RGCN(<br>    layer_sizes=[32, 32],<br>    activations=[&quot;relu&quot;, &quot;relu&quot;],<br>    generator=generator,<br>    bias=True,<br>    num_bases=20,<br>    dropout=0.5,<br>)</pre><p>We are using “RGCN class” from stellargraph to create a ‘rgcn’ object in the above code snippet. This creates a stack of Relational Graph Convolutional layers that implements a relational graph convolution neural network model. The model minimally requires specification of the layer sizes as a list of int corresponding to the feature dimensions for each hidden layer, activation functions for each hidden layers, and a generator object. In above code snippet we are passing a 2 hidden layer of size 32 each. The activation functions used for each hidden layer is ‘relu’. The other arguments passed are ‘num_bases’ which specifies number of basis matrices to use for the weight matrices of the RGCN layer, ‘bias’ if True means a bias vector is learnt for each layer in the RGCN model.</p><p>Each RGCN layer creates a weight matrix for each relationship in the graph. If “num_bases == 0” these weight matrices are completely independent. If “num_bases != 0” each weight matrix is a different linear combination of the same basis matrices. This introduces parameter sharing and reduces the number of the parameters in the model. You can read about parameter sharing and find the detail mathematical expressions in the paper.</p><pre>x_in, x_out = rgcn.in_out_tensors()<br>predictions = Dense(train_targets.shape[-1], activation=&quot;softmax&quot;)(x_out)<br>model = Model(inputs=x_in, outputs=predictions)<br>model.compile(<br>    loss=&quot;categorical_crossentropy&quot;,<br>    optimizer=keras.optimizers.Adam(0.01),<br>    metrics=[&quot;acc&quot;],<br>)<br><br>history = model.fit(train_gen, validation_data=test_gen, epochs=20)</pre><p>‘in_out_tensors()’ method builds a RGCN model for node prediction. It returns a tuple: (x_in, x_out), where ‘x_in’ is a list of Keras input tensors for the specified RGCN model and ‘x_out’ contains a model output tensor(s) of shape (batch_size, layer_sizes[-1])</p><p>We add a softmax layer to transform the features created by RGCN into class predictions and create a Keras model. Then we train the model on the stellargraph generators.</p><h4>Model Evaluation</h4><pre>sg.utils.plot_history(history)</pre><p>The below graphs show accuracy and loss trend on training and validation data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*AWIU1df6TcCM8It8UdyMAg.png" /></figure><pre>test_metrics = model.evaluate(test_gen)<br>print(&quot;\nTest Set Metrics:&quot;)<br>for name, val in zip(model.metrics_names, test_metrics):<br>    print(&quot;\t{}: {:0.4f}&quot;.format(name, val))</pre><pre>Test Set Metrics:<br>        loss: 0.4342<br>        acc: 0.9167</pre><p>Now we assess the accuracy of our trained model on the test set. As you could see from the above output, our model is giving 92% accuracy on test data. Hence, we can say our model is performing pretty well on this example dataset!</p><h3>Node embeddings</h3><p>In the first part of the article, we predicted the class (research group) in which a researcher will fall, this part of the article focuses on clustering of the affiliations from node embeddings generated from the RGCN graph.</p><p>We will evaluate node embeddings as the activations of the output of the last graph convolution layer in the GCN layer stack and visualize them. In the below code we are coloring nodes by their true subject label. We expect to see nice clusters of researchers in the node embedding space, with researchers from the same group belonging to the same cluster.</p><p>To calculate the node embeddings rather than the class predictions, we create a new model (embedding_model) with the same inputs as we used previously (x_in) but now the output is the embeddings (x_out) rather than the predicted class. Additionally note that the weights trained previously are kept same in the embedding_model.</p><pre>from sklearn.decomposition import PCA<br>from sklearn.manifold import TSNE<br><br># get embeddings for all people nodes<br>all_gen = generator.flow(affiliation.index, targets=affiliation)<br>embedding_model = Model(inputs=x_in, outputs=x_out)<br>emb = embedding_model.predict(all_gen)<br><br>X = emb.squeeze(0)<br>y = affiliation.idxmax(axis=&quot;columns&quot;).astype(&quot;category&quot;)<br><br>#to visualize high-dimensional data<br>if X.shape[1] &gt; 2:<br>    transform = TSNE<br><br>    trans = transform(n_components=2)<br>    emb_transformed = pd.DataFrame(trans.fit_transform(X), index=affiliation.index)<br>    emb_transformed[&quot;label&quot;] = y<br>else:<br>    emb_transformed = pd.DataFrame(X, index=affiliation.index)<br>    emb_transformed = emb_transformed.rename(columns={&quot;0&quot;: 0, &quot;1&quot;: 1})<br>    emb_transformed[&quot;label&quot;] = y<br><br>#visualization of the distribution of embeddings <br>fig, ax = plt.subplots(figsize=(7, 7))<br>ax.scatter(<br>    emb_transformed[0],<br>    emb_transformed[1],<br>    c=emb_transformed[&quot;label&quot;].cat.codes,<br>    cmap=&quot;jet&quot;,<br>    alpha=0.7,<br>)<br>ax.set(aspect=&quot;equal&quot;, xlabel=&quot;$X_1$&quot;, ylabel=&quot;$X_2$&quot;)<br>plt.title(<br>    &quot;{} visualization of RGCN embeddings for AIFB dataset&quot;.format(transform.__name__)<br>)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*TsmJ5rpsAEGR5bY8STyqFQ.png" /></figure><p>Aside from a slight overlap the classes are well separated despite only using 2-dimensions. This indicates that our model is performing well at clustering the researchers into the right groups.</p><p>I hope this article has helped you to better understand Node Classification using R-GCN. If you are interested to know other types of graph implementation from StellarGraph library, you can go to my other <a href="https://medium.com/@ksharma161076/stellargraph-library-for-heterogenous-graphs-f3a8e427100a?source=your_stories_page-------------------------------------">article</a>. Thank you for sticking around till the end.</p><p>References:</p><ol><li><a href="https://stellargraph.readthedocs.io/en/latest/demos/node-classification/rgcn-node-classification.html"><em>Node classification with Relational Graph Convolutional Network (RGCN) — StellarGraph 1.3.0b documentation</em></a></li><li><a href="https://arxiv.org/pdf/1703.06103.pdf"><em>https://arxiv.org/pdf/1703.06103.pdf</em></a></li><li><a href="https://github.com/tkipf/relational-gcn"><em>tkipf/relational-gcn: Keras-based implementation of Relational Graph Convolutional Networks (github.com)</em></a></li><li><a href="https://en.wikiversity.org/wiki/AIFB_DataSet"><em>AIFB DataSet — Wikiversity</em></a></li><li><a href="https://github.com/stellargraph/stellargraph"><em>stellargraph/stellargraph: StellarGraph — Machine Learning on Graphs (github.com)</em></a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d76b44ffc6e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[StellarGraph Library for Heterogenous Graphs]]></title>
            <link>https://medium.com/@ksharma161076/stellargraph-library-for-heterogenous-graphs-f3a8e427100a?source=rss-9e30734de5aa------2</link>
            <guid isPermaLink="false">https://medium.com/p/f3a8e427100a</guid>
            <category><![CDATA[cnn]]></category>
            <category><![CDATA[graph]]></category>
            <category><![CDATA[gnn]]></category>
            <category><![CDATA[machine-learning-ai]]></category>
            <dc:creator><![CDATA[Khushboo Sharma]]></dc:creator>
            <pubDate>Sat, 04 Mar 2023 14:31:25 GMT</pubDate>
            <atom:updated>2023-03-06T04:57:46.158Z</atom:updated>
            <content:encoded><![CDATA[<p><em>In recent years, Graph Neural Networks (GNN) have rapidly gained popularity. The reason for the success of GNN is its state of the art results compared to traditional neural networks(CNN). GNNs are neural networks that can be directly applied to graphs, and provide an easy way to do node-level, edge-level, and graph-level prediction tasks. In this article, I have picked up a library (StellarGraph) for graph machine learning, which is very powerful but yet less explored.</em></p><p><strong>StellarGraph</strong> is a python library that provides scalable algorithms for machine learning on graphs. Graphs are commonly used to represent complex systems such as social networks, biological systems, and transportation networks. It makes discovering patterns from graph very easy. We can solve various kind of problem using this library, some examples are:</p><ol><li>Predicting relationship/link between two nodes (e.g. social networks)</li><li>Classification of node/graph (e.g. fraud detection)</li><li>Interpretation of node classification (e.g. image identification)</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x5T5C5Y8aMruCqcCvBhT3w.jpeg" /><figcaption>Examples of GNN Application (image by Zhang, Liang, Liu and Tang, for reference click on <a href="https://www.frontiersin.org/articles/10.3389/fgene.2021.690049/full">Image Reference</a>)</figcaption></figure><p>StellarGraph supports different kind of graphical representation such as:</p><ul><li><strong>homogeneous </strong>(with nodes and links of one type/kind)</li><li><strong>heterogeneous</strong> (with more than one type/kind of nodes and/or links)</li><li><strong>knowledge graphs</strong> (extreme heterogeneous graphs with thousands of types of edges)</li><li>graphs with or without data(features) associated with nodes</li><li>graphs with edge weights</li></ul><p>In recent years, <strong>Heterogenous Graph</strong> are becoming extremely popular and scientist have proved it to be very efficient in various research papers.</p><p>We call a graph Heterogenous, if we have multiple nodes or multiple links/edges or multiple nodes and multiple edges. For node classification or Link prediction task with multiple nodes, StellarGraph provides an algorithm called Heterogeneous GraphSAGE (<strong>HinSage</strong>) which is a scalable method for inductive representation learning on graphs. It learns a function that maps a node’s features and its local graph structure to a low-dimensional vector representation. Another library is <strong>Relational Graph Convolutional Network (RGCN)</strong> which identifies the relationship/link between different nodes and helps in predicting the desired output.<strong> </strong>Finally,<strong> Metapath2Vec</strong> is another library that learns node embeddings in heterogeneous networks. It uses metapath-guided random walks to generate sequences of nodes that capture the structural semantics of the network.</p><p>As mentioned earlier, we can use HinSAGE when our data has multiple nodes. For example, model which predicts user-movie ratings (Link prediction), Fraud detection(Node classification) with different kinds of nodes (like client, merchant and transaction node). In my next follow up article, I will be writing in depth about using HinSAGE algorithm on user-movie ratings and Fraud Detection (with code snippet).</p><p>In conclusion, StellarGraph and its associated libraries provide powerful tools for machine learning on graphs. Heterogeneous graphs are a useful way to model complex systems, and there are several libraries that can be used to work with these graphs. Additionally, GNNs are a promising approach for representation learning on graphs and have been successfully applied to various domains. I hope that this document has provided a useful introduction to StellarGraph graphs.</p><p>References:</p><p><a href="https://stellargraph.readthedocs.io/en/stable/">https://stellargraph.readthedocs.io/en/stable/</a></p><p><a href="https://neptune.ai/blog/graph-neural-network-and-some-of-gnn-applications">Graph Neural Network and Some of GNN Applications</a></p><p><a href="https://www.frontiersin.org/articles/10.3389/fgene.2021.690049/full">https://www.frontiersin.org/articles/10.3389/fgene.2021.690049/full</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3a8e427100a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Start your Machine Learning Journey Today]]></title>
            <link>https://medium.com/@ksharma161076/start-your-machine-learning-journey-today-ed8a5a8ec705?source=rss-9e30734de5aa------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed8a5a8ec705</guid>
            <dc:creator><![CDATA[Khushboo Sharma]]></dc:creator>
            <pubDate>Sat, 18 Feb 2023 11:35:02 GMT</pubDate>
            <atom:updated>2023-02-18T11:35:02.492Z</atom:updated>
            <content:encoded><![CDATA[<p><em>These days ML or AI are commonly heard or used words, and ChatGPT is the recent new hype around it. We hear everyday new advancement in these areas of technology and forced to think that machines will replace humans. This is not completely true nor false. Because from homes to office or wherever we go, we see many of the manual works getting replaced by machines. But, machines can only learn from past data, so if there are no humans to create data, there will be no way a machine can learn. But humans are unpredictable so they could do what machines can’t do.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6S-SzVPbYXCRlRo1" /><figcaption>Photo by <a href="https://unsplash.com/@tetrakiss?utm_source=medium&amp;utm_medium=referral">Arseny Togulev</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This article is for the ones who are looking to start a career in machine learning and doesn’t know where to start. I will be walking you through some basic concepts and will guide you on what to do next. So let’s start…</p><p><em>“You are never too old to learn something new, or too young to learn too much.”</em></p><h3><strong>What is Machine Learning?</strong></h3><p>If you google this particular question, you will find many theoretical answers to it. But in plain text, machine learning is a field of study in which we are trying to make machines as smart as humans (so that we can chill and relax while machines are doing our chores :-) ).</p><p>In the field of ML, by passing historical data we try to make machine find patterns and tells us best possible outcome. By learning patterns from the ingested data, a machine tries to give us future predictions or desired results for which it was created. Some examples are weather forecasting, fraud detection, recommending movies and many more. Sometimes we get state of the art results, but many a times results are worse (in such case we try harder by trying different experiments).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1yyLJEKlEqFIP-hY" /><figcaption>Photo by <a href="https://unsplash.com/@noaa?utm_source=medium&amp;utm_medium=referral">NOAA</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote>Few terminology used in Machine Learning:</blockquote><blockquote>Initial dataset : Dataset chosen for starting the experiment</blockquote><blockquote>Features : Input values to the machine represented as X</blockquote><blockquote>Labels : Whatever we want to predict (desired output) represented as y</blockquote><blockquote>Model : This is a relationship between X and y . The machine that we are trying to build.</blockquote><blockquote>Training: Process in which we train the model to learn patterns</blockquote><blockquote>Inference: Process in which we test if our model is doing good or bad</blockquote><p>The machine learning study is broadly divided into three parts:</p><ol><li><strong>Supervised Machine Learning:</strong></li></ol><p>If the initial dataset to the model have label, then such type of machine learning is Supervised ML. As a developer we will know the correct answers to the ingested data, but the machine will learn patterns from it which it will use for future predictions. We have three types of Supervised ML: Classification, Regression and Forecasting</p><p><strong>2. Unsupervised Machine Learning:</strong></p><p>This one is used when there is no label present in the initial dataset. It means that the machine has to learn on its own about possible outcomes by finding patterns, differences and similarities in the dataset. This branch of machine learning is a bit difficult when compared to the previous one because the machine has to learn from scratch. This is divided into two types: Clustering and Association.</p><p><strong>3. Reinforcement Learning</strong></p><p>Reinforcement learning is again a kind of learning in which there is no labelled data provided. Here the machine learns from trial and error. A feedback criteria is set where the machine is rewarded for good results and punished for incorrect outputs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jKPd1rIOVe6vFZ25tKatZg.png" /></figure><h3><strong>Conclusion:</strong></h3><p>From above topics you have gained a very basic idea on machine learning, it’s types and where these are used. After this you could pick up any one type of ML (my suggestion is to go with Supervised ML). Take an online course pertaining to your chosen type and then start building your first project. Hands-on is the best way to learn any course involving programming. You could also go to Kaggle for practicing. Below is my course suggestion to start after this.</p><p>Course suggestion (you should know basic python for this) : <a href="https://www.coursera.org/learn/machine-learning?">Supervised Machine Learning: Regression and Classification | Coursera</a></p><p>I will be writing many articles on Data Science field, so please follow and subscribe for receiving notification on my new articles.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed8a5a8ec705" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>