<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ricky Sun on Medium]]></title>
        <description><![CDATA[Stories by Ricky Sun on Medium]]></description>
        <link>https://medium.com/@RickySun?source=rss-48da3986742c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*08FcKgotjbNsd0K2tdKp-w.png</url>
            <title>Stories by Ricky Sun on Medium</title>
            <link>https://medium.com/@RickySun?source=rss-48da3986742c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 07:06:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@RickySun/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Step-by-Step Guide: How to Convert Tables to Graph]]></title>
            <link>https://medium.com/ultipa/a-step-by-step-guide-how-to-convert-tables-to-graph-6cd19da6909c?source=rss-48da3986742c------2</link>
            <guid isPermaLink="false">https://medium.com/p/6cd19da6909c</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[graph]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Ricky Sun]]></dc:creator>
            <pubDate>Fri, 17 Nov 2023 03:21:24 GMT</pubDate>
            <atom:updated>2025-08-14T01:54:43.521Z</atom:updated>
            <content:encoded><![CDATA[<p>By Wanyi Sun</p><h3>Background</h3><blockquote><em>“Even though I’ve mastered many graph query commands, when I started modeling my graph, I still felt like a novice.”</em></blockquote><p>These were the words of one of our clients, reflecting a common sentiment among beginners of graph databases when they transit from a familiar relational database to the world of graph databases.</p><p>While we often stress the importance of query languages in our guidance for newcomers, the primary challenge faced by most users in practical applications isn’t crafting query statements but seamlessly migrating their data to a graph database. This challenge is especially prominent when transitioning from conventional relational databases.</p><p>Whether the user’s data is structured in tables or not, it often cannot be directly converted into graph data. Graph data adheres to specific requirements: nodes must possess unique IDs serving as their identity cards, while edges must have designated FROM and TO, both referencing node IDs within the graph. Only then can we truly appreciate the beauty of an edge, “a magnificent bridge linking nodes together!”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/0*7Dj2L4DcgZUovHi-" /></figure><p>A visualization of graph</p><p>This arduous data migration journey usually involves numerous rounds of fine tuning data schemas[1]. If there were any golden rules or sacred principles in this battle of data transformation, they would be: meticulously focus on the primary keys and foreign keys within the original tables. These keys will become the IDs for nodes and the FROM and TO for edges, making them the pivotal elements for successfully obtaining graph data.</p><p>Let’s delve into a practical example demonstrating how to transform the table structure of a Hospital Information Management System built using SQL Server into a graph model and then import the resulting graph data into an Ultipa graph system.</p><h3>Tables in SQL Server</h3><h3>1. Table Structures</h3><p>In simplifying matters, consider this Hospital Information Management System holding fundamental details about doctors, patients, departments, and hospital beds, alongside records of doctors’ consultations and patients’ hospitalizations. All these data are stored across six distinct tables:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/210/0*izg_gpfDadX4SYXO" /></figure><p>Tables created for a hospital information management system in SQL Server</p><p>Let’s create a diagram to visually depict the relationships among these tables:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6kE2GO9TeftiAw7D" /></figure><p>A clear visualization of table relationships, highlighting the links from Foreign Keys to Primary Keys</p><p>In the diagram provided, the arrows starting from foreign keys and ending at primary keys indicate how tables are interlinked and how SQL join queries are executed. For instance, to gather information about patients, including their names, genders, and diagnoses, seen by a specific doctor during a specific timeframe, one must focus on the field PNO, linking the DIAGNOSIS table with the PATIENT table. For readers well-versed in SQL, this type of table join query is undoubtedly a familiar sight.</p><p>This initial table structure design also has its imperfections. For example, in the DIAGNOSIS table, the doctor’s name DNAME is recorded instead of their unique identifier, DNO. This design flaw results in a missing foreign key, hindering the DIAGNOSIS table from establishing a connection with the DOCTOR table to obtain detailed doctor information. Although remedial approachs could be taken by adding some common information about doctors in the DIAGNOSIS table, such as the department name DEPART, to sidestep the complexities of join queries, the challenges stemming from redundant storage, such as disk space consumption and data consistency checks, cannot be underestimated.</p><p>In the subsequent design of the graph database model, these dilemmas will no longer pose hurdles.</p><h3>2. Data in Tables</h3><p>Let’s input sample data into the aforementioned tables and amalgamate the information from all tables into a singular diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JPxoCgOQJLTmdyXc" /></figure><p>Comprehensive overview of data from the 6 tables in this Hospital Information Management System</p><p>While the tabular representation, though systematic, provides a detailed and transparent overview of individual data fields, it lacks the ability to illuminate the high-dimentional interactions between data records. Questions like “which department’s doctor attended to which patient” or “which patient was allocated to which ward in a specific department” are complex to deduce solely from tables. This intuitive understanding becomes notably challenging when contrasted with the forthcoming graph model we are about to construct.</p><h3>Graph Modeling with Ultipa</h3><h3>1. Distinguish Nodes and Edges</h3><p>Let’s begin by adopting the existing work: conceptualize each table as a schema, with the data fields serving as the properties within that schema. The challenge lies in distinguishing nodes from edges.</p><p>In essence, all tables can be likened to real-world entities, but only the most genuine entities serve as nodes. Therefore, we’ve identified DEPARTMENT, DOCTOR, BED, and PATIENT as nodes, utilizing their primary keys as node IDs (denoted by _id):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tGQBA-SHKM_W7e8S" /></figure><p>Node and Edge Differentiation</p><p>INPATIENT should not be misconstrued as a subset of PATIENT, since it captures the relationship between PATIENT and BED as evidenced by its two foreign keys. Define INPATIENT as an edge and specify its two foreign keys as the FROM and TO of the edge (denoted by _from and _to).</p><p>(Feel free to interchange the FROM and TO of the edge as long as the edge’s meaning remains coherent. For instance, setting PATIENT’s _id as FROM and BED’s _id as TO signifies ‘patient check-in to a bed’, while the reverse suggests ‘bed receives a patient’.)</p><p>Applying a similar rationale, designate DIAGNOSIS as an edge to encapsulate the connection between DOCTOR and PATIENT. In determining its FROM and TO, fill in the missing foreign key linking to DOCTOR by replacing DNAME with DNO. This missing link issue was previously addressed when explaining the table structure and has now been rectified in the edge design phase.</p><blockquote><em>Node IDs are extracted from the original table’s primary key, whereas the FROM and TO for edges are derived from the original table’s foreign keys, supplemented where necessary.</em></blockquote><p>Curious minds might wonder, “Do edges have IDs?” The answer is a resounding yes. In fact, we typically use the primary keys of tables designated as edges as their respective IDs. This logic mirrors the approach for node IDs. However, the focus isn’t placed on edge IDs because, for edges, the designation of FROM and TO holds greater significance.</p><h3>2. Adjust Schemas</h3><p>Two foreign keys in the original tables are still unaddressed: DEPART in the DOCTOR and BED tables, both of which link to the primary key of DEPARTMENT. This raises the question of how to handle foreign keys originate from tables designated as nodes.</p><p>As illustrated in the diagram below, solve these two foreign keys by introducing two new edge schemas, WORKAT and BELONGTO, which equals adding two fresh tables to the original structure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/971/0*WU-KbK7AEBLgCQhp" /></figure><p>Introducing new edge schemas WORKAT and BELONGTO</p><p>The introduction of WORKAT and BELONGTO enables the removal of the DEPART from DOCTOR and BED. As these new schemas are minimalized with only FROM and TO, additional properties can be incorporated on-demand to accommodates real-world scenarios.</p><p>Up to this juncture, all the primary keys and foreign keys in the tables are addressed. One last refinement is to isolate the RNUM field, denoting room numbers, from the BED table into a node schema ROOM, since many hospitals require calculating room occupancy or managing facilities within the room:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/971/0*P2xRVO4_Oe6MdJuN" /></figure><p>Abstracting node schema ROOM and sharing edge schema BELONGTO</p><p>Some residual steps are: removing the redundant RNUM from both the BED and INPATIENT, sharing BELONGTO between BED-ROOM, ROOM-DEPARTMENT, and DOCTOR-DEPARTMENT since all these relationships require only properties FROM and TO.</p><h3>3. Inject Graph Data</h3><p>Now cleanse and transform the tabular data into graph data in alignment with the above established graph model. Assuming each table’s data is stored as an individual CSV file (e.g., DOCTOR’s data saved as DOCTOR.csv) with the field names serving as headers in the file, the transformation process entails header modifications, column content replacements, and the creation of new CSV files. The detailed steps are as follows:</p><ul><li>In DEPARTMENT.csv, PATIENT.csv, DOCTOR.csv and BED.csv, rename headers of columns containing primary keys to ‘_id’.</li><li>In IMPATIENT.csv, rename header ‘PNO’ to ‘_from’, rename header ‘RBNUM’ to ‘_to’, and delete the entire column ‘RNUM’.</li><li>In DIAGNOSIS.csv, rename header ‘PNO’ to ‘_to’, rename header ‘DNAME’ to ‘_from’, and replace the data under ‘DNAME’ with the corresponding data from the ‘_id’ column in DOCTOR.csv.</li><li>Create BELONGTO.csv with headers ‘_from’ and ‘_to’.</li><li>Copy the data under ‘_id’ and ‘DEPART’ in DOCTOR.csv to ‘_from’ and ‘_to’ in BELONGTO.csv, ensuring the corresponding relationships of data in the same row are maintained. Then, delete the entire ‘DEPART’ column in DOCTOR.csv.</li><li>Create ROOM.csv with header ‘_id.’. Copy the unique values under ‘RNUM’ in BED.csv to the ‘_id’ column in ROOM.csv.</li><li>Copy the data under ‘_id’ and ‘RNUM’ in BED.csv to ‘_from’ and ‘_to’ in BELONGTO.csv, ensuring the corresponding relationships of data in the same row are maintained.</li><li>Copy the unique combined values under ‘RNUM’ and ‘DEPART’ in BED.csv to ‘_from’ and ‘_to’ in BELONGTO.csv, ensuring the corresponding relationships of data in the same row are maintained. Then delete the entire ‘RNUM’ and ‘DEPART’ columns in BED.csv.</li></ul><p>As a reminder, it may be necessary to process primary keys to ensure that their associated IDs are unique[2] throughout the entire graph. While this doesn’t apply to the data presented in this article, it is a common challenge encountered by numerous users.</p><p>Multiple approaches are available to <a href="https://www.linkedin.com/feed/update/urn:li:activity:7099200941156962304">import graph data into a designated graphset</a> within the Ultipa graph system. Such as upload CSV files individually through <a href="https://www.ultipa.com/document/ultipa-manager-user-guide/interface-overview/v4.3">Ultipa Manager</a>, a user-friendly visualization tool for managing the Ultipa Graph system, or bulk import all files at once via the command-line utility, <a href="https://www.ultipa.com/document/ultipa-transporter/usage-overview/v4.3">Ultipa Importer</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z9q51WyH00CxCRxb" /></figure><p>Graph model illustrated as schemas and properties in Ultipa Manager</p><blockquote><em>In Ultipa Manager, the ID, FROM and TO are not visible in the schema list. These are system-generated properties that cannot be modified or deleted.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RRSHqDpcQ5IikriX" /></figure><p>2D visualization of graph data in Ultipa Manager</p><p>Ultipa Manager’s 2D rendering feature utilizes vibrant colors and icons, ensuring that relationships between different entities are distinctly visible. A quick glance allows easy identification of the interconnected nodes in the graph. This intuitive and convenient visualization sets graph databases apart, surpassing the capabilities of traditional relational databases. It greatly enhances the ease and pleasure of data analysis and exploration.</p><h3>Conclusion</h3><p>We trust this article has provided valuable insights into graph modeling, especially for newcomers venturing into the world of graph databases. In practical scenarios, constructing graph models involves more intricate situations. On the other hand, the graph model itself is not fixed, adaptable modifications are required to meet the changing business needs all the time.</p><p>Notes:</p><p>[1] Schema: In certain relational databases like MySQL or PostgreSQL, the hierarchical data management structure within a server connection is defined as ‘database-schema-table-field’. In Ultipa graph systems, this translates to ‘graph-schema-property’, where a schema in the Ultipa graph system corresponds to a table in relational databases, and schema properties are akin to table fields.</p><p>[2] In Ultipa, node IDs possess global uniqueness throughout the entire graph, not confined to a specific schema. For instance, if ‘abc’ is the ID for a particular doctor, it cannot be used as the ID for any patient or any other node within the same graph. Conversely, primary keys in relational databases only require uniqueness within their respective tables, allowing different tables to have duplicate primary keys. Consequently, certain primary keys need additional processing before they can be utilized as IDs for graph data, so do associated foreign keys.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6cd19da6909c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ultipa/a-step-by-step-guide-how-to-convert-tables-to-graph-6cd19da6909c">A Step-by-Step Guide: How to Convert Tables to Graph</a> was originally published in <a href="https://medium.com/ultipa">Ultipa Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why SQL Isn’t the Right Fit for Graph Databases]]></title>
            <link>https://medium.com/ultipa/why-sql-isnt-the-right-fit-for-graph-databases-bfaf8d1af765?source=rss-48da3986742c------2</link>
            <guid isPermaLink="false">https://medium.com/p/bfaf8d1af765</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[graph-database]]></category>
            <category><![CDATA[gql]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Ricky Sun]]></dc:creator>
            <pubDate>Fri, 17 Nov 2023 03:19:11 GMT</pubDate>
            <atom:updated>2025-08-14T01:54:26.399Z</atom:updated>
            <content:encoded><![CDATA[<p>By Wanyi Sun</p><h3>Background</h3><blockquote><em>“Why don’t your graph products support SQL or SQL-like query languages?”</em></blockquote><p>This question used to be frequently asked by some of our clients in the past, but has become increasingly rare over time.</p><p>Although once overlooked, the necessity for a graph database to possess a query language seamlessly designed and adapted to its underlying data structure has been conclusively demonstrated by innovative products and embraced by the market. Graph databases exhibit notable differences from relational databases in terms of data model, data storage, performance, and more, tailoring to diverse application scenarios and business objectives. These distinctions affirm that SQL is ill-suited to serve as a query language for a native graph computing engine.</p><p>For a more comprehensive comparison, we will demonstrate SQL queries against a tabular dataset stored in SQL Server, as well as their equivalent queries composed in UQL to run against a graph dataset with the same content loaded into Ultipa. Both datasets were analyzed in another article, <a href="https://www.linkedin.com/feed/update/urn:li:activity:7128656690308145152">A Step-by-Step Guide: How to Convert Tables to Graph</a>.</p><h3>Datasets</h3><p>In case you haven’t had a chance to review it yet, the tabular dataset we’ll be using comprises data from six tables within a Hospital Information Management System: DOCTOR, PATIENT, DEPARTMENT, BED, DIAGNOSIS, and INPATIENT. This dataset is relatively small, consisting of about 60 lines of records. The data fields for each table are outlined below, with a focus on their interconnections (arrows pointing from foreign keys to primary keys):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nOXpNukPDwt6bq-x" /></figure><p>A concise visualization of table structures, accentuating the connections from Foreign Keys to Primary Keys</p><p>The hierarchical data management structure in a SQL Server connection is denoted as ‘database-table-field,’ which can be translated to ‘graph-schema-property’ in an Ultipa graph system. In other words, a schema in the Ultipa graph system corresponds to a table in relational databases, and schema properties are analogous to table fields.</p><p>The graph dataset is derived from the aforementioned tabular dataset by transforming table structures into node schemas (with properties like _id, etc.) and edge schemas (with properties like _from, _to, etc.). This process involves tasks equivalent to modifications, as outlined below, to the original tables:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hoU11nVLq404hTVx" /></figure><p>A comprehensive depiction of the graph model, illustrated through modifications to the original table structures</p><p>The resulting graph dataset encompasses five node schemas: DOCTOR, PATIENT, DEPARTMENT, BED, and ROOM, along with three edge schemas: DIAGNOSIS, INPATIENT, and BELONGTO. Node IDs are extracted from the table’s primary key, while the FROM and TO for edges are derived from the table’s foreign keys, supplemented where necessary.</p><h3>Table Query vs. Schema Query</h3><p>Given that the concept of a table in SQL Server corresponds to that of a schema in Ultipa, let’s begin by comparing the querying process for, say, the DOCTOR table and the DOCTOR node schema.</p><blockquote><em>Query objective: Retrieve all the information for every doctor.</em></blockquote><p>SQL:</p><pre>SELECT * FROM guest.DOCTOR</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/0*Zgdt4TlBUSobiwoZ" /></figure><p>Result of single-table query using SQL</p><p>UQL:</p><pre>n({@DOCTOR} as N) RETURN N{*}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5-OS6b8xUty4ARkd" /></figure><p>Result of single-schema query using UQL</p><p>Although both queries are equally clear and produce identical results, SQL resembles an English sentence requesting to “select all data fields from the DOCTOR table,” while UQL, intentionally deviating from this type of natural language-based syntax, utilizes symbols to convey its execution logic, which is ‘n()’ that signifies a node in the graph.</p><p>This approach aligns with the nature of a graph, where nodes and edges are fundamental components, unlike relational databases where tables exist as the core data model. In the next round, we will encounter ‘e()’ in UQL denoting an edge.</p><h3>JOIN Query vs. Path Query</h3><p>Graph data earns its acclaim by faithfully replicating the complexities of the real world. This is exemplified by the ease with which one can query chained relationships, called path query, between graph data, especially when compared to the JOIN query of relational tables.</p><blockquote><em>Query objective: Retrieve detailed information about all patients treated by Dr. Michael Roberts, including their diagnosis results and the doctor’s name.</em></blockquote><p>SQL:</p><pre>SELECT<br>      PATIENT.PNO,<br>      PATIENT.PNAME,<br>      PATIENT.PSEX,<br>      DIAGNOSIS.DIAGNOSISRESULT,<br>      DIAGNOSIS.DNAME<br>FROM<br>      guest.DIAGNOSIS<br>      JOIN guest.PATIENT ON guest.DIAGNOSIS.PNO = guest.PATIENT.PNO<br>WHERE<br>      DNAME = &#39;James Anderson&#39;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/0*RW7sywcOjGDmtYTn" /></figure><p>Result of table JOIN query using SQL</p><p>UQL:</p><pre>n({@DOCTOR.DNAME == &quot;James Anderson&quot;} AS D)<br>   .e({@DIAGNOSIS} AS A)<br>   .n({@PATIENT } AS P)<br>RETURN table(<br>   P._id,<br>   P.PNAME,<br>   P.PSEX,<br>   A.DIAGNOSISRESULT,<br>   D.DNAME<br>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NA8AD6vP-lHnBNGK" /></figure><p>Result of path query using UQL</p><p>While it may not have been emphasized prominently in the preceding codes for querying a single table or schema, an intriguing observation emerges with these more extensive code excerpts: the syntax expressions of these two query languages are reversed. SQL commences by defining the data to be retrieved and subsequently outlines the filtering conditions the data must satisfy. On the other hand, UQL first delineates the criteria the data should meet and then specifies the data to be retrieved. This reversal in syntax order can impact developers’ thought processes, leading to distinct programming experiences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BQ4_Y0dEDepcrtrp" /></figure><p>Comparison of syntax orders, SQL (left) vs. UQL (right)</p><p>Let’s shift our focus back to the representation of data relationships. SQL uses JOIN to link tables and applies ON to anchor key fields (primary keys, foreign keys). In contrast, UQL employs a path template n().e().n() to systematically define the interconnected nodes and edges, essentially eliminating the need for additional anchoring steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Nf43YIwP2c8Yn-fs" /></figure><p>Comparison of relationship expressions, SQL (left) vs. UQL (right)</p><p>Why can a graph query language, in this case, UQL, achieve such a streamlined expression? This is because the inherent anchoring relationships between tables, namely the links from foreign keys to primary keys, are seamlessly embedded within the edge data (_from and _to). This fundamental integration is precisely why graph databases outperform relational databases in efficiently managing associative relationships.</p><p>Moreover, UQL directly incorporates filtering conditions into every node and edge declaration, the n() and e(), unlike SQL’s WHERE clause, which adds filtering conditions after joining all tables and anchoring. This approach makes UQL syntax more concise, cohesive, and enhances code readability.</p><blockquote><em>UQL ingeniously consolidates every node and edge, along with their filtering conditions, into a natural, chain-like path model.</em></blockquote><p>It’s worth highlighting that although the SQL and UQL snippets above are similar in length and produce identical results, their actual querying functions differ from the bottom. In the example above, SQL only accomplishes the connection between two tables, PATIENT and DIAGNOSIS, retrieving the doctor’s name (DNAME) from the DIAGNOSIS table. In comparison, UQL establishes relationships among PATIENT, DIAGNOSIS, and DOCTOR, essentially performing a JOIN across three tables (schemas). It returns D.DNAME, a property of the node schema DOCTOR. Understanding these underlying distinctions adds irony to the performance comparison: the SQL query took 3ms, whereas the UQL query, handling more data correlations, only took 0.5ms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Y5w4h93JumanvepY" /></figure><p>Comparison of query times, SQL (left) vs. UQL (right)</p><p>The shift from 3ms to 0.5ms might appear inconsequential, but as the number of chained relationships being queried increases, it becomes remarkably pronounced. A recursive 5-table JOIN query in SQL against a small table of 10,000 rows will take 38 seconds, whereas the equivalent query achieved by a 5-step path template in UQL only takes 0.001 second. In more general cases, SQL tends to struggle when executing lengthy chained table JOINs against large datasets. While the inefficient leverage of concurrency does contribute to this substantial distinction, the Cartesian product of table records in all JOIN procedures is a fundamental reason for the exponential level of performance degradation in SQL queries compared to UQL.</p><h3>Tips on Using Graph Query</h3><p>So far, we have been presenting the query results of UQL in a traditional tabular format for the purpose of comparison with SQL. However, we strongly advocate representing the results of path queries on small-scale graphs using 2D/3D visualizations. This approach facilitates a more intuitive understanding and analysis of the data.</p><p>The query objective achieved in this article represents one-step paths in the graph. Upon reconfiguring the return values of the provided UQL into these paths and executing the query, the 2D results obtained in Ultipa Manager appear as follows:</p><pre>n({@DOCTOR.DNAME == &quot;James Anderson&quot;})<br>   .e({@DIAGNOSIS})<br>   .n({@PATIENT}) AS p <br>RETURN p{*}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MYeEcmDpzsUjEHjE" /></figure><p>Visualization of paths with force-directed layout in Ultipa Manager</p><h3>Conclusion</h3><p>In this article, our goal is not to showcase all aspects of how a graph query language differs from a structured data query language. Instead, we aim to convey how a graph database should fundamentally design its query language and why SQL may not be suitable for querying graph data.</p><p>The simplicity in code composition and query efficiency against a native graph provide a valuable insight — perhaps when handling data, we should embrace a more intuitive and natural approach, akin to understanding the world through a graph, rather than getting entangled in the abstract data networks of relational databases.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bfaf8d1af765" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ultipa/why-sql-isnt-the-right-fit-for-graph-databases-bfaf8d1af765">Why SQL Isn’t the Right Fit for Graph Databases</a> was originally published in <a href="https://medium.com/ultipa">Ultipa Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is the World Flat? -The Super Data Network Behind Globalization]]></title>
            <link>https://medium.com/ultipa/is-the-world-flat-the-super-data-network-behind-globalization-d754e8a3cb20?source=rss-48da3986742c------2</link>
            <guid isPermaLink="false">https://medium.com/p/d754e8a3cb20</guid>
            <category><![CDATA[graph-database]]></category>
            <category><![CDATA[advanced-data-analytics]]></category>
            <category><![CDATA[graph-xai]]></category>
            <category><![CDATA[supply-chain]]></category>
            <dc:creator><![CDATA[Ricky Sun]]></dc:creator>
            <pubDate>Thu, 25 May 2023 04:40:29 GMT</pubDate>
            <atom:updated>2025-08-14T01:50:55.723Z</atom:updated>
            <content:encoded><![CDATA[<h3>Is the World Flat? The Super Data Network Behind Globalization</h3><p>The earth is round, but the world is flat. — by Thomas L. Friedman, The World Is Flat: A Brief History of the Twenty-First Century</p><h3>I. The Trade Route Behind an Artifact</h3><p>Over the weekend, I visited a museum exhibit featuring Egyptian mummies, masks, coffins, and covers. After conducting some research online, I was astounded by the historical clues hidden in the upstream and downstream supply chains behind these artifacts and how archaeologists traced and restored the entire supply chain network of ancient Egypt (the Ptolemaic dynasty) through chemical testing, and it came to my mind a clearer and broader picture of the bustling import and export trade of ancient Egypt:</p><p>Export products: gold (gold mines in ancient “gold town” Nubt), papyrus (the main material for the burial book of the dead), and textiles…</p><p>Import products: cedar wood from Lebanon, asphalt from the Dead Sea, lapis lazuli from Afghanistan, rubies, pearls, and spices from various coastal areas along the Red Sea…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/0*Py9vN8A5f0LAmVs3" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GIH_P4Ma_Xn4QEeX" /></figure><p>Figure 1: The Deep-down Connections Behind the Egyptian Cultural Relics</p><p>As shown in the figure, the gold used to make the mummy is from Egypt, while the cedar wood used for the coffin is imported from Lebanon. The black asphalt painted on the coffin comes from the Dead Sea, and the turquoises used for the eyes come from Afghanistan. Various spices used for mummy preservation can be traced to places as far away as India.</p><p>From ancient times to the present, supply chains have always been shaping our lives. In his best-selling book “Connectography: Mapping the Future of Global Civilization”, Parag Khanna argues that we have entered the era of super connectivity that is defined by the tangled global supply chain systems. The impacts on our daily life are everywhere, whether we can realize it or not, either through goods, transportation, industrial infrastructure, and information, even remote working that many people are used to is part of an information supply chain.</p><p>Also if you look into every link on the supply chain, you can discover how these subtle transactions are combined and affecting the global trends — the free market that Adam Smith called, the comparative advantage of David Ricardo, and the labor division theory of Emile Durkheim are all embodied — capital, labor, and production flow globally: they flow to the places that can connect supplies and demands most effectively!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/0*zSowQdC__KcfZvwn" /></figure><p>Figure 2: Image from the internet</p><h3>II. The Foundation of Smart Supply Chain — Data Processing Capability</h3><p>With the growth of technology, the global supply chain ecosystem has become a complex and intertwined web of relationships. The traditional view of the supply chain as a linear structure comprised of producers, intermediaries, and sales parties, has been replaced by a vast topology network encompassing an uncountable number of links such as planning, procurement, production, warehousing, logistics, and sales. These links are made up of data. Therefore, at its core, the supply chain is a matter of data. So how, in the mid of this complexity, can enterprises navigate cycles, respond to anticipated changes with feasible strategies in a best way?</p><p>GDBMS and graph augmented intelligence can be a satisfactory answer for this question. In modern smart supply chain management systems, graph computing technology has been widely used in application scenarios and plays a crucial role in production, maintenance, production planning and scheduling, material and inventory management. By employing GDBMS, a supply chain management system can benefit from many aspects, including:</p><p>Supplier Relationship Management: By storing information of suppliers, products, purchase orders, etc. as nodes and edges, the organization-level management of supplier relationships can become more effective by visualizing and analyzing supplier relationships. It helps identify opportunities and address risks right at the point.</p><p>Inventory Management: It can benefit from deep-down attribution analysis of inventory levels, order quantities, and delivery cycles. Inventory surplus and stock-out risks can be identified efficiently, providing companies advice on orders and production plan adjustments.</p><p>Logistics Management: The high-dimensional data structure to store logistics nodes, paths, transportation tools, and other data as nodes and edges enables efficient logistics network realization, logistics risk alerts, and transport arrangements.</p><p>Quality Management: Information of products, suppliers, and raw materials can be stored as nodes and edges in graph databases, which allows companies to target product quality issues speedily and optimize quality management in a process view.</p><p>However, not every graph database and their graph computing technology can fulfill the demands above effectively. To build a smart supply chain management system, a database vendor must be equipped with strong computing power as a solid underlying scaffold of the system.</p><h3>1. High-concurrency real-time computing power: Insight into the butterfly effect</h3><p>No risk exists in isolation in the global supply chains, and their effects are transmitted and interrelated with each other. In the past few years, factors such as natural disasters, wars, or political disputes are constantly causing incidents of “gray rhinos” and “black swans.” For instance, the Russian-Ukrainian conflict led to a parts supply crisis for Audi, BMW, and other veteran car companies, which forced several century-old automakers to halt their production. This incident demonstrates how any butterfly flapping its wings could cause these side effects.</p><p>Penetrating the butterfly effect essentially requires high graph computing technology. Designed for the most demanding industrial scenarios, Ultipa Graph XAI supports 30 hops + deep traversals, queries, and computing in complex graph sets, enabling enterprises to realize ultra-deep real-time penetration of datasets of any graph scale, i.e., giant graphs with billions of nodes. By doing so, businesses can gain insights into all the least traceable and most complicated associations in the sea-volume data in real-time, thereby getting fully prepared for changes.</p><p>Butterfly effects in real-world are often much more voracious in computing power than that from theories: Tesla is one example where the production of only one car requires approximately 10,000 components. These components, either big or small, are provided by thousands of companies, involving numerous suppliers, manufacturers, distributors, and retailers. As a result, there are countless stages of upstream and downstream supply and delivery processes. Any minor accidents at any stage may cause Musk’s delivery commitments to fail. Therefore, graph computing technology is an effective tool to participate and prepare for such risks, ensuring seamless operations across the sprawling supply chains.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/0*9zji-Giplce5k1ee" /></figure><p>Figure 3: Image from the Internet</p><h3>2. High-performance and scalable graph algorithm package: 100+ algorithms to reduce costs and increase efficiency</h3><p>Although the impacts of the global supply chain are almost ubiquitous, the supply chain itself is often invisible and insensible to us. People can only see those ending elements that connect supply and demand. This leads to the fact that business-oriented individuals, including production, procurement, logistics, warehousing, raw materials, etc., all have sea-volume data and require completely different data granularity, making it difficult for people from different sections and departments to communicate in the same dimension. As a result, it is impossible to achieve real-time synchronization of heterogeneous data from different sources or platforms to achieve valuable insights.</p><p>For automotive companies, a significant priority is the large-scale procurement of auto parts. However, this process is highly labor-intensive, time-consuming, and resource-intensive, requiring a lot of cross-department planning and coordination. Various critical decisions, such as product categories, quantities, selecting suppliers, cost accounting, and cost-reduction analysis, must be made to ensure a smooth procurement process.</p><p>In order to address these challenges, algorithms play a significant role in computing the interrelationships of operating entities in a supply chain. Specifically, an algorithm is a niche computing method designed for data treatment and procession to address a certain issue. The outputs of algorithms have two types: the first type being results produced by the algorithm, such as clustering, classification, prediction values, and the second type being computing rules produced by the algorithm. Ultipa Graph XAI can integrate data from multiple sources, use 100+ high-performance graph algorithms (Ultipa has the most abundant graph algorithm library worldwide), and execute these algorithms according to the supply chain data. Querying the data of each link through Ultipa GQL, which supports flexible Demi-schema, helps automakers integrate their requirements from various production lines around the world. This way, they can quickly evaluate the rationality of the requirements according to the budget, calculate procurement costs, and realize a well-planned and transparent procurement process with accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/330/0*PjTIP7B6-P7WGZwP" /></figure><p>Figure 4: 2D, 3D Data Visualization in Ultipa XAI Graph-Augmented Intelligent GUI</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/0*Cq2AtwUee955Rg_l" /></figure><p>Figure 5: A Comparison of Ultipa GQL (UQL) with Cypher and GSQL</p><p>For example, if the price of a certain brand that is steadily purchased by a certain production line suddenly rises due to some reasons, Ultipa Graph will automatically recommend a substitute brand that of acceptable price and of good quality to achieve an optimal allocation of resources and operating efficiency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/0*-7EIr1HyO9m0w-v0" /></figure><p>Figure 6: A Visualized Path Query Result from Ultipa Manager</p><p>As shown in Figure 6, this path template queries for all 5-step paths starting from location1 and ending at the factory node or manufacturer node, and we can identify the risk spreading paths and related impacts caused by certain risks in advance, and provide the best match intuitively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/0*szhOuBZW8x3M943s" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/0*PU0n_MeDO0dRDYqU" /></figure><p>Figure 7: Tesla’s Supply Chain (A Subgraph) Queried and Visualized by Ultipa Manager</p><p>When solving an issue, the algorithm selected can greatly impact runtime performance. In fact, a well-designed algorithm should not only address the problem but also ensure high running efficiency as well as low resource consumption. In pursuit of these goals, Ultipa offers comprehensive, high-performance, and low-latency graph algorithm packages, making it the most competitive choice globally.</p><p>Moreover, Ultipa’s algorithm design is boasted by Ultipa’s graph augmented intelligence (GAI) and XAI (eXplainable AI) which intuitively learns from repetitive tasks, making the design even simpler, clearer, and more effective over time. Different algorithms can work together and their collaboration can be continuously self-improved to solve challenging problems, such as those applied to automatic driving scenarios, i.e. computer vision algorithms. In fact, the training model datasets are the key to good algorithm design. These innovative designs from Ultipa enable users to complete tasks previously deemed impossible within a short timeframe.</p><h3>3. High-dimensionality of graphs: computational complexity and data</h3><p>Nowadays, digital transformation is a challenge and an opportunity for all industries worldwide, requiring enterprises globally to shift from traditional relational databases to big data, fast data, and deep data, such as graph data. This is because computing is ubiquitous in everyday scenarios such as IoT, generating vast amounts of tangled data every second, and these data need to be collected, stored, penetrated, and processed properly, just like mining resources.</p><p>However, traditional relational databases have structural limitations and cannot handle the sophisticated storage, query, and computation requirements of intelligent supply chains for complicated relationships. For multi-table join queries, computational complexity grows proportionally to the Cartesian product of the amount of data in each table. This issue decreases efficiency exponentially as the amount of data increases. In contrast, Ultipa’s graph database uses a k-nearest neighbor (KNN) algorithm with an exponential efficiency improvement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/0*-Ci8IzC-Th1AAnIE" /></figure><p>Figure 8: Architecture Differences between Graph Databases and Relational Databases.</p><p>When using relational databases and graph databases to do deep penetration to a large-scale dataset, it can be noticed that the performance differences from the 2nd to 5th layers rise exponentially: when doing first-layer penetration, there may not be any essential differences between the two methods, but from the second layer onwards, there will be exponential (more than 10 times) changes. The game for RDBMS quickly ends when it cannot return any results for 4–5 layer penetrations any more, for it has exceeded the computing scope and has unfortunately run into a shutdown.</p><p>Not all graph databases are created equal. It is worth mentioning that while Ultipa Graph can achieve deep penetration, other graph databases may not be able to do so because of their technological limitations. Some may lack computational capabilities despite having storage capacity, while others may struggle with data migration efficiency. Additionally, some graph databases may use NoSQL or MapReduce architecture, but without sufficient optimization for graph computing, it may result in lower efficiency as the cluster becomes more scalable. Some even just move all data into memory, causing a surge in memory usage and frequent OOM, resulting in crashes.</p><p>To address these challenges, the best approach should be “distributed + integrated storage and computing + multi-level storage optimization + deep optimization of graph query.” Designing and implementing a graph database requires knowledge and expertise in this specialized field, fully equipped with which, Ultipa innovated the patent-pending distributed HTAP graph database to offer a best-of-breed solution for organizations looking to build intelligent supply chain digital infrastructure together with IoT and digital twins. Ultipa simplifies the support for knotty data types and relationships between data, allowing ease of traversal along the relationship network and easy expansion of data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/0*XND4E6H4pI2QUaPP" /></figure><p>Figure 9: Real-world Relations Between Entities can be 100% Simulated by Graph Databases</p><p>Moreover, with Ultipa Manager, enterprises can coordinate and automate management processes by leveraging the underlying engine of Ultipa as well as the customizable visualization GUI at the application layer. It enables staff, whether developers or business personnels, to interact with business data in a real-time fashion and understand the progress of and relationship between data. Thus, Ultipa allows for the integration of customers, manufacturers, and suppliers that are from different business dimensions into a cohesive and coherent body.</p><p>The best solution to consider for this is “distributed + integrated storage and computing + multi-level storage optimization + deep optimization of graph query.” Ultipa’s distributed HTAP graph database provides a new option for this challenge and is serving as the intelligent supply chain digital infrastructure together with IoT and digital twins.</p><p>Moreover, enterprises can coordinate and automate management processes through Ultipa Manager by leveraging the underlying engine of Ultipa. With customizable GUI at the application layer, enterprises can retrieve, store, and utilize real-time business data and allow all staff, either developers or business personnels, to take full control of the progress and relationship between data at any time, integrating customers, manufacturers, and suppliers that seem to come from different business dimensions into a cohesive and coherent whole.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/0*U_yi8t-EvRcXP7kS" /></figure><p>Figure 10: Customized Visualization GUI at the Application Layer.</p><h3>In Conclusion</h3><p>Marx famously argued that production relations are determined by productivity, and it holds truth in this case: as technological advancements in artificial intelligence, IoT, cloud computing and more continue to rapidly evolve, information and material flows have reached unprecedented levels, leading to increasing numbers of companies participating in the global economic cycle. This is quickly filling the great productivity divide that has spanned centuries between the East and West, and the world seems to become flatter than ever since…</p><p>However, it is important to acknowledge the other side of the coin. In the labyrinthine world landscape, the “Dell Theory” may not hold validity in the interconnectedness of the global supply chain, and the Thucydides Trap continues to loom over nations. History serves as a reminder for us that disputes among nations would frequently affect the supply chain, while the supply chain seems to have nothing to combat with these influences: despite accounting for 13% of the world’s GDP through exports, Britain, once an empire where the sun never set, was unable to cancel the outbreak of World War I, and similar conflicts are still ongoing, such as the war between Russia and Ukraine today. In light of these events, it raises questions about the fragility of supply chain connections and whether the world is truly being flattened…Nevertheless, there is hope in GDBMS, graph computing technology, graph augmented intelligence, which unlocks invaluable insights from data to empower enterprises to implement effective risk management. It provides a powerful tool in times of uncertainty, allowing people to make informed decisions and reduce their exposure to risks.</p><p>Special thanks to Ultipa Sr. Content Strategist WY, Lina, and Pearl for their contributions to this article.</p><p>Written by/ Emma</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d754e8a3cb20" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ultipa/is-the-world-flat-the-super-data-network-behind-globalization-d754e8a3cb20">Is the World Flat? -The Super Data Network Behind Globalization</a> was originally published in <a href="https://medium.com/ultipa">Ultipa Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Graph Augmented Intelligence & XAI]]></title>
            <link>https://medium.com/ultipa/graph-augmented-intelligence-xai-41db2edc9d08?source=rss-48da3986742c------2</link>
            <guid isPermaLink="false">https://medium.com/p/41db2edc9d08</guid>
            <category><![CDATA[graph-embedding]]></category>
            <category><![CDATA[ai-and-big-data]]></category>
            <category><![CDATA[graph-database]]></category>
            <category><![CDATA[xai]]></category>
            <category><![CDATA[graph-learning]]></category>
            <dc:creator><![CDATA[Ricky Sun]]></dc:creator>
            <pubDate>Wed, 15 Mar 2023 00:23:07 GMT</pubDate>
            <atom:updated>2025-08-14T01:51:43.062Z</atom:updated>
            <content:encoded><![CDATA[<p>The Convergence of AI and Big-Data, and Moving to Graph Era</p><p>By Prof. Seasoned Edges</p><p>There are broad-spectrum real-world applications that can be powered with graph technologies. Social networks dwell on graphs which best model how people follow and befriend each other; Bio-tech and pharmaceutical companies leverage graphs to understand protein interactions and chemical compounds efficacies; Supply chains, telco networks, power grids are naturally presented as graphs for root-cause analysis; Financial transactions naturally form networks and fraud detections, smart recommendations or asset-liability management are possible to run across these networked data for improved productivity, prediction accuracy or business intelligence. Many industries are looking to graph’s smart (and deep) data processing capabilities to help with their businesses.</p><p>The big-data era started around 2010, as more and more industries are interested in machine learnings (and deep learnings and AI) to boost their business predictability, some have been using deep learning and specifically varied neural networks to extract more predictive powers. There are 3 major problems lingering around though:</p><ul><li>Siloed Systems within AI Eco-system</li><li>Low Performance AI</li><li>Black-Box AI</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*C4rq_htOD_w-c_d1.png" /></figure><p>Graph-0: AI Learning Techniques and Their Performance &amp; Explainability</p><p>Graph-0 illustrates the three major problems well.</p><p>First, there are quite a few AI learning techniques, but they are NOT offered in a one-stop-shop fashion, most AI users and programmers are dealing with multiple siloed systems, software and hardware wise, which means a lot of data ETL are needed all the time (we’ll give detailed examples to illustrate how serious the problem is next).</p><p>Secondly, AI (deep/machine) learning seems to be facing the dilemma of performance and explainability contradictory to each other, you can NOT have the best of both, and needless to say that as we go deeper into the deep-learning and ANN/CNN space, things are so black-box that human decision makers are NOT okay with this status-quo.</p><p>Lastly but certainly not the least important problem is most of these learning processes are NOT high performance, the data preparation, ETL, training and application against production data can be a very complex and lengthy process. Some large retail banks tried to use machine learning and AI to predict monthly credit card spending (turnover) for their hundreds of millions of card holders, but the prediction process takes weeks (as you would imagine, if they start the process mid-month every month, you won’t get a result until a few days passing the deadline, which renders the whole effort pointless. Besides, such ML/AI based prediction method is problematic in terms of accuracy which implies higher-cost associated with the bank’s liquidity management.</p><p>All in all, these problems deserve elevated attention, and ideally, infrastructure level innovation should be attained to address and solve these challenges.</p><p>To be a bit analytical on the needed innovation to address the problems we are facing — the trend we are seeing, and what we need, is the convergence of AI and Big-Data. We desire AI to be smarter, faster and explainable on the one hand, and we request Big-Data to be fast, deep and flexible in terms of data processing capabilities. The greatest common factor of this AI+Big-Data convergence leads to graph augmented intelligence and XAI. Graph database is the infrastructure and data-intelligence innovation that we have been looking for. We’ll show you why and how in this essay.</p><p>The graph below (Graph-1) shows this converging trend:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/0*iiY-o4U9iIQdqLMA.png" /></figure><p>Graph-1: The Convergence of AI and Big-Data to Graph Augmented XAI</p><p>Traditionally, mathematical and statistical operations are rather limited on graphs, to understand the reason, we have to look at the evolution of graph data structures which is covered a separate essay <em>The Specific Evolution of Graph Data Structures</em>. To recap, in either adjacency matrix or adjacency list centric data structures, very few math or stat operations can be done, not to mention these data structures are NOT high-currency capable.</p><p>Due to these limitations, developers have invented the concept of graph embedding, which basically transforms a property graph into vector space, so that a lot more mathematical and statistical operations can be performed with higher efficiency. This transformation essentially is to transform high-dimensional data to be represented in lower dimension while still preserving certain properties (such as graph topology and node/edge relationships). The other benefit of such transformation is that vector operations are much simpler and faster.</p><p>Slightly expanded discussions around vector computing are necessary. It’s an effective way to accelerate graph computing. A major advantage that’s highly relevant to graph embedding is that — the underlying data structure is vector-space comparable, which enables convenient mathematical and statistical computing operations. Moreover, from a technological stack perspective, it’s possible to parallel everything, from infrastructure, data structure, algorithm to engineering implementation. The pursuit for high concurrency has never ceased to thrust forward, and this will ensure the otherwise time-consuming operations like graph embedding and learning can be completed as fast as timely as possible!</p><p>We’ll use the Word2Vec example to explain why vector-computing empowered graph embedding can be orders of magnitudes faster, and simpler. Word2Vec method can be seen as the basis, and prototypical, for all node, edge or struct centric graph embedding methods. It basically transforms words into embedding vector space, so that mathematically in the lower dimensional vector space, similar words have similar embeddings — and “You shall know a word by the company it keeps” as coined by the famous English linguist J.R. Firth in 1957 — this is to say that similar words with similar meanings tend to have similar neighboring words!</p><p>In the three following graphs (Graph-2, Graph-3 &amp; Graph-4), a Skip-gram algorithm and SoftMax training method implementing Word2Vec are illustrated, there are other algorithms (i.e., Continuous Bag of Words) and methods (i.e., Negative Sampling) which are abbreviated for discussion simplicity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/0*ffhn9QV_F3P6Deyf.png" /></figure><p>Graph-2: Word2Vec from Source Text to Training Samples</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/0*UMYSXREuaTCrSVR7.png" /></figure><p>Graph-3: Architecture of Word2Vec Neural Network</p><p>Graph-2 shows some of the training samples extracted from a sentence with a window size of 2 (meaning 2 words pre- and 2 words post- the inputed/focused/highlighted word). Note that in improved word2vec method, techniques like sub-samplings are used to remove samples that contain words like “the”, so that to improve overall performance.</p><p>To represent a word and feed it to a neural network, the one-hot vector method is used, as illustrated in Graph-4, the input one-hot vector for the word “ants” is a vector of 10,000 elements, but with only 1 and all the other positions are 0s. Similarly, the output vector also contains 10,000 elements, but filled with probability (floating points) of each nearby word.</p><p>Assuming we have only 10,000 words in our vocabulary, and 300 features, or parameters that you would use to fine tune this Word2Vec model. This translates into 300 neurons in the hidden layer and 300 features in the word vector (See Graph-4).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/0*gIwuWpftemqzBqXC.png" /></figure><p>Graph-4: Gigantic Word Vectors (3,000,000 Weights!)</p><p>As you can see from Graph-4, the hidden layer and output layer each carries 3,000,000 weights, this is NOT feasible for real-world applications. In real-world circumstances, the vocabulary size can be as large as in the range of millions, multiplied by hundreds of features, which will make the size of one-hot vector reaching billions, even with the help of sub-sampling, negative sampling and word-pair-phrase techniques, this sheer computational challenge is NOT acceptable. The one-hot vectors in both the input and output layers are clearly way too sparse a data structure, it’s like the adjacency matrix, it may be partitioned for parallel computing on GPUs when it’s very small in terms of the number of elements (❤2,768 for instance) the matrix can hold for each dimension, but for much larger real-world applications, this will be overwhelming in terms of resource and time consumption.</p><p>Now, recall what we wanted to achieve with Word2Vec, essentially, we are trying to predict, in a search and recommend engine setup, when a word (or more) is entered, what other words we would like to recommend. This is a statistical game, and underlying, it’s a mathematical problem, for one word asked, a handful of other words with the highest probability of appearing as close neighbors for this asked word. This is naturally a graph challenge, every single word is considered a node (or vertex) in the graph, and its neighbors are forming edges connecting with it directly, this goes on recursively. And each node and edge can have its own weight, type, label, and attributes (as many as needed).</p><p>What we just described in the above paragraph can be represented in Graph-5, with two data structure variations. On the surface, they look like big-table data structures, but under the hood, it’s the high-concurrency multi-dimensional vector-space implementations that make things fly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*76_XTYWFtUKAv419.png" /></figure><p>Graph-5: Graph Data Structures with Natural Graph Embeddings</p><p>Given the vector-space data structure, word2vec problem really boils down to two steps:</p><ul><li>Assigning a probability weight for each directed edge connecting two words, and this can be done statistically;</li><li>Search and recommendation are simple and short graph traversal tasks which simply pick the top-X number of vertices connecting the starting vertex, and this operation can be done recursively to traverse deeper in the graph — this is like to search and recommend in real-time (similar to real-time suggestion, but potentially more accurate and intelligent, if you recall this exactly resembles how human brain functions with NLP).</li></ul><p>What we just described is a complete white-box process, every single step is specific, explainable and crystal clear. It’s very much unlike the black-box and hidden layer(s) that you have to deal with cluelessly as in Graph-3 and Graph-4. This is why <strong><em>graph will drive XAI (short for eXplainable Artificial Intelligence) forward</em></strong>!</p><p>More specifically, vector-space computing is naturally graphically embedded — there is no further computation or operations needed, when the data are presented in the graph in the format of vectors (a form of index-free adjacency), the embeddings are done.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/0*Welu56I6qqbKh0EF.png" /></figure><p>Graph-6: Breakdown of DeepWalk</p><p>DeepWalk is a second example to investigate. It is a typical vertex-centric embedding approach, and uses random walks to produce embeddings. With the help of vector-space computing, random-walk is as easy as selecting a specific node and move on to a random neighboring node and deeply traverse the graph for a random number of steps (i.e., ~ 40 steps). Pay attention to the depth/length of the walk, we understand that every hop deeper into the graph, the computational challenge is exponential, the simple math here is that, if an average node has 20 neighbors, going 40 steps deep is to look through neighbors as many as 2040, no production system today can handle this in a brutal-force way. However, with the help of highly concurrent vector computing, each step of traversing (sampling) in the graph takes O(1), going 40 step further is basically O(40*20) in terms of overall latency (though compute-wise, all the concurrency tasks still parallelly deal with all neighbors encountered and filter out any duplicated or revisited nodes effectively).</p><p>The step-by-step breakdown of DeepWalk is illustrated in Graph-6. With highly concurrent vector computing, the random walking, training and embedding processes are reduced into native graph operations like K-Hop-ing and Path-Finding, which are straightforward, explainable therefore XAI friendly!</p><p>DeepWalk has been criticized with 1). the inability of generalization, not capable of handleing highly dynamic graphs (each new coming node must be re-trained) well, and 2). not possible to preserve the local neighborhood (because deep random walks do NOT preserve neighborhood information).</p><p>Node2Vec method has been invented to address at least the second shortcoming of DeepWalk. Graph-7 illustrates the many steps taken to make the whole learning process complete. Please note that there are 10 steps (subprocesses) involved, each step by itself can further breakdown into a series of computing process. A graph embedding method like this usually takes long time to complete. But, this is NOT the case with Ultipa’s graph embedding algorithms, we convert all these subprocesses into concurrent fashion so that to maximizing the leverage of underpinning hardware computing juices as well as minimizing turnaround time. Specifically, all of the following subprocesses are run in parallel fashion:</p><ul><li>Node2Vec’s random walks</li><li>Preparation of precomputed Exp. Table</li><li>Learning of Vocabulary (from Training File)</li><li>Removal of Low-Frequency Vocabularies</li><li>Sub-sampling of Frequent Words (Vectors)</li><li>Initialization of Network (Hidden Layer, Input and Output Layers)</li><li>Initialization of Unigram Table (Negative Sampling)</li><li>Training of Skip-Gram</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/0*cvDVt-j3rol6LKnm.png" /></figure><p>Graph-7: Outer Steps of Node2Vec Relying on Word2Vec</p><p>The parallelization of these steps helps achieve much better latency and performance improvement, always remember that every single second wasted on not achieving high concurrency is a waste of resources, therefore not environment friendly, and we are not kidding.</p><p>There are other graph embedding techniques that bear similar concepts and steps:</p><ul><li>Sampling and labeling</li><li>Training of certain model (i.e., Skip-Gram, n-Gram, and etc.)</li><li>Computing embeddings</li></ul><p>Some are done locally at vertex/subgraph level, others are done against the entire graph therefore mathematically more time consuming, whichever approach we take, being able to achieve the following <strong>goals</strong> would be meaningfully fantastic:</p><ol><li>Attain to high performance via high concurrency on data structure, architecture and algorithm level;</li><li>Explain every step of operations in a definite, specific, white-box way.</li><li>Ideally, complete the operations in one-stop-shop fashion.</li></ol><p>If we recall Graph-0 at the beginning of this essay, the Explainability-vs-Performance chart is intriguing. Decision Trees have better explainability, because we understand what they do step-by-step, trees are essentially simple graphs, they don’t contain circles which are considered perplexing mathematically, therefore harder to explain, and most so-called graph algorithms or embedding implementations today are based on low-performance adjacency list or academic-oriented adjacency matrix. On the other hand, overly sophisticated neural networks and deep learning techniques are going into a direction that’s black-box centric and difficult to explain. In real-world applications, AI can NOT go away with the human beings not being able to understand what exactly is going on under the hood — it’s against the XAI rule of thumb — Explainability!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/0*GAREqKcFqCmxrIhe.png" /></figure><p>Graph-8: Integrated Ultipa Real-Time Graph Database &amp; Ultipa Manager</p><p>The breakthrough of graph data structures and implementation of highly parallel graph computing architecture would make achieving the three goals highly tangible. This is what <a href="https://www.ultipa.com/">Ultipa Graph</a> has been invented for and geared toward.</p><p><strong><em>Ultipa is all about Graph Augmented Intelligence &amp; XAI.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/0*hBaq5SmcWkDUkVi3.png" /></figure><p>Graph-9: Ultipa Manager in 3D Mode</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41db2edc9d08" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ultipa/graph-augmented-intelligence-xai-41db2edc9d08">Graph Augmented Intelligence &amp; XAI</a> was originally published in <a href="https://medium.com/ultipa">Ultipa Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Liquidity Lesson Learned from SVB’s Collapse]]></title>
            <link>https://medium.com/ultipa/the-liquidity-lesson-learned-from-svbs-collapse-62a99e4a0102?source=rss-48da3986742c------2</link>
            <guid isPermaLink="false">https://medium.com/p/62a99e4a0102</guid>
            <category><![CDATA[lrc]]></category>
            <category><![CDATA[graph-database]]></category>
            <category><![CDATA[ultipa]]></category>
            <category><![CDATA[liquidityrisk]]></category>
            <category><![CDATA[liquidity]]></category>
            <dc:creator><![CDATA[Ricky Sun]]></dc:creator>
            <pubDate>Mon, 13 Mar 2023 05:45:08 GMT</pubDate>
            <atom:updated>2025-08-14T01:54:02.824Z</atom:updated>
            <content:encoded><![CDATA[<p>Prof. Seasoned Edges 2023–03–12</p><p><strong>Foreword</strong></p><p>Over the past weekend, Silicon Valley Bank (hereinafter referred to as <strong>SVB</strong> ) made the financial market start to worry whether there will be a butterfly effect , triggering a new round of crisis similar to the bankruptcy of Lehman Brothers ?</p><p>The essence of SVB’s collapse is that the interest rate risk (market risk) of maturity mismatch causes credit risk , which in turn triggers the reputation risk of panic in the market and depositors. The liquidity risk that finally broke out was the last straw that overwhelmed the camel, and the government shut down SVB within 48 hours.</p><p>Judging from historical experience, liquidity crises are sudden and unpredictable. Since it will be sudden and unpredictable, instead of handing it over to the AI ​​system to make randomized (and unfaithful) predictions, it is better to rely on the ability of the real-time computing system for real-time monitoring of liquidity. <strong>The optimal or more desired computing power is not long-term prediction, but real-time computing.</strong></p><p>Technically speaking, <strong>this crisis has made the financial industry clearly realize the importance of an advanced liquidity risk management system, especially intraday or even real-time liquidity risk management, for banks to “meet external supervision and strengthen internal efficiency”</strong> . However, the frequently T+1 or even T+N high-latency technology stack based on the traditional SQL architecture is extremely difficult to meet the requirements of increasingly real-time liquidity management. As a result, <strong>liquidity risk management systems</strong> based on real-time graph computing and graph database technology stacks are rapidly emerging.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*fEtvyfCOebl0ZvESIT0p1w.jpeg" /></figure><p>Figure 1: SVB bankruptcy (FDIC news)</p><p>Below, the author will review the retrospective of the SVB event, the limitations of traditional technologies, and the application and breakthrough of graph database technology in the liquidity risk management system.</p><h3>SVB exploded?</h3><p>SVB collapsed due to <strong>illiquidity and insolvency!</strong></p><p>This is one of the events that caused global impact after the collapse of Lehman Brothers and Washington Mutual since 2008. At present, this crisis is still fermenting, and all sectors of society are concerned about whether this black swan will become another fuse after <strong>the 2008 financial crisis ?</strong></p><p>History is always surprisingly similar. Like world-renowned banks such as Lehman Brothers and Washington Mutual , <strong>SVB</strong> is the 16th largest bank in the United States, with total assets of more than 200 billion U.S. dollars (equivalent to about two Baoshang Banks or one-tenth of JPMorgan Chase’s assets Scale), has served Facebook, Twitter and other star companies, but when <strong>the liquidity crisis</strong> came, in less than 48 hours, a capital giant fell to the ground.</p><p>We know that, in fact, <strong>after the 2008 financial crisis, it has become the consensus of industry regulators and academics to pay attention to liquidity risk management:</strong></p><p><strong>(1) The industry pays more and more attention to liquidity risk management</strong> . Since 2009, the Basel Supervisory Committee has proposed two liquidity risk measurement indicators — LCR (Liquidity Coverage Ratio) and NSFR (Net Stable Funding Ratio), which have since established a unified global standard for banking supervision.</p><p><strong>(2) Top academic circles are very concerned about liquidity risk</strong> . The three winners of the Nobel Prize in Economics last year, without exception, were awarded to economists who emphasized the importance of “liquidity” in the banking system. Now it seems that since last year, the global economic subprime mortgage crisis The influence has intensified, and the award is the best proof.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*3OUEEkUfBvb4eXt0IF82sA.jpeg" /><figcaption>2022 Interview with Philip Dibwig</figcaption></figure><p>Figure 2: Interview with Philip Dibwig, one of the 2022 Nobel Laureates in Economics</p><p>In long-term research, experts have found that <strong>risks have the characteristics of correlation, mutual transformation, transmission and coupling</strong> .</p><p>Just as SVB’s liquidity crisis triggered a series of continuous declines in US stocks, we have clearly seen that the ripple effect is still spreading, and Silicon Valley Bank is just the first domino. As of now, the four major U.S. banks (JPMorgan Chase, Bank of America, Wells Fargo, and Citibank) have lost nearly $50 billion in market value in the transaction, and the impact is still expanding. Among them, some of Wells Fargo Bank’s account holders have reported that their balances are showing negative to zero numbers.</p><p>In addition, following the bankruptcy of SVB, Market Watch (a news website owned by Dow Jones) released a list of 10 regional banks that may suffer the same fate as SVB. They are Customers Bancorp, First Republic Bank, Sandy Spring Bancorp, New York Community Bancorp, First Foundation Inc., Ally Financial Inc., Dime Community Bancshares, Pacific Premier Bancorp, Prosperity Bancshares, and Columbia Financial.</p><p>With the evolution of this crisis, we can’t help but ask, today, <strong>building a powerful combo of risk-control toolkits has already become the consensus of modern risk control management</strong> , so why a bank as large as SVB has not used a set of “financial weapons” to help its prophetic vision of risk? At least it won’t make bank executives misjudge or underestimate the power of liquidity risk.</p><h3>What is the nature of SVB’s bankruptcy?</h3><p>We have seen from many news that almost everyone is looking for bankruptcy caused by the macro environment (US interest rate hike), failed investment strategies (US Treasury bonds), etc., but the only thing that has not been interpreted from the perspective of financial technology. <strong>The author believes that the essence of SVB’s bankruptcy is the inadequacy of financial technology as a tool for timely monitoring, measurement, attribution analysis and forecasting. This is also the limitation of database technology based on traditional SQL architecture</strong> .</p><p>Then, problems encountered by traditional relational databases, including data warehouses or data lakes, can be summarized into five categories:</p><p><strong>1.</strong> <strong>Rigid, low-dimensional modeling capabilities</strong> . The data modeling mode of traditional databases based on two-dimensional relational tables is very rigid, and it is difficult to simply express the multi-dimensional relationships required by the business. If you recall the ER model diagram, you will understand that the ER diagram is originally an association model of entity relationship, but it is necessary to make the high-dimensional, multi-entity association relationship model into a split two-dimensional table for low- <strong>dimensional</strong> expression . This mode itself means a kind of inefficiency caused by dimensionality reduction and fragmentation, as well as the obliteration of flexibility.</p><p><strong>2.</strong> <strong>Problematic computing power and low efficiency</strong> . The relational database (RDBMS/SQL) is centered on the storage engine, and the computing engine is a second-class citizen. However, financial services are changing rapidly, and higher computing efficiency is required more often. Although many data warehouses claim to accelerate SQL queries, the logic of data warehouse acceleration is based on more tables (intermediate tables, temporary tables), and more The wide table extracts and combines data in various ways, and does not change the low-dimensionality of data weaving (two-dimensional table) in essence.</p><p><strong>3.</strong> <strong>timeliness is poor</strong> . Relational databases and the use of SQL query language make it difficult to perform simple recursive queries, such as drill-down, penetration, attribution, traceability analysis, in-depth path analysis, complex scenario simulation and full-scenario stress testing, etc., resulting in poor timeliness.</p><p><strong>4.</strong> <strong>Poor interpretability</strong> . A large number of SQL codes and stored procedures in the existing liquidity management system have similar “black box” characteristics, and the maintenance cost is high. Just imagine how poor the maintainability and interpretability of tens of thousands of lines of SQL code are. .</p><p><strong>5.</strong> <strong>The problem of short chains</strong> . Relational databases or data warehouses and data lakes solve the problem of “short-chain” transactions (such as flash sales), and it is difficult to solve the challenges of “long-chain” transactions and complex queries in financial scenarios. Short-chain queries are more suitable for metadata-oriented and distributed writing and reading scenarios; however, long-chain queries, such as liquidity risk management, capital measurement and analysis, etc. Correlation analysis, drill-down analysis, attribution analysis, cross-indicator analysis, etc., require a more advanced graph computing (graph database) architecture to empower.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*wtbAvB-A1zGST-NvfpKHkg.png" /><figcaption>Computational complexity problems of relational databases</figcaption></figure><p>Figure 3: Computational complexity problems of relational databases</p><p><strong>The migration from relational databases (and data warehouses and data lakes) to graph databases is an irreversible trend. The driving factor behind it is that industry and academia urgently need a new, efficient, flexible, and high-dimensional architecture. appear</strong> . Therefore,<strong> GQL</strong> , as an international standard language following SQL, has become the only international graph database language standard that will be released at the end of this year since 1983. A powerful expression of graph’s importance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*nTAztlx1K-rp5o5dq9mLIw.png" /><figcaption>Exponential Performance Advantages of Graph over SQL/RDBMS</figcaption></figure><p>Figure 4: As the query depth increases, the query efficiency gap between graph databases and relational databases increases exponentially.</p><h3>What disruptions does graph technology bring?</h3><p>In the past 40 years of development, although the traditional relational database technology based on SQL (such as represented by Oracle) is still in the dominant position in the market share, facing the whole world from big <strong>data</strong> to In the current <strong>deep data</strong> and <strong>fast data</strong> transformation, it is no longer able to efficiently handle massive, complex, and dynamic data capabilities, let alone carry out business demands such as predicting the butterfly effect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wfCUhPTATAJ60Y-Ttnw5yQ.png" /><figcaption>Relational Data to Big Data to Graph Data</figcaption></figure><p>Figure 5: The Mega Trend of moving to RDBMS to Big-Data to Graph &amp; Deep Data</p><p>At the same time, the financial scenario of a bank is a <strong>long-chain computing</strong> scenario. Only by realizing real-time, comprehensive, deep penetration, transaction-by-transaction traceability, and accurate measurement monitoring and early warning can we ensure that financial risk control avoids risks such as the butterfly effect. happened. <strong>From the perspective of data processing architecture, it is extremely difficult to achieve without the strong support of graph technology with deep drill-down, correlation, real-time calculation and analysis capabilities</strong>!</p><p>Compared with the liquidity risk management system built on the traditional architecture, <strong>Ultipa Graph database</strong> has achieved innovation and breakthrough in liquidity risk management through innovative patented technologies such as high-density concurrent vector computing engine and multi-level storage computing acceleration.</p><p><strong>Summary: Based on the world’s first Ultipa LRM system that uses a real-time graph database technology framework to empower financial companies to manage liquidity risks , it has the following characteristics:</strong></p><p>· <strong>White-box Explainability </strong>realized with real-time graph computing<strong>.</strong> White-box explainability is realized through high-performance, easy-to-digest high-visualization web-based GUI and underpinning graph computing techniques, which not only retains highly explainable end results, but also leaves intermediate processes for easy auditing.</p><p>Ultipa LRM system provides powerful computing power to conduct real-time attribution analysis on changes in liquidity regulatory indicators (such as LCR) — according to multi-dimensional metrics such as customers, branches, industries, customer groups, and sub-indicators. Liquidity changes are deeply analyzed to the finest granularity (transaction by transaction, account by account) and correlations with other financial indicators (i.e., Loans, Deposits, RAROC, RWA, NII/NIM, EVA, AUM, etc.) are also investigated in a unified dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kf_C0sZtLB2PVFvjniHkrw.png" /><figcaption>Ultipa Real-time LRM Graph System</figcaption></figure><p>Figure 6: Ultipa Graph LRM System</p><p>· <strong>Real-time-ness</strong>. We know that liquidity risk management is an important regulatory indicator that financial institutions need to report on a regular basis (quarterly, monthly or daily). However, large commercial banks relying on Oracle system take very long time to process, the timeliness is problematic, not to mention Oracle’s inability to conduct attribution analysis. Ultipa LRM system gets things done in real-time, saving institutions tons of time to focus on their business insights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*0byRIFqX4hanI0qPPH8KcA.png" /></figure><p>Figure 7: Graph is exponentially faster (easier) to empower LRM</p><p>In addition, due to the natural characteristics of high-dimensional data modeling, graph database technology exponentially reduces the complexity of calculation and attribution analysis in liquidity management. As can be seen from the figure below, for graph databases , the larger the amount of data and the more complex the associated query, the more obvious the advantage than the traditional relational database.</p><p>· <strong>Dynamic &amp; Accurate</strong>: From the perspective of financial business, the current liquidity risk management indicators used by commercial banks are mostly static indicators due to the limitations of the existing technical architecture, which leads to the indicators calculated from historical data, and only reflecting the liquidity status of the bank at a certain historical time rather than dynamic, real-time and comprehensive monitoring. This kind of “after-the-fact” monitoring is prone to good indicators but high risks (due to long latency). Therefore, only graph database with real-time and deep data penetration capabilities can help banks to gain advanced insights into their ALM data both with accuracy on the fly.</p><p>· <strong>Graph-connected Data Sets and Graph Traversals</strong>. Judging from the research and development practice of the author and the team in the fields of graph database empowerment supervision and daily operation of the financial industry for many years, the root of insights generated from complex data lies in whether the technology can truly achieve real-time, comprehensive, accurate, in-depth traceability, attribution and capabilities for deeply penetrating graph-connected data, and graph queries. This is precisely the pain point that traditional relational databases cannot handle. Of course, not all graph database vendors can achieve such hard-core capabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HVyRqvr_rPaTn9KobwqTjQ.png" /><figcaption>Oracle vs. Ultipa</figcaption></figure><p>Figure 8: Traditional liquidity management system represented by Oracle vs. Ultipa real-time graph database system</p><p>The financial crisis in 2008 made the industry realize that various risks do not exist in isolation, and that different risks have a chain effect. How to measure and control the correlation between various types of risks is a test of the performance of underpinning and empowering technology, which is also the most difficult problem in the field of financial risk management.</p><p>Taking SVB as an example, the rapid decline in its deposits led to insufficient asset margin for SVB, and it was forced to sell the assets, which in turn caused panic caused by liquidity risks, and the market sentiment turned into a comprehensive and systematic crisis… Then, can it be identified and predicted in advance through measurement tools?</p><p>Ultipa’s LRM system has made innovative breakthroughs. <strong>Graph computing can calculate the relationship between data through deep penetration and mining of massive and complex data, and solve complex multi-layer nested relationship mining problems</strong> .</p><p>Taking Evergrande Group as an example, after the Evergrande Group’s crisis broke out, the risk first spread to Evergrande’s affiliated companies, which is the first layer; if there is a problem with the affiliated companies, their employees and suppliers are quickly impacted, and this is the second layer. Suppliers stop supplying goods, migrant workers refuse to resume work, Evergrande’s construction projects in progress have to be halted, and the risk will eventually spread to home buyers. This is the third layer. The risk is spread layer by layer, and the “networked chain effect” is obvious.</p><p>It is worth mentioning that <strong>from the perspective of modern risk management, risk itself is a neutral concept, which is not only the cause of possible losses, but also the source of profits</strong> . Graph technology represented by Ultipa is committed to <strong>enhancing intelligence + explainable AI through graphs</strong> , empowering enterprises to create value, in-depth insight, crisis early warning, and refined management in the process of digital transformation.</p><p>About the author: Prof. Seasoned Edges is an expert in cloud computing, big data, high-performance storage and computing system architecture.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62a99e4a0102" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ultipa/the-liquidity-lesson-learned-from-svbs-collapse-62a99e4a0102">The Liquidity Lesson Learned from SVB’s Collapse</a> was originally published in <a href="https://medium.com/ultipa">Ultipa Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What should you know about Graph Database’s Scalability?]]></title>
            <link>https://medium.com/ultipa/what-should-you-know-about-graph-databases-scalability-47c794da5c0b?source=rss-48da3986742c------2</link>
            <guid isPermaLink="false">https://medium.com/p/47c794da5c0b</guid>
            <category><![CDATA[database-sharding]]></category>
            <category><![CDATA[graph-database]]></category>
            <category><![CDATA[distributed-database]]></category>
            <category><![CDATA[scalable-database-design]]></category>
            <category><![CDATA[data-architecture]]></category>
            <dc:creator><![CDATA[Ricky Sun]]></dc:creator>
            <pubDate>Thu, 12 Jan 2023 07:47:21 GMT</pubDate>
            <atom:updated>2025-08-14T01:52:43.219Z</atom:updated>
            <content:encoded><![CDATA[<p>By Prof. Seasoned Edges</p><p>Having a distributed and scalable graph database system is highly sought after in many enterprise scenarios. This, on the one hand, is heavily influenced by the sustained rising and popularity of big-data processing frameworks, including but not limited to, Hadoop, Spark and NoSQL databases; on the other hand, as more and more data are to be analyzed in a correlated and multi-dimensional fashion, it’s getting difficult to pack all data into one graph on one instance, having a truly distributed and horizontally scalable graph database is a must-have.</p><h3>Do Not Be Misled</h3><p>Designing and implementing a scalable graph database system has never been a trivial task. There are countless number of enterprises and particularly Internet giants have explored ways to make graph data processing scalable. Nevertheless, most solutions are either limited to their private and narrow use cases, or offering scalability in vertical fashion with hardware acceleration which only proves, again, that the reason why mainframe architecture computer was deterministically replaced by PC-architecture computer in the 90s was mainly because vertical scalability is generally considered inferior and less-capable-n-scalable than horizontal scalability, period. It has been a norm to perceive that distributed databases use the method of adding cheap PC(s) to achieve scalability (storage &amp; compute), and attempt to store data once and for all on demand. However, doing the same cannot achieve equivalent scalability without massively sacrificing query performance on graph systems.</p><p>Why scalability in a graph (database) system is so difficult (to get)? The primary reason is that graph system is high-dimensional, this is in deep contrast to traditional SQL or NoSQL systems which are predominantly table centric, essentially columnar and row stores (and KV stores in a more simplistic way) and have been proved to be relatively easy to implement with a horizontally scalable design.</p><p>A seemingly simple and intuitive graph query may lead to deep traversal and penetration of large amount of graph data, which tends to otherwise cause a typical BSP (Bulky Synchronous Processing) system to exchange heavily amongst its many distributed instances, therefore causing significant (and unbearable) latencies.</p><p>On the other hand, most existing graph systems prefer to sacrifice performance (computation) while offering scalability (storage).<a href="#_msocom_1">[cj1]</a> This would render such systems impractical and useless in handling many real-world business scenarios. A more accurate way to describe such systems is that they probably can store a large amount of data (across many instances) but cannot offer adequate graph-computing power — to put it another way, these systems fail to return with results when being queried beyond meta-data (nodes &amp; edges).</p><p>This article aims to demystify the scalability challenge(s) of graph database, meanwhile putting a lot of focus on performance issues. Simply put, you will have better and unobstructed understanding about scalability and performance in any graph database system, and gain more confidence in choosing your future graph system.</p><p>There are quite a bit of noises in the market about graph database scalability, some vendors claim they have unlimited scalability while others claim to be the first enterprise-grade scalable graph databases. Who should you believe or follow? The only way out is to equip yourself with adequate knowledge about scalability in graph database system, so that you can validate it by yourself and don’t have to be misled by all those marketing hypes.</p><p>Admittedly, there are many terms for graph database scalability, some can be dearly confusing, to name a few: HA, RAFT or Distributed Consensus, HTAP, Federation, Fabric, Sharding, Partitioning, etc.</p><p>Can you really tell the difference, sometimes minute, and often with overlapping features, of all these terms? We’ll unravel them all.</p><h3>3 Schools of Distributed Graph System Architecture Designs</h3><p>First, make sure you understand the evolution pathway from a standalone (graph database) instance to a fully distributed and horizontally scalable cluster of graph database instances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*U6eSLAVNc3ylpkZAkacb6w.png" /><figcaption><em>Graph 1: Evolution of Distributed (Graph) Systems</em></figcaption></figure><p><strong>A distributed system may take many forms</strong>, and this rich diversification may lead to confusion. Some vendors misleadingly (and ironically) claim their database systems to be distributed even on a single piece of underpinning hardware instance, while other vendors claim their sharded graph database cluster can handle zillion-scale graph dataset while, in reality, the cluster can’t even handle a typical multi-hop graph query or graph algorithm that reiteratively traverse the entire dataset.</p><p>Simply put, there are <strong>ONLY 3</strong> schools of scalable graph database architecture designs as captured in the table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EwQNXEFYoX3A85X6fu_ERw.png" /><figcaption><em>Table 1: Comparison of 3 schools of Distributed Graph Systems</em></figcaption></figure><h3><strong>HTAP Architecture</strong></h3><p>The <strong>1st school</strong> is considered a natural extension to the master-slave model, and we are calling it distributed consensus cluster where typically 3 instances form a graph database cluster. The only reason to have 3 or an odd number of instances in the same cluster is because it’s easier to vote for a leader of the cluster.</p><p>As you can see, this model of cluster design may have many variations, for instance, Neo4j’s Enterprise Edition v4.x supports the original RAFT protocol and only 1 instance handles workload, while the other 2 instances passively synchronizing data from the primary instance — this of course is a naïve way of putting RAFT protocol to work. A more practical way to handle workload is to augment the RAFT protocol to allow all instances to work in a load-balanced way. For instance, having the leader instance handle read-and-write operations, while the other instances can at least handle read type of queries to ensure data consistencies across the entire cluster.</p><p>A more sophisticated way in this type of distributed graph system design is to allow for HTAP (Hybrid Transactional and Analytical Processing), meaning there will be varied roles assigned amongst the cluster instances, the leader will handle TP operations, while the followers will handle AP operations, which can be further broken down into roles for graph algorithms, etc.</p><p>The pros and cons of graph system leveraging distributed consensus include:</p><p>· Small hardware footprint (cheaper)</p><p>· Great data consistency (easier to implement)</p><p>· Best performance on sophisticated and deep queries</p><p>· Limited scalability (relying on vertical scalability)</p><p>· Difficult to handle a single graph that’s over 10 billion plus nodes &amp; edges.</p><p>What’s illustrated below is a novel HTAP architecture from Ultipa with key features like:</p><p>· High-Density Parallel Graph Computing</p><p>· Multi-Layer Storage Acceleration (Storage is in close proximity to compute)</p><p>· Dynamic Pruning (Expedited graph traversal via dynamic trimming mechanism)</p><p>· Super-Linear Performance (i.e., when computing resource such as the number of CPU cores is doubled, the performance gain can be more than doubled)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*BG8jyBF1958oOWGZzsZ0Vw.png" /><figcaption><em>Graph 2: HTAP Architecture Diagram by Ultipa Graph</em></figcaption></figure><p>Note that such HTAP architecture works wonderfully on graph data size that’s below 10B nodes + edges. Because lots of computing acceleration are done via in-memory computing, and if each billion nodes &amp; edges consume about 100GB of DRAM, it may take 1TB of DRAM on a single instance to handle a graph of 10 billion nodes and edges.</p><p>The upside of such design is that the architecture is satisfactory for most real-world scenarios. Even for G-SIBs (Globally Systemically Important Banks), a typical fraud detection, asset-liability management or liquidity risk management use case would consume around 1 billion data, a reasonably sized virtual machine or PC server can decently accommodate such data scale and be very productive with an HTAP setup.</p><p>The downside with such design is the lack of horizontal (and unlimited) scalability. And this challenge is addressed in the 2nd &amp; 3rd school of distributed graph system designs (see Table 1).</p><p>The two graphs below show the performance advantages of HTAP architecture. There are 2 points to watch out for:</p><p>1. Linear Performance Gain: A 3-instance Ultipa HTAP cluster’s throughput can reach ~300% of a standalone instance. The gain is reflected primarily in AP type of operations such as meta-data queries, path/k-hop queries, and graph algorithms, but not in TP operations such as insertions or deletions of meta-data, because these operations are done primarily on the main instance before synchronized with secondary instances.</p><p>2. Better performance = Lower Latency &amp; Higher Throughput (TPS or QPS).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jkgdjj0FK82DxwVPVWEbaA.png" /><figcaption><em>Graph 3: Performance Advantages of HTAP Architecture</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tk6lZjmas8_NjXmDOt87rA.png" /><figcaption><em>Graph 4: TPS comparison of Ultipa &amp; Neo4j</em></figcaption></figure><h3>Grid Architecture</h3><p>In the <strong>2nd school</strong>, there are also quite a few naming variations for such type of distributed and scalable graph system design (some are misleading). To name a few: Proxy, Name server, MapReduce, Grid or Federation. Ignore the naming differences, the key difference between the 2nd school and the 1st school lies with the name server(s) functioning as a proxy between client side and server side.</p><p>When functioning as a proxy server, the name server is only for routing queries and forwarding data. On top of this, except for running graph algorithm, the name server has the capacity to aggregate data from the underpinning instances. Furthermore, in federation mode, queries can be run against multiple underpinning instances (query-federation); for graph algorithms, however, federation’s performance is poor (due to data migration, just like how map-reduce works). Note that the 2nd school is different from 3rd school in one area: data is functionally partitioned but not sharded in this school of design.</p><p><strong>For graph datasets</strong>, functional partitioning is the logical division of graph data, such as per time series (horizontal partitioning) or per business logics (vertical partitioning).</p><p>Sharding, on the other hand, aims to be automated, business logic or time series ignorant. Sharding normally considers the location of network storage based partitioning of data, it uses various redundant data and special data distribution to improve performance, such as making cuttings against nodes and edges on the one hand, and replicating some of the cut data for better access performance on the other hand. In fact, sharding is very complex and difficult to understand. Automated sharding, by definition, is designed to treat unpredictable data distribution with minimal-to-zero human intervention and business-logic ignorant, but this ignorance can be very problematic when facing business challenges entangled with specific data distribution.</p><p>Let’s use concrete examples to illustrate this. Assuming you have 12 months’ worth of credit card transactions data. In artificial partition mode, you naturally divide the network of data into 12 graph sets<a href="#_msocom_2">[cj2]</a> , one graph set with one-month transactions on each cluster of 3 instances, and this logic is predefined by the database admin. It emphasizes on dividing the data via the metadata of the database and ignoring the connectivity between the different graph sets. It’s business-friendly, it won’ slow down data migration and has good query performance. On the other hand, in auto-sharding mode, it’s up to the graph system to determine how to divide (cut) the dataset and the sharding logic is transparent to database admin. But it’s hard for developers to immediately figure out where the data is stored, therefore leading to potential slow data migration problems.</p><p>It would be imprudent to claim that auto-sharding is more intelligent than functional partitioning, simply because auto-sharding involves less human intervention.</p><p>Do you feel something is wrong here? It’s exactly what we are experiencing with the ongoing rising of artificial intelligence, we are allowing machines to make decisions on our behalf, and it’s not always intelligent! (In a separate essay, we will cover the topic of global transition from artificial intelligence to augmented intelligence, and why graph technology is strategically positioned to empower this transition.)</p><p>In Graph-5, a grid architecture pertaining to the 2nd school of design is illustrated, the 2 extra components added on top of Graph-2’s HTAP architecture are name server(s) and meta server(s). Essentially all queries are proxied through the name-server, and the name-sever works jointly with the meta-server to ensure the elasticity of the grid, the server clusters instances are largely the same as the original HTAP instance (as illustrated in Graph-2).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eMOEnzvlJ_R90p_aDABz7w.png" /><figcaption><em>Graph 5: Grid Architecture w/ Name Server &amp; Meta Server</em></figcaption></figure><p>Referring to Table-1, the pros and cons of the grid architecture <a href="#_msocom_3">[cj3]</a> design can be summarized as:</p><p>· All pros/benefits of a typical HTAP architecture are retained.</p><p>· Scalability is achieved with performance intact (comparing to HTAP architecture).</p><p>· Restricted scalability — server clusters are partitioned with DBA/admin intervention.</p><p>· Introduction of name-server/meta-server making cluster management sophisticated.</p><p>· The name-server is critical and complex in ensuring business logics are performed distributively on the server clusters and with simple merge &amp; aggregation functionalities on it before returning to the clients.</p><p>· Business logics may be required to cooperate with partitioning and querying.</p><h3>Shard Architecture</h3><p>Now, we can usher in the 3rd school of distributed graph system design with unlimited scalability — the shard (see Table-1).</p><p>On the surface, the horizontal scalability of a sharding system also leverages name server and meta server as in the 2nd school of design, but the main differences lie with:</p><p>1. Shard servers are genuinely sharded.</p><p>2. Name servers do NOT have knowledge about business logics (as in 2nd school) directly. Indirectly, it can roughly judge the category of business logics via automatic statistics collection. This decoupling is important, and it couldn’t be achieved elegantly in the 2nd school.</p><p>The shard architecture has some variations, some vendor calls it fabrics<a href="#_msocom_4">[cj4]</a> (it’s actually more like grid architecture in the 2nd school), others call it map-reduce, but we should deep dive to the core data processing logics to unravel the mystery.</p><p>There are only two types of data processing logics in shard architecture:</p><p>· Type 1: Data are processed mainly on name servers (or proxy servers)</p><p>· Type 2: Data are processed on sharded or partitioned servers as well as name servers.</p><p>Type 1 is typical as you see in most map-reduce systems such as Hadoop, data are scattered across the highly distributed instances, however, they need to be lifted, shifted over to the name servers before they are processed there.</p><p>Type 2 is different in that the shard servers have the capacity to locally process the data (this is called: compute near or collocated with storage or <strong>d</strong>ata <strong>c</strong>entric <strong>c</strong>omputing) before they are aggregated and secondarily processed on the name servers.</p><p>As you would imagine, type 1 is easier to implement as it’s a mature design scheme by many big-data frameworks, however, type 2 offers better performance with more sophisticated cluster design and query optimization. Shard servers in type-2 offer computing power, while type-1 have no such capability.</p><p>The graph below shows a type-2 shard design:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*-0HAB4bS2bVWpZfWHztPUg.png" /><figcaption><em>Graph 6: Shard Architecture w/ Name Server &amp; Meta Server</em></figcaption></figure><p>Sharding is nothing new from a traditional SQL or NoSQL big-data framework design perspective. However, sharding on graph data can be a Pandora box, and here is why:</p><p>· Multiple shards will increase I/O performance, particularly data ingestion speed.</p><p>· But multiple shards will significantly increase the turnaround time of any graph query that spans across multiple shards, such as path queries, k-hop queries, and most graph algorithms (the latency increase can be exponential!).</p><p>· Graph <strong>query planning and optimization</strong> can be extremely sophisticated, most vendors today have done very <strong>shallowly</strong> in this front, and there are tons of opportunities in deepening query optimization on-the-fly:</p><p>o Cascades (Heuristic vs. Cost)</p><p>o Partition-pruning (shard-pruning actually)</p><p>o Index-choosing</p><p>o Statistics (Smart Estimation)</p><p>o Pushdown (making computing as close to the storage as possible) and more.</p><p>In Graph-7, we captured some preliminary findings on Ultipa HTAP cluster and Ultipa Shard cluster, as you can see, data ingestion speed improves by 4x (super-linear), but everything else tends to be slower by 5X or more (PageRank slower by 10x, LPA by 16X, etc.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vsFGqicH89UNhFN9vA9QRg.png" /><figcaption><em>Graph 7: Preliminary findings on performance difference between HTAP &amp; Shard Architecture</em></figcaption></figure><h3><strong>Stay Tuned</strong></h3><p>There are tons of opportunities to continuously improve the performance of the shard architecture. The team at Ultipa has realized that having a truly advanced cluster management mechanism and deeper query optimization on a horizontally scalable system are the keys to achieving endless scalability and satisfactory performance. We’ll report back with our new findings on the latest Ultipa v5.0 in March.</p><p>Lastly, the 3 schools of distributed graph system architectures illustrate the diversity and complexity involved when designing a sophisticated and competent graph system. It’s hard to say one architecture is absolutely superior to another, given cost, subjective-preference, design philosophy, business logics, complexity-tolerance, serviceability and many other factors — it would be prudent to conclude that the direction of architecture evolution for the long term clearly is to go from the 1st school to the 2nd school and eventually to the 3rd school, but most customer scenarios can be satisfied with the first two schools, and human-intelligence (DBA intervention) still makes pivotal sense in helping achieving an equilibrium of performance and scalability, particularly in the 2nd and 3rd school of designs.</p><p>Long live the formula:</p><p><em>Graph Augmented Intelligence = Human-intelligence + Machine’s-graph-computing-power</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=47c794da5c0b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ultipa/what-should-you-know-about-graph-databases-scalability-47c794da5c0b">What should you know about Graph Database’s Scalability?</a> was originally published in <a href="https://medium.com/ultipa">Ultipa Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Graph Database and Query Language 101: Speed & Simplicity (Part-II)]]></title>
            <link>https://medium.com/ultipa/graph-database-and-query-language-101-speed-simplicity-part-ii-ada397fea6e?source=rss-48da3986742c------2</link>
            <guid isPermaLink="false">https://medium.com/p/ada397fea6e</guid>
            <category><![CDATA[graph]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[graph-database]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Ricky Sun]]></dc:creator>
            <pubDate>Thu, 18 Aug 2022 02:36:47 GMT</pubDate>
            <atom:updated>2025-08-14T01:55:22.657Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part II</h3><p>We have covered the SPEED and performance part in <a href="https://medium.com/@RickySun/graph-database-and-query-language-101-speed-simplicity-f8ca7e600ec7">Part-I of this series of articles</a>. It’s time to talk about a unique feature by graph database, which really should be universally available — recursive operations during networked (linked-entity) analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hbv3sewifIl_gKyf" /></figure><p>Taking UBO (Ultimate Beneficiary Owner) identification as an example, regulators around the world are trying to identify the ultimate business owners who may be hiding behind layers of intermediate shell companies. On the other hand, large enterprises or individuals tend to diversify their business operations by investing in other companies which in turn further invest downstream and eventually building out a gigantic network of industrial and commercial graph (a special form of knowledge graph if you will) with millions of companies and natural persons (individuals who are owners, stake-holders, senior management, or legal representatives).</p><p>In Graph-9, we illustrate how SQL and GQL handles UBO-identification respectively. As you can see, it is very awkward for SQL to handle recursive queries which is to start from a subject company and to trace (penetrate) upstream for stakeholders for up to five layers. You have to write many lines of SQL code, which is counter-intuitive! And, on a small table of only 10,000 rows (investment relationship), SQL needs dozens of seconds to churn out results, which means it will take forever on larger tables of millions to billions of rows.</p><p>On the other hand, writing GQL can be surprisingly straightforward. In Graph-10, an easy-to-understand one-liner was fired up from Ultipa Manager (an integrated Web-GUI for Ultipa graph database management) to query a large dataset of millions of entities for investors that are hiding 5-hop away from the subject company. The results in linked-entity format are returned in real-time with clear identification of the subject company, intermediate companies, and ultimate investors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*olfIHRlnTToi8uSU" /></figure><p>In Graph-9 and Graph-10, we briefly illustrate the capability difference between SQL and GQL when conducting networked/linked-analysis. It’s worthwhile expanding on this topic analytically, let’s revisit our query from Part-I:</p><p>Find 200 pairs of users (starting from users whose age=22, ending with users age=33) who are exactly X-hop away from each other. Assuming X=[1–20]</p><p>From query language design perspective, how would you define and articulate the question and fetch the answer accordingly?</p><p>The key part is to make the query narrative, which means naturally resembling human language therefore easy to understand, meanwhile making it recursive and succinct so that you don’t have to labor on writing your logic in a convoluted (i.e., SQL) way.</p><p>n({age==22}).e()[10].n({age==39}) as paths return paths limit 200</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/0*D4Lr5Mia-jkduBC9" /></figure><p>The above one liner written in Ultipa GQL was pretty self-explainable, but just in case you are in shock of getting the gist of it, the recursive part lies with the e()[10], which specifies that the search depth is exactly 10-hop, of course you can modify it to be e()[5:11] which stands for traversing depth from 5 to 11-hop inclusively, or e()[:10] which means up to 10-hop, or e()[*:10] where * stands for shortest-path (BFS) search but with max traversal depth by 10.</p><p>Of course, the query can be expanded in many ways, such as filtering by attributes on the edges, or doing mathematical and statistical calculations. For instance, searching along the path by identifying with only relationship=“Invest”, the results are captured in Graph-12, with the following updated query:</p><p>n({age==22}).e({name==”Invest”})[3:5].n({age==39}) as p return p limit 200</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tCv5s_Ch5aRrWJEf" /></figure><p>Empowered with powerful recursive queries, a lot of things can be accomplished with great simplicity. In the remaining section, we would like to draw comparisons among Neo4j, TigerGraph and Ultipa. As the Chinese saying goes: Truth comes out of comparison. Let’s reveal the truth next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FncFdS0j4PlmT3p_" /></figure><p>In Graph-13, we compare Cypher, Gremlin, UQL and GSQL while having them describe a typical linked-data triple, where a labeled entity (Person) has a job(relation) function as a Chef (Job). As you can clearly see, the following takeaways are listed here for review:</p><p>· GSQL is incredible complex, and for such a simple triple, it would require writing 10+ lines of code, for a real complex scenario, you just can’t imagine your coding and maintenance experience.</p><p>· Gremlin’s language design is weird, it looks both verbose and confusing, some functions can clearly be consolidated into fewer chained components.</p><p>· The main difference between Cypher and UQL lies in the Cypher still uses WHERE sub-clause as SQL does, but UQL allows this part to be done naturally inline.</p><p>· Lastly, UQL supports both schematized and schema-free, the former is to allow schematized (strong-type) matching, while the latter is more flexible to support dataset that’s not-yet-schematized. In UQL, this is called demi-schema, it offers great flexibility when dealing with heterogeneous types of data.</p><p>Most of today’s system implementing smart recommendation involve some sort of social recommendation, which was coined: collaborative filtering. The core concept is actually easy — it’s essentially “graphy”:</p><p>To recommend something, some products, to a customer, we usually first look through the customer’s friends’ shopping behavior, and recommend based on the findings…</p><p>Let’s see how the 3 graph systems tackle the collaborative filtering challenge.</p><p>Neo4j uses Cypher to write the below logic:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Sqialkt9m2yHQ05V" /></figure><p>The Cypher logic goes as:</p><p>1. Find Ricky’s friends and friends-of-friends (1st part)</p><p>2. Collect distinct (unique) friends</p><p>3. Find what products these friends bought (2nd part)</p><p>4. Return the top-5 purchased products</p><p>The Neo4j logic has a prerequisite on the dataset, it assumes that the friends type of relationship are available. This is a major assumption that most e-commerce websites can hardly satisfy, not even on the world’s largest Taobao or T-mall sites — they don’t keep track of the user’s social network information!</p><p>The other obvious limitation of Neo4j &amp; Cypher is that it does NOT allow deep traversal on the graph, you will have to pause after searching for 1 to 2 hop deep and construct the intermediate dataset (friends), then continue with the 2nd part of the search. This symptom can be aggravated in scenarios like UBO (Ultimate Business Owner) finding, where you need deep traversal sometimes 10 or even 20-hop deep. Neo4j can NOT handle that kind of scenario with simplicity or speed.</p><p>TigerGraph uses GSQL (short for GraphSQL) to conduct “collaborative filtering”, the below chunk of code from Tigergraph online documents illustrate how this can be done. First, supply a set of users (input_user), next, find all target users that liked the inputed users, thirdly, proceed from these target users and continue to search for users that they liked and aggregate (ACCUM) and return in ordered fashion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nxCx83WmemcEy7wd" /></figure><p>First of all, this example essentially is a 2-hop template-based path search with data aggregation. Besides, the above GSQL is way too complicated, awkward and lengthy. One key drawback about SQL is that it drains a lot of human cognition power — a SQL stored procedure longer than 10 lines is getting unbearably difficult to interpret. And this cluster of GSQL code gives us all the bad memories about SQL.</p><p>Just for the sake of comparison and show-case of simplicity, if UQL were to accomplish the same, you only need to write:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/0*iI7T9a-DXzPYvbPg" /></figure><p>Pretty simple and straightforward, isn’t it?</p><p>· First, start a template search (t()), starting from a set of nodes n();</p><p>· Second, traverse edges named “Liked By” (or any other attribute);</p><p>· Third, continue to traverse edges typed “Liked”, reaching nodes aliased c;</p><p>· Fourth, return distinct c and group by b.</p><p>Let’s get back to our original question on collaborative filtering, on an e-commerce graph dataset, it’s comprised of customers, products and shopping or browsing behavior connecting users and products. Think in a very graphy way, you will basically need to achieve the following steps:</p><p>1. Starting from a user, find all products (viewed, added or purchased)</p><p>2. Find all users that have had certain actions (i.e., purchase) on these products</p><p>3. Find all other products that are taken actions by users in Step 2.</p><p>4. Return some of the products (i.e., top ranked) and recommend to the user.</p><p>The results are shown on the following diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Fw8QFqy3QPE2_SOQ" /></figure><p>The UQL used to realize this was a template-based path query, as shown from below, which narrates the path filtering criteria as defined from above. The eventual to-be-recommended products may be further filtered, ranked or processed in a more sophisticated way, but the query language by itself is ostensibly self-evident and easy to digest:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/0*tbKfd0Cp86In6UxR" /></figure><p>In this series of articles, we have discussed the two key features of graph database:</p><p>· Speed: If speed and performance are not important, what else can be more important? The whole promise of graph database is rooted in unprecedented data penetration capabilities which directly translates into speed!</p><p>· Simplicity: Graph data is high-dimensional, and SQL is far from being capable of handling graph queries. If we were to design GQL, we want the queries to be humanly easy-to-digest and, of course, easy-to-code, so that secondary-development atop graph database can offer faster time-to-value and time-to-market.</p><p>The beauty of a data query language is best manifested, not in its complexity, but in its simplicity. It should be easy to understand, therefore requiring minimum cognitive loading, and it should be lightning fast! All the tedious and formidable complexities should be shielded off by the underpinning system therefore transparent to human users. It’s like the ancient Greek titan Atlas carrying the Earth on his shoulder, and not to burden the users living on the Earth (=users of the query language and the database).</p><p>Writing tens of or even hundreds of lines of SQL code has aggravated cognitive load. The concept we believe in is that:</p><p>· Anyone who understands business can master a graph database query language, not necessarily a data scientist.</p><p>· The query language should be extremely easy to use, and all the complexities are embedded within the underlying database, and transparent to users.</p><p>· Lastly, graph database has the huge potential to replace a significant share of the SQL workload, in a 2021 report by Gartner, it’s predicted that 80% of all business intelligence innovations will be powered by graph analytics by 2025, the growth trajectory of graph is exponential!</p><p>Some people have claimed that RDBMS &amp; SQL will never be replaced, we found this hard to believe, RDBMS had replaced navigational databases in the 70s and been rather successful for almost 5 decades, but if history has taught us anything, it is that the obsession with anything will not last forever, this is especially true in the world of Internet and Information Technologies.</p><blockquote><em>“What you cherish, perish”</em></blockquote><blockquote><em>“What you resist, persist”</em></blockquote><blockquote><em>If human brains were to be the ultimate database,</em></blockquote><blockquote><em>Graph database is the shortest path to be there.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ada397fea6e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ultipa/graph-database-and-query-language-101-speed-simplicity-part-ii-ada397fea6e">Graph Database and Query Language 101: Speed &amp; Simplicity (Part-II)</a> was originally published in <a href="https://medium.com/ultipa">Ultipa Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Graph Database and Query Language 101: Speed & Simplicity (Part-I)]]></title>
            <link>https://medium.com/ultipa/graph-database-and-query-language-101-speed-simplicity-f8ca7e600ec7?source=rss-48da3986742c------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8ca7e600ec7</guid>
            <category><![CDATA[graph]]></category>
            <category><![CDATA[graph-database]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Ricky Sun]]></dc:creator>
            <pubDate>Thu, 18 Aug 2022 02:33:23 GMT</pubDate>
            <atom:updated>2025-08-14T01:55:11.459Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part 1</h3><p>Note: This is the 1st part of a series of articles talking about Graph Database &amp; Graph Query Language focusing on the two most relevant perspectives — Speed &amp; Simplicity. In part one we’ll be showing a case study explaining how truly high performance computing can help speed up graph queries and analytics; In the 2nd part, we’ll be touching base with query language simplicity with real-world examples.</p><p>We have learned that relational databases use SQL and two-dimensional tables to model after the world, this is in deep contrast to how graph databases address the problem. In short, graph databases use high-dimensional data modeling strategies to 100% resemble the world — and because the real-world is highly dimensional, graph is considered 100% natural. This naturalness comes with a cost, especially for people who are accustomed to the trade-off hypothesis or the zero-sum game theories. That cost is that you have to be brave enough and smart enough to think out of the box, out of the existing framework, out of the woods, out of the limited belief that there is NO better way of modeling the real-world problems beyond SQL or RDBMS!</p><p>In today’s business worlds, we are seeing more and more businesses relying on graph analytics and graph computing to empower their business decisions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qx4RypQD0ASTYtmG" /></figure><p>In a series of reports from November 2019 to June 2021, Gartner predicted that Graph analytics is not only 1 of the top 10 Data &amp; Analytics trends (see the above diagram), but also laid out that: By 2025, 80% business decisions innovation worldwide will be powered by graph analytics.</p><p>Now, let’s pause for a second and ponder on the top features you would desire graph databases to offer? What are they? This is a question equivalent to asking: What features are missing in SQL and RDBMS?</p><p>As always, this is an open question and the answer can be highly diversified and warrants no standardization. However, out of all possible answers, 3 things come out of our gut feeling, and if you think deeply, you will resonate with us, they are:</p><p>· Simplicity (i.e., Enjoy writing 100s of lines of SQL?)</p><p>· Recursive Operations (i.e., Recursive Queries)</p><p>· Speed (i.e., Real-time-ness)</p><p>In a reversed order, let’s go through this 3-strike-out list: Speed and lightning fast performance is deeply rooted in the original promise of what graph database offers to the world. Because conventional table joins are insanely slow to churn out data due to the well-known cartesian-product or table-scan problem, a fundamental change of the way we model and process data therefore is both necessary and pivotal.</p><p>If you think from the high-dimensional space, graph is the most natural way to connect the dots (vertices) by connecting entities into a network, a tree or a graph, and working against the graph is simply to find the neighbors of any vertex, paths reaching to or leaving from the vertex, common attributes sharing between any pair of vertices (similarity), centrality of a graph or sub-graph, ranking of a vertex in the graph, and etc.</p><p>There are many ways to work against the graph and sought for values out of it. However, as the graph is getting larger, denser or the queries are getting more sophisticated, the performance degradation is considered a norm by most people… Well, this doesn’t have to be so.</p><p>Considering the following scenario, in a social (or financial) graph of millions of users who interact with each other, find up to 200 pairs of users (starting from users whose age=22, ending with users age=33) who are 1-hop, 2-hop, and all the way to 20-hop away from each other. Gauge the performance degradation, if there are any.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/0*Rk3JgprLSEcHdgse" /></figure><p>The above screenshots show the turn-around time (roundtrip latency) of running the queries on Ultipa Graph (a high-performance graph database that’s capable of real-time ultra-deep graph traversal) against a reinforced Amazon-0601 Graph Dataset (3.5 Million Edges and 0.5 Million Vertices with multiple attributes like id, name, age, type of relationship, rank, and etc.). These queries traverse the graph based on the template path query criteria, starting from 1-hop, gradually going deeper to 20-hop. The following chart captures the query latency comparison:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iBUYmc0_7eibit4v" /></figure><p>As you can see that when we are traversing deeper into the graph, the overall latency growth trend stays flat or sub-linear instead of being exponential. There is no obvious performance degradation. If you think about locating 200 paths in a graph, each with a formidable length of 20 (20 edges and 21 vertices along the path), plus the filtering parameters of starting users having age=22, ending users age=33, the complexity of such kind of queries is pretty formidable.</p><p>This almost flat, definitely sub-linear, query latency growth is achieved with 3 core technologies as illustrated in the below diagram:</p><p>· High-Density Parallel Graph Computing</p><p>· Ultra-Deep Graph Traversal (and Dynamic Graph Pruning)</p><p>· Multi-layer Storage and Computing Acceleration</p><p>And, they jointly and interwovenly bring about 3 major benefits:</p><p>· Faster Time-to-Value and Time-to-Market</p><p>· Much Lower TCO</p><p>· Superb Usability and user experience</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gGtJz2pstSaayCzM" /></figure><p>Graph DBMS, unlike relational DBMS, NoSQLs or Hadoop/Spark, scales out differently. The key difference lies with graph’s unique strength: Its ability to process data wholistically. In another word, graph DBMS must allow deep penetration (and correlation) of data, while all other RDBMS, NoSQL or Big Data frameworks do NOT have similar capacity. To draw an analogy in the context of RDBMS/SQL, to deeply penetrate data is to have multiple tables join together, and this can be disastrous.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cNUOEH0puwA9q2KU" /></figure><p>Graph-4 illustrates this problem: when 2 tables are joining together, the cartesian-product effect will create a large tables that multiplies rows of both tables together. If there are multiple large tables (i.e., millions of rows) involved, the size of the resulting table can be outrageous, and this outrageousness means it’s impossible to get your query results in real-time, not only that, practically speaking, you can’t even get a result in time.</p><p>In Graph-5, we illustrate that a typical attribution analysis (a.k.a., contribution analysis or performance/impact analysis) using SQL/RDBMS would need 1,000 server’s 10-day’s computing time and tons of storage to calculate all possibilities, the sheer computing complexity by SQL/RDBMS is unbearable, they are too rigid, too slow, and too complex.</p><p>A paradigm shift is due to happen and it’s graph computing and graph database coming to the rescue. Graph-6 illustrates the difference between a typical relational database and a graph system, as we are digging deeper (more hops) into data, the latency grows exponentially with relational DBMS, but linearly with graph DBMS. This makes all the differences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*j3b4z-CnPHw_H7Cf" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SB4ct3Pj9HNY5CIo" /></figure><p>There are graph vendors out there promoting unlimited scalability graph DBMS that’s capable of a single graph with trillions of vertices and edges. We found this to be a myth. And there are 2 points to bust this myth:</p><p>· In what scenarios you have to use a single graph with trillions of vertices/edges? A trillion-scale graph must be disseminated into many physical instances, for any query that requires processing against all instances while they all depending on each other for intermediate results will exponentially slow down the overall system performance, this is a typical dilemma of any BSP (Bulky Synchronous Processing ) system. Can you imagine how slow a large BSP system can be if you ask it to handle 3-hop or deeper graph queries?</p><p>· A wiser design of a horizontally distributed graph DBMS is to orchestrate a system that contains multiple (or many) graphs. Just as you would for a RDBMS with many tables. Typical scenarios like financial transactions, which tend to have timestamps, and you can logically disseminate one year’s transactions into 12 months, and each month’s transactions can be self-contained within a single graph (essentially connecting all transacting accounts with transactions that happen in the current month), and jointly, 12 months’ worth of graphs constitute a logically complete jumbo graph.</p><p>The points in the above paragraph should remind the readers who wonder why JanusGraph, AWS Neptune or ArangoDB (and a handful other commercial graph products) are so sluggish and can NOT return in a timely fashion while trying to accomplish the same kind of graph queries. Their core weakness lies in their unwise and blindly complicated distributed architecture design.</p><p>Putting distribution strategy aside, the most important techniques to accelerate graph queries (like path-finding or k-hopping) are through density computing and deep-traversal-n-runtime-pruning. However, these 2 supposedly fundamental techniques are long forgotten by many programmers. Admittedly, we are living in a world of high level programming languages, millions of newer generation young programmers are believing that the omni-potent power and ease of Python is ruling the world. However, in terms of processing power, it’s simply problematic, so is Java and any languages that dwell on GC (garbage collection).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-6nVAUhNrDQY2xc0" /></figure><p>In ACM’s 2018 Annual Turning Lecture , Turing Award recipients David Patterson and John Hennessy outlined the opportunity to speed up Python program using parallel C on 18-core Intel X86–64 CPU by over 62,000 times! It’s achieved with:</p><p>· Lower-level Programming Language (to maximize HW juice)</p><p>· Memory Optimization (ditto)</p><p>· Leverage CPU’s Advanced Instruction-Set Features (ditto)</p><p>All in all, these techniques are all contributing to the realization of “high-density” computing. Moreover, runtime dynamic graph pruning and deep graph traversal are realized with novel data structure design. In a previous essay by the authors titled The Specific Evolution of Graph Data Structures, a novel Adjacency Hash* data structures were introduced and can be used effectively to store graph topology and enable runtime filtering (a.k.a dynamic graph trimming or pruning) — of course, the most dramatic part is that: Most majority of the operations on the graph are having a big-O notation, in terms of time complexity, of O(1) — instead of O(log N) or O(N) or even O(N * log N) … This is critically important to ensure that when we traverse the graph deeply, we won’t have to deal with the exponentially growing search complexities. Here is an outline of the “Exponential Challenge”, taking the boosted Amazon-0601 Graph Dataset as an example.</p><p>· |E|/|V| = 8 (average degree of vertex)</p><p>· Complexity of Searching 1-hop paths ~= 8</p><p>· Complexity of Searching 2-hop paths ~= 64</p><p>· Complexity of Searching 5-hop paths ~= 8*8*8*8*8 = 32,768</p><p>· Complexity of Searching 10-hop paths ~= 1,073,741,824 (1Billion Searches)</p><p>· Complexity of Searching 200 pairs of 10-hop paths ~= 200 Billion!</p><p>Clearly, if the complexity was really over 200 Billion, there is NO WAY for any graph system to return within milliseconds. The above complexity was only theoretical. In reality, the maximum possible searching complexity for an O(1) operations is by no means exceeding the total number of edges in the graph, which stands at 3.5 million in our case, therefore, we can assume that starting from 7-hop (32768 * 8 * 8) to any deeper hops, the max complexity is capped at 3.5 million (for Amazon 0601, the real complexity, in terms of this Amazon dataset is actually caped at ~0.2 million, specially around 6 to 8-hop deep)</p><p>· Adjusted complexities for 6 to 10-hop paths = max 200,000 each hop.</p><p>Therefore, it’s possible to review the breakdown of the latency as in Graph-8:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MyndRNgpXlchos1C" /></figure><p>Comparing the numbers in Graph-8 against those in Graph-1 &amp; Graph-2, it’s NOT hard to come to the takeaways: By leveraging maximum concurrency (parallelization) and CPU caching, Ultipa achieved latency in close proximity to underlying hardware platform’s max throughput (Average latency of returning 200 10-hop deep paths is about 10ms, which is somewhere in between DRAM and In-CPU-Cache).</p><p>In <a href="https://medium.com/@RickySun/graph-database-and-query-language-101-speed-simplicity-part-ii-ada397fea6e">Part-II</a>, we’ll be showcasing how simple, intuitive yet power graph query language can make you life so much easier. Please stay tuned.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8ca7e600ec7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ultipa/graph-database-and-query-language-101-speed-simplicity-f8ca7e600ec7">Graph Database and Query Language 101: Speed &amp; Simplicity (Part-I)</a> was originally published in <a href="https://medium.com/ultipa">Ultipa Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to read Graph Database Benchmark (Part-II)]]></title>
            <link>https://medium.com/ultipa/how-to-read-graph-database-benchmark-part-ii-38ac7da38768?source=rss-48da3986742c------2</link>
            <guid isPermaLink="false">https://medium.com/p/38ac7da38768</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[graph-database]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Ricky Sun]]></dc:creator>
            <pubDate>Tue, 16 Aug 2022 05:43:33 GMT</pubDate>
            <atom:updated>2025-08-14T02:04:11.976Z</atom:updated>
            <content:encoded><![CDATA[<p>This is the 2nd part of <a href="https://medium.com/dev-genius/how-to-read-graph-database-benchmarks-part-i-996daa830a94">How to read Graph Database Benchmark series</a>, and is dedicated to query results validation of 3 types of operations:</p><p>· K-hop</p><p>· Shortest-Path</p><p>· Graph Algorithms</p><p>Let’s start with K-hop queries. First, let’s be clear of the definition of K-hop, there have been 2 types of K-hop:</p><p>1. Kth-Hop Neighbors, which are exactly K hops away from the source vertex</p><p>2. All Neighbors from the 1st Hop all the way to the Kth Hop</p><p>No matter which type of K-hop you are looking at, 2 essential points affect the correctness of the query:</p><p>A. K-hop should be implemented using BFS, instead of DFS</p><p>B. Results Deduplication: The results should NOT contain any duplicated vertices on the same hop or across different hops (Many graph DBMS do have this dedup problem as we speak.)</p><p>Some vendors use DFS to find shortest paths, this approach has 2 fatal problems:</p><p>A. Outright Wrong: There is a high probability that the results are wrong as DFS can’t guarantee a vertex belongs to the right hop (depth) which falls on a shortest-path.</p><p>B. Inefficient: On large and densely populated datasets, it’s impractical to traverse all possible paths in DFS fashion, for instance, Twitter-2010 has many hotspot nodes having millions of neighbors, any 2-hop or deeper queries mean astronomical computational complexity!</p><p>Let’s validate the 1-Hop result of vertex ID=27960125 in Twitter-2010, first, we start from the source file which shows 8 edges (rows of connecting neighbors), but, what exactly is its 1-hop?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9kEroWM7Ehd5UXYV" /></figure><p>The correct answer is 7! Because the node 27960125 has a neighbor ID=27498091 which appears twice, because there are 2 edges in between these 2 vertices. If we deduplicate, we have 7.</p><p>To expand here on K-hop queries, it is possible to launch K-hop per edge direction, the screenshots below show 4 ways for running K-hop: regular bidirectional K-hop, outbound K-hop, inbound K-hop and K-hop with all neighbors. These queries combined can help validate any K-hop query.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9x4uoABoQ7uvah5j" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OvZ2ltWZZnityMhY" /></figure><p>If we compare with publicly accessible Tigergraph’s K-hop query results, the vertex ID=27960125 has only 6 1-hop neighbors (shown in Graph 9), and the similar counting mistakes prevail throughout the entire published results on Github.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YtaPz_hj7-kVO-ID" /></figure><p>There are 3 possible causes for the Tigergraph K-hop problem:</p><p>· Data Modeling Mistake: Each edge is stored only once (unidirectionally), so that inverted edge traversal is impossible</p><p>· Query Method Mistake: The K-hop query is only conducted unidirectionally, instead of bidirectionally</p><p>· Code Implementation Bug: Such as results not being deduplicated, we’ll continue our examination in multiple hop queries next</p><p>Data modeling mistake is fatal, it means business logic will NOT be correctly reflected in the underpinning data models. Taking anti-fraud, AML or BI scenarios as an example, account A receives a transaction from account B, however, the system only stores an edge pointing from A to B (A →B) but not the inverted edge from B to A (B←A), this would make it impossible to trace the transaction starting from account B. This clearly is unacceptable.</p><p>Similarly, query method and coding-logic bugs are also serious, query should go bidirectionally instead of unidirectionally, the former is exponentially more complex when traversing multiple hops. The same holds true for storage space usage and data ingestion time.</p><p>If we continue to track vertex ID=27960125’s 2-hop result, the Tigergraph’s mistakes become more dramatic yet more difficult to validate. Note Tigergraph returns 1128 neighbors, but there are duplicated neighbors, and the result is only based on outbound edges. The correct number of 2-Hop neighbors should be 533108, the difference here is 473 times, 47,300%! Tigergraph’s query results carry 3 mistakes simultaneously: data-modeling mistake, query method mistake and deduplication mistake.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8BlbTqdAutPagLVk" /></figure><p>Unfortunately, Tigergraph’s problem isn’t an exception, we are seeing similar problems with Neo4j, ArangoDB and other graph systems as well. For instance, Neo4j by default does NOT deduplicate K-hop results, if you force it to dedup, it will run exponentially slower. And, ArangoDB has a form of shortest-path query that returns only 1 path, which is fast but ridiculously wrong when there are many paths in the results pool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R1b_LdwgEJjYbII9" /></figure><p>Graph 11 shows 4 ways for conducting K-hop queries using Ultipa CLI, which helps validate query results with ease and precision.</p><p>GUI tools can help us visually and intuitively query and interact with the dataset and results. The screenshot below shows the 1-hop query results for a vertex with 13 neighbors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u--gGj7cQZzHPChO" /></figure><p>How to validate the results from above? A simple way is to expand on the vertex in BFS fashion and expand for only 1 layer (hop). The diagram below shows when expanding on the subject vertex using spread() function, it has 19 connecting edges, but the unique neighbors are only 13, after 6 edges are de-duplicated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nkuImEogo4_xfgFD" /></figure><p>Now let’s address validations for shortest-path and graph algorithm.</p><p>We now know shortest-path is a special form of K-hop query, it’s different by 2 characteristics:</p><p>· High-dimensional results which are comprised of paths that are further composed of nodes and edges that are assembled in a certain sequence</p><p>· All paths are to be returned: Only calculating and returning 1 path is NOT enough. In financial fraud detection, AML, BI attribution analysis, there may exist many paths and returning just one is totally unacceptable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JXgd8umgDfL5HdV9" /></figure><p>The 2 diagrams above show that there are multiple shortest paths between ID 12 and 13, however, if you search by certain direction, you may get different answers as search filtering logic kicks in. For simple cases like this, you may grep the source file to find out that there are 2 bidirectional edges in between vertex ID 12 and 13, and if you do it recursively, you can validate more sophisticated queries. Note the keyword “recursively” I just used, it’s exactly why GQL/Graph-DBMS will be mainstream and SQL/RDBMS will fade away, because SQL’s weakest link is its inability to handle recursive queries, and GQL/Graph is automated on this.</p><p>Next, we use Jaccard Similarity algorithm as an example to illustrate how to validate algorithm results. Taking the below diagram as an example, to calculate Jaccard Similarity of the green vertex and red vertex, you must find their common neighbors (2) and total neighbors (5), so that the similarity = 2/5 = 0.4 = 40%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hDQIOFhvDBNC4PvP" /></figure><p>If the Jaccard Similarity algorithm is integrated, launching it directly will churn out the correct answer of 0.4. If you desire to write several lines of GQL to implement it in an organic and white-box way, you can do this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QUZZmDigNH91_Egt" /></figure><p>In Twitter-2010, computing Jaccard Similarity of any pair of vertices is to find each vertex’s 1-hop neighbors, taking ID=12 and ID=13 as an example, they both are hotspot supernodes having over 1 million neighbors, it would be impractical to validate the algorithm results manually. However, as long as you understand the logics under the hood of the algorithm, there are ways to divide and conquer it. The steps below show 2 ways to validate the algorithm:</p><p>1. Launch the Jaccard algorithm:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*I0ArH7kst7zE8WBn" /></figure><p>2. Validation Method 1: Using multiple GQLs to implement the Jaccard Similarity algorithm, the coding logic is the same as illustrated in Graph 17:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*em16zaFQyQtGDcPD" /></figure><p>3. Validation Method 2: First checking vertex ID 12 &amp; 13’s 1-hop neighbors:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*O_41cq7ZEb_4iYtI" /></figure><p>4. Validation Method 2: Finding all distinct paths that are exactly 2-hop deep between ID 12 &amp; 13, the number of paths is equal to the number of common neighbors of the two IDs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cAVVT21rZ-5Uc_Ee" /></figure><p>5. Validation Method 2: The similarity = Results-from-Step-4 / (12’s 1-hop + 13’s 1-hop — Results from step 4) = 0.15362638</p><p>6. As you can see, the result from step 2 and step 5 are exact. We know the answer is right.</p><p>In this series of articles (two), we traveled extra distance to give readers a comprehensive overview of how to read and digest a graph database system’s benchmark report, and methods to validate the results within. It’s utterly important to ensure the results are both complete and correct, after all, we are talking about data intelligence and infrastructure revolution, it would be pointless if a disruptive technology like graph DBMS can’t ensure the correctness of its data modeling or query results.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=38ac7da38768" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ultipa/how-to-read-graph-database-benchmark-part-ii-38ac7da38768">How to read Graph Database Benchmark (Part-II)</a> was originally published in <a href="https://medium.com/ultipa">Ultipa Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Read Graph Database Benchmarks (Part-I)]]></title>
            <link>https://medium.com/ultipa/how-to-read-graph-database-benchmarks-part-i-996daa830a94?source=rss-48da3986742c------2</link>
            <guid isPermaLink="false">https://medium.com/p/996daa830a94</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[graph-data-structure]]></category>
            <category><![CDATA[graph-database]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <dc:creator><![CDATA[Ricky Sun]]></dc:creator>
            <pubDate>Mon, 15 Aug 2022 11:28:08 GMT</pubDate>
            <atom:updated>2025-08-14T01:55:34.300Z</atom:updated>
            <content:encoded><![CDATA[<p>The main goal of this article is to introduce basic knowledge about graph database, graph computing and analytics, and to help readers interpret benchmark testing reports, and to validate if the results are correct and sensible.</p><h3>Basic Knowledge</h3><p>There are mainly 2 types of data operations in any graph database:</p><p>1. Meta-data Operation: Also known as operation against vertex, edge, or their attributes. There are mainly 4 types of meta operations, specifically CURD (Create, Update, Read or Delete).</p><p>2. High-dimensional Data Operation: By saying “high-dimensional”, we are referring to high dimensional data structures in the resulting datasets, for instance, a group of vertices or edges, a network of paths, a subgraph or some mixed types of sophisticated results out of graph traversal — essentially, heterogeneous types of data can be mixed and assembled in one batch of result therefore making examination of such result both exciting and perplexing.</p><p>High-dimensional data manipulation poses as a key difference between graph database and other types of DBMS, we’ll focus on examining this unique feature of graph DB across the entire article.</p><p>There are 3 main types of high-dimensional data operations, which are frequently encountered in most benchmark testing reports:</p><p>1. K-Hop: K-hop query against a certain vertex will yield all vertices that are exactly K-hop away from the source vertex in terms of shortest-path length. K-hop queries have many variations, such as filtering by certain edge direction, vertex or edge attribute. There is a special kind of K-hop query which is to run against all vertices of the entire graph, it is also considered a type of graph algorithm.</p><p>2. Path: Path queries have many variations, shortest-path queries are most used, with the addition of template-based path queries, circle-finding path queries, automatic-networking queries and auto-spread queries, etc.</p><p>3. Graph Algorithm: Graph algorithms essentially are the combination of meta-data, K-hop, path queries. There are rudimentary algorithms like degrees, but there are highly sophisticated algorithms like Louvain, and there are extremely complex, in terms of computational complexity, algorithms such as Betweeness-Centrality or full-graph K-hop, particularly when K is much larger than 1.</p><p>Most benchmark reports are reluctant to disclose how high-dimensional data operations are implemented, and this vagueness has created lots of trouble for readers to better understand graph technology. We hereby clarify, there are ONLY 3 types of implementations in terms of how graph data is traversed:</p><p>1. BFS (Breadth First Search): Shortest Path, K-hop queries are typically implemented using BFS method. It’s worth clarifying that you may use DFS to implement, say, shortest path finding, but in most real-world applications, particularly with large volumes of data, BFS is guaranteed to be more efficient and logically more straightforward than DFS, period.</p><p>2. DFS (Depth First Search): Queries like circle finding, auto-networking, template-based path queries, random walks desire to be implemented using DFS method. If you find it hard to understand the difference between BFS and DFS, refer to the below diagram and a college textbook on Graph Theory.</p><p>3. Combination of Both (BFS + DFS): There are scenarios where both BFS and DFS are applied, such as template-based K-hop queries, customized graph algorithms, etc.</p><p>Graph 1 illustrates the traversal difference between BFS and DFS. In short, under BFS mode, if qualified 1st hop neighbors are not all visited first, the access to any 2nd hop neighbor will not start, and traversal will continue in this way until all data(neighbors) are visited hop after hop. Based on such description, it’s not hard to tell that if a certain K-hop or shortest-path query only returns a pre-defined limited number of neighbors or paths (say, 1 or 10), it’s guaranteed that the query implementation is wrong! Because you do NOT know the total number of qualified neighbors or paths beforehand.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3no6F0VMRfNy6wnA" /></figure><h3>Typical Benchmark Datasets</h3><p>There are 3 types of datasets for graph system benchmarking:</p><p>1. Social or Road Network Dataset: Typical social network datasets like Twitter-2010 and Amazon0601 have been commonly used for graph database benchmarking. There are also road network or web-topology based datasets used in academic benchmarks such as UC Berkeley’s GAP.</p><p>2. Synthetic Dataset: HPC (High-Performance Computing) organization Graph-500 has published some synthetic datasets for graph system benchmark. International standard organization LDBC (Linked Data Benchmark Council) also use data-generating tools to create synthetic social-network datasets for benchmarking purposes.</p><p>3. Financial Dataset: The latest addition to the family of graph benchmark datasets is financial dataset, mostly are found in private and business-oriented setup, because the data are often based on real transactions. There are several public datasets such as Alimama’s e-commerce dataset, and LDBC’s FB (FinBench) dataset which is being drafted and to be released toward the end of 2022 (to replace the LDBC’s SNB dataset which does NOT reflect real-world financial business challenges in terms of data topology and complexity).</p><p>As graph database and computing technologies and products continue to evolve, it’s not hard to see that more graph datasets will emerge and better reflect real-world scenarios and business challenges.</p><p>Twitter-2010 (42 Million vertices, 1.47 Billion edges, sized at 25GB, downloadable from <a href="http://an.kaist.ac.kr/traces/WWW2010.html">http://an.kaist.ac.kr/traces/WWW2010.html</a>) is commonly used in benchmarking, we’ll use it as a level playing field to explain how to interpret a benchmark report and how to validate results in the report.</p><p>Before we start, let’s get ourselves familiar with a few concepts in graph data modeling by using the Twitter dataset as an example:</p><p>· Directed Graph: Every edge has a unique direction, in Twitter, an edge is composed of a starting vertex and an ending vertex, which map to the 2 IDs separated by TAB on each line of the data file, and the significance of the edge is that it indicates the starting vertex (person) follows the ending vertex (person). When modeling the edge in a graph database, the edge will be constructed twice, the first time as StartingVertex →EndingVertex, and the second time as EndingVertex →StartingVertex (a.k.a, the inverted edge), this is to allow traversing the edge from either direction. If the inverted edge is not constructed, queries’ results will be inevitably wrong (We’ll show you how later).</p><p>· Simple-graph vs. Multi-graph: If there are more than 2 edges of the same kind between a pair of vertices in either direction, it’s a multi-graph, otherwise, it’s a simple-graph. Twitter and all social network datasets are considered simple graphs because it models a simple following relationship between the users. In financial scenarios, assuming user accounts are vertices and transactions are edges, there could be many transactions in between two accounts, therefore many edges. Some graph databases are designed to be simple-graph instead of multi-graph, this will create lots of problems in terms of data modeling efficiency and query results correctness.</p><p>· Node-Edge Attributes：Twitter data does NOT carry any node or edge attribute other than the designated direction of the edge. This is different from transactional graph in financial scenarios, where both node and edge may have many attributes so that filtering, sorting, aggregation, attribution analysis can be done with the help of these attributes. There are so-called graph database systems that do NOT support the filtering by node or edge attributes, which are considered impractical and lack of commercial values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*E7W_5UMAbATnfn8d" /></figure><h3>Report Interpretation</h3><p>Generally speaking, there are at least 5 parts of a graph database benchmark report:</p><p>1) Testing Bed/Environment: Software and hardware environment, testing datasets, etc.</p><p>2) Data Ingestion: Volume data loading, database startup time, etc.</p><p>3) Query Performance: Queries against meta-data, K-hop, shortest-path, algorithm, etc.</p><p>4) Real-time Update: Modification to graph data’s topology (meta-data) then query against the updated data to validate results.</p><p>5) Feature Completeness &amp; Usability: Support of API/SDK for secondary development, GQL, management and monitoring, system maintenance toolkits and disaster recovery, etc.</p><p>Most benchmark reports would cover the first 3 parts, but not the 4th or 5th parts — the 4th part reflects if the subject database is more of the OLTP/HTAP type or OLAP focused, with the main difference lying at the capability to process dynamically changing datasets. Clearly, if a graph system claims to be capable of OLTP/HTAP, it must allow data to be updated and queried all the time, instead of functioning as a static data warehouse like Apache Spark, or data projections must be finished first before queries can be run against the just-finished static projection like Neo4j does.</p><h3>Testing Bed</h3><p>Almost all graph database products are built on X86 architecture as X86–64 CPU powered servers dominate the server marketplace. The situation has recently changed given the rise of ARM, due to its simplistic RISC (instead of X86’s CISC) therefore greener design. There are growing number of commodity PC servers basing on ARM CPUs nowadays. However, only 2 graph database vendors are known to natively support ARM-64 CPU architecture, they are Ultipa and ArangoDB, while other vendors tend to use virtualization or emulation methods which tend to be dramatically slower than native method.</p><p>The core hardware configurations are CPU, DRAM, external storage and network bandwidth. The number of CPU cores are most critical, as more cores mean potentially higher parallelism and better system throughput. Most testing beds settle on 16-core and 256GB configuration, however not all graph systems can leverage multiple cores for accelerated data processing, for instance, even enterprise edition Neo4j leverages up to 4-vCPU in terms of parallelization. If another system can leverage 16-vCPU, the performance gain would be at least 400%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FcHWE1mlUgk7ip7P" /></figure><p>The table above illustrates typical hardware configuration for graph system benchmarking. It’s worthwhile pointing out that graph database is computing first, unlike RDBMS, Data Warehouse or Data Lake systems which are storage first and compute is 2nd class citizen that’s attached to the storage engine. In another word, traditional DBMS addresses I/O-intensive challenges first, while graph system solves computing-intensive challenges first. While both challenges may intersect with each other, that’s why a performance-oriented graph database should set both computing and storage as its first-class citizens.</p><p>The software environment is trivial as all known graph systems settle on Linux OS, and leverage container and virtualization technologies for encapsulation and isolation.</p><h3>Data Ingestion</h3><p>Data ingestion has a few indicators to watch out for:</p><p>· Dataset volume, topological complexity (or density)</p><p>· Ingestion time</p><p>· Storage space</p><p>Data volume concerns the total number of vertices and edges, plus complexity (or density), which is normally calculated using the following formula (for directed simple graph):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/244/0*TKfs6h00_J7jeONl" /></figure><p>Where |V| annotates the total number of vertices, and |E| for total number of edges. Per this formula, the highest possible density of a smiple graph is 1. However, as we pointed out earlier, most real-world graphs are NOT simple-graph, but multi-graph, meaning multiple edges may exist between any two vertices, therefore it makes more sense to use vertex-to-edge ratio (|E|/|V|) as a “density” indicator. Taking Twitter-2010 as an example, its ratio is 35.25 (while density is 0.000000846). For any graph dataset with a ratio higher than 10, deep traversal against the dataset poses as great challenge due to exponentially growing complexity, for instance, 1-hop = 10 neighboring vertices, 2-hop = 100 vertices, 3-hop = 1,000 vertices, and 10-hop = 10,000,000,000.</p><p>Ingestion time and storage usage show that how soon can a graph database load the entire dataset, and how much storage space it uses on filesystem. Clearly, for loading time, the shorter the better. Storage space is less of a concern nowadays as storage is super cheap, different database systems may have quite different storage mechanism in terms of replication, sharding, partitioning, normalization and other techniques that may affect computing efficiency and high-availability. The table below shows the data ingestion performance of different graph DBMSes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tjaUf1eE3dQetAMbzUat6A.png" /><figcaption>Twitter Data Set Ingestion Time Comparison (Table)</figcaption></figure><p>Note that Tigergraph stores each edge unidirectionally instead of bidirectionally, this alone saves 50% storage space, but it’s incorrect. If Tigergraph were to store properly, it would occupy at least 24GB. If a system only stores an edge unidirectionally, it would generate incorrect results for queries like K-hop, Path or algorithm, and we’ll get to that later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pmqSJ0hNYCaMInNnOhJ_9g.png" /><figcaption>Twitter Data Set Ingestion Time Comparison (Graph)</figcaption></figure><h3>Query Performance</h3><p>There are mainly 3 types of graph queries:</p><p>1. Query for Neighbors：also known as K-hop</p><p>2. Path: Shortest path queries are commonly used for simplicity (ease-of-comparison)</p><p>3. Algorithm: Commonly benchmarked algorithms include PageRank, Connected Component, LPA, and Similarity. Some performance-oriented graph databases also benchmark Louvain community detection algorithm which is considered more complex and challenging</p><p>K-Hop measures query latency at different depth. Taking Twitter-2010 as an example, it makes sense to measure the performance of 1-Hop and all the way to 6-Hop. Most databases can handle 1-hop without problem, but 2-hop and beyond are much more challenging, very few systems can return beyond 3-hop. If a system can finish 6-hop (considered deep traversal) within 10 seconds, it’s considered a real-time graph database, because 6-hop traversal of most vertices of Twitter dataset would basically touch base with 95% of all 1.5 billion edges and vertices, which translates to the computing power of traversing over 150 million nodes and edges every second.</p><p>For any given dataset and any designated starting vertex, K-hop query result is deterministic, therefore all benchmarking systems should return the exact result. Common mistakes with K-hop are:</p><p>· DFS instead of DFS for traversal (wrongful implementation)</p><p>· No deduplication (redundant and incorrect results)</p><p>· Partial traversal (violation of the definition of K-hop)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*P5uvU-DKVCrRpwNx" /></figure><p>Note, Ultipa Graph is the only system that has published its ultra-deep traversal results, vertices having neighbors that are 23-hop away, which reflects the system’s capability to deeply penetrate data in real-time while other systems may take hours or even days to complete, or never finish.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t8oGtKWxwxOluaTL" /></figure><p>Shortest-Path query is considered a special form of K-hop query, the difference lies at fixing both the starting vertex and ending vertex and find all possible paths in between. Note that in dense dataset like Twitter-2010, there are possibly millions of shortest paths between a pair of vertices. Only 3 graph database vendors have published their benchmark results against Twitter, but Ultipa is the only vendor that has published the exact number of paths as shown in the below table. Some vendors return with only 1 shortest path, which is both ridiculously wrong and problematic. In many real-world scenarios, such as anti-money laundering paths, transactional networks, it’s imperative to locate all shortest paths between two parties as soon as possible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*X2QfEo1BrIgjJudp" /></figure><p>Graph Algorithm reflects a graph DMBS’ computing power in terms of mass data processing. The number of supported graph algorithms and running speed directly tell a system’s capability and feature completeness. Many graph DBMSes are based on some graph computing engine or framework, which can only handle static dataset, once the dataset is modified, a new projection (loading data into DRAM, comparable to ETL) has to be constructed before the graph algorithms can be relaunched. This is the case for Apache Spark GraphX and Neo4j GDS, the ETL process may take a rather long time, which should be included in the algorithm running time.</p><p>PageRank is the most famous graph algorithm, takes its name from Google co-founder Larry Page. It’s the web search engine’s core ranking algorithm, and one of the 1st algorithm that has been crafted to run parallelly on large-scale distributed systems due to its simple iteration logics and BSP-friendly design. The algorithm running results conveniently tell the ranking (importance) of each vertex (entity) is comparing with the rest of the dataset. For commercial grade implementation, there are also features like database write-back, filesystem write-back, streaming write-back, results ranking, etc.</p><p>The 2 essential features of PageRank to watch out for:</p><p>· PageRank must run against the entire dataset, not partially</p><p>· Results should be fully ranked, and return Top-N as needed</p><p>We noticed that some graph DBMS only calculates against a partial dataset, this is a direct violation of the definition of PageRank which is to iterate through entire dataset. Neo4j GDS is a typical example that allows the algorithm to run against, say, 1,000 vertices if the input parameter provided has a limit of 1,000. Taking Twitter-2010 as an example, the result would be 100% wrong, because running against 1,000 vertices is only 0.000025% of the entire dataset. On the other hand, if the results are not ranked, it’s up to the business layer (application layer) to do the ranking which clearly isn’t acceptable. We found that most graph DBMS benchmarking results do not touch base on these issues.</p><p>The table below lists the benchmarking logics of PageRank, and list of top-10 returning vertices:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GJzTnAu4LLp-Ycqk" /></figure><p>LPA (Label Propagation Algorithm) is a classic community-detection algorithm, it propagates predefined labels across all vertices iteratively until a stable state is achieved. Comparing to Pagerank, LPA is computationally more complex, and returns total number of communities per vertex-label aggregation. There are also various write-back options to look at when invoking LPA.</p><p>Louvain community detection algorithm is computationally more complex than LPA, and iterate through all vertices repeatedly to calculate maximum modularity of the entire dataset. Very few graph DBMS vendors prefer to benchmark Louvain on dataset larger than 100 million nodes and edges. The original Louvain algorithm was implemented using C++ yet in a sequential fashion, on a typical billion-scale dataset, it would require 2.5 hours to finish, if this is to be done in Python, it would last at least a few hundred hours (weeks). Ultipa is the only vendor that has published Louvain’s benchmark results on dataset sized over 1 billion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ns5K4MD8vCi68k4G" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*olQqtXiNeinLK9Zn" /></figure><h3>Real-time Updates</h3><p>There are 2 types of update operations in graph DBMS:</p><p>· Meta-data oriented (TP): The operation only affects meta-data like vertex, edge, or their attribute field(s). From traditional DBMS perspective, this is a typical TP-operation (ACID). In graph DBMS, this also means the topology of the graph changes, such as inserting or deleting a node, an edge, or changing an attribute that may affect query results.</p><p>· TP+AP operation: This refers to combo operations, where first a meta-data operation is conducted then an AP type of query is launched. This is common with many real-time BI or decision-making related scenarios, such as online fraud detection, where a new transaction or loan-application shows up, and a series of graph-based network and behavior analysis follow the suite.</p><p>It is not uncommon that some graph systems use caching mechanism to pre-calculate and store-aside results which do NOT change accordingly after the data contents are change. To validate if a graph database system has the ability to update data contents and allow for accurate data querying in real-time, we create a new edge connecting the provided starting vertex with one of its 3rd-hop neighboring vertex, and compare the 1-to-3-to-6 Hop results before and after the edge is inserted.</p><p>The table below shows how the K-hop results change before and after the edge-insertion event. If a system does NOT have the capacity to handle dynamically changing graph data, the K-hop results will stay put (and wrong).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JKTNmQmmxfGPJkcw" /></figure><p>In the 2nd part of this article, we’ll introduce mechanism to validate the benchmark results, remember that graph DBMS is white-box and explainable AI, we’ll show you how everything can be explained in a white-box fashion in <a href="https://medium.com/@RickySun/how-to-read-graph-database-benchmark-part-ii-38ac7da38768">part 2</a>. Stay tuned.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=996daa830a94" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ultipa/how-to-read-graph-database-benchmarks-part-i-996daa830a94">How to Read Graph Database Benchmarks (Part-I)</a> was originally published in <a href="https://medium.com/ultipa">Ultipa Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>