<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by ArangoDB on Medium]]></title>
        <description><![CDATA[Stories by ArangoDB on Medium]]></description>
        <link>https://medium.com/@arangodb?source=rss-d2de8c290e2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*xOWmlMuqPDDFzV2Q.jpg</url>
            <title>Stories by ArangoDB on Medium</title>
            <link>https://medium.com/@arangodb?source=rss-d2de8c290e2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 11:09:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@arangodb/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Three Ways to Scale Your Graph]]></title>
            <link>https://medium.com/arangodb/three-ways-to-scale-your-graph-be5139483aaf?source=rss-d2de8c290e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/be5139483aaf</guid>
            <category><![CDATA[arangodb]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[graph-database]]></category>
            <dc:creator><![CDATA[ArangoDB]]></dc:creator>
            <pubDate>Fri, 09 Jun 2023 08:10:13 GMT</pubDate>
            <atom:updated>2023-06-09T08:10:13.931Z</atom:updated>
            <content:encoded><![CDATA[<p>As businesses grow and their data needs increase, they often face the challenge of scaling their database systems to keep up with the increasing demand.</p><p>What happens when your single server machine is no longer sufficient to store your graph that has grown too large? Or when your instance can no longer cope with the increasing amount of user requests coming in?</p><p><strong>Sharding is the solution.</strong></p><p><em>While you can vertically scale your resources on your single server machine by adding more resources, sharding allows you to horizontally scale your dataset throughout multiple machines.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PMZvYVOCvE2nav7K.png" /></figure><h3>What is Sharding?</h3><p>Sharding is a technique used in database management systems to horizontally scale and partition data across multiple servers or nodes. The idea behind sharding is to divide a large dataset into smaller subsets called shards and store each shard on a different server. By doing this, the system can process and manage the data more efficiently and effectively.</p><p>Sharding is particularly useful when dealing with large datasets that cannot be stored on a single server due to hardware limitations or performance constraints. By dividing the data into smaller chunks and distributing them across multiple servers, sharding can help improve query performance, reduce response times, and enable systems to scale to handle large data volumes.</p><h3>Advantages of Sharding in ArangoDB</h3><p>The sharding aspect in ArangoDB offers several advantages, including:</p><ul><li><strong>Improved Performance</strong>: By distributing data across multiple servers, ArangoDB can process queries faster, reduce response times and improve overall performance.</li><li><strong>Scalability</strong>: Sharding allows ArangoDB to scale horizontally by adding more servers to the cluster. This enables the system to handle larger datasets without compromising performance.</li><li><strong>Fault Tolerance</strong>: ArangoDB’s sharding architecture provides fault tolerance by ensuring that data is replicated across multiple servers. This means that if one server fails, the system can continue to function without data loss.</li></ul><p>Sharding in ArangoDB is based on shard keys. To determine which shard a document should be stored in, ArangoDB uses a hash-based sharding mechanism. When a document is inserted into a collection, ArangoDB hashes the value of the document’s shard key. The shard key is a user-defined field that specifies how the data should be partitioned. By default this is the _key attribute of a document.</p><h3>Graph Basics</h3><p>A graph in ArangoDB consists of a <em>Graph Definition</em>. In short, the <em>Graph Definition</em> itself gets a descriptive name and a list of participating collections. Collections, which are the containers for the documents themselves, are either of the type called <em>Document Collection</em> or <em>Edge Collection</em>. The data nodes are being stored in the <em>Document Collection</em>, the relations connecting those nodes are called <em>edges</em> and are being stored in <em>Edge Collections</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/353/0*AC3XmRi3lX_d8lAq.png" /></figure><p>An edge is a relation that connects two nodes with each other. In ArangoDB an edge is always persisted with a fixed direction, but can be accessed in <a href="https://www.arangodb.com/docs/stable/aql/graphs-traversals.html#traversing-in-mixed-directions">any direction</a> within graph queries. To get all details and option parameters about <em>Graphs, Vertices, Edges</em> and <em>Graph Definitions</em> in ArangoDB itself, <a href="https://www.arangodb.com/docs/stable/graphs.html">please read our full documentation</a>. In case you want to learn more about graphs in general, please refer to <a href="https://www.arangodb.com/2022/07/what-is-a-graph-database/">this</a> blog article.</p><h3>Example Graph Concept</h3><p>To help you understand the major differences between single server instances and clustered environments, we’ll introduce a simple example graph concept we’ll make use of throughout this blog post.</p><p>Our example graph consists only of a limited number of nodes to keep it simple. Those nodes are connected via edges. On a single server instance all of that data would be stored on a single machine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XjvQosyiKVjbssbp.png" /></figure><h3>Challenges with Distributed Graphs</h3><p>On a single server instance, all data we have is stored locally. For any graph search, whether it is a Traversal (DFS, BFS or Weighted) or a Shortest Path (Weighted, K-Paths or All-Paths) calculation, we achieve the best performance as we do not need to perform any additional network requests or manage collections being split into shards onto different database servers. This statement is true until the machine’s capacity is reached.</p><p>Therefore we need to organize our graph in such a way that we can scale out to multiple machines. At the same time, we must organize our data in a smart way so that we can run as many computations on the database servers as possible (e.g. to allow parallel computations). This optimization will then lead to a reduced amount of network communication, as network requests are usually one of the main reasons to make the performance worse.</p><p>The better the graph data itself is organized, the better the query performance will be. Better data organization means, we try to achieve the best data locality we can get as this will drastically reduce the total amount of network requests we need to do, while at the same time we can fan out to multiple machines and continue our graph algorithm there in parallel.</p><p>In the upcoming sections we will cover the different types of graph concepts in ArangoDB with a special look at their distribution in a clustered environment and their sharding characteristics.</p><h3>Sharded Graphs Concepts</h3><p>ArangoDB offers multiple graph types to create and handle graphs in a clustered environment with different sharding strategies. All types are based on the same manner using <em>Graph Definitions</em>.</p><p>The Community Edition of ArangoDB includes:</p><ul><li>General Graph</li></ul><p>The Enterprise Edition of ArangoDB adds two major graph types on top:</p><ul><li>EnterpriseGraphs</li><li>SmartGraphs</li></ul><p><em>(ArangoDB offers even more specialized graph types we’ll dig into in an upcoming blog article.)</em></p><p>All of them support various configuration parameters during creation, e.g.:</p><p>The amount of Shards used per Collection, which will then automatically be applied to any Document or Edge Collection which is part of the Graph Definition. It describes the number of logical partitions (Shards) that a collection will be divided into.</p><p>The Replication Factor, which specifies the number of replicas that should be created for each shard in a database cluster.</p><p>The Write Concern, which specifies the minimum number of database servers or replicas that must acknowledge a write operation before the operation is considered successful.</p><h4>General Graph</h4><p>The General Graph is the basic graph type in ArangoDB that allows you to create and manage graphs. It is suitable for small-scale graph use cases and does not require any specific configuration or setup. In a General Graph, the data will be randomly distributed across all configured machines and each machine will take an equal portion of data. It is very easy to realize as no knowledge about the data is required. This graph type will always work, but there will be a few disadvantages with this approach. As we distribute randomly, Neighbors will land on different machines. Therefore, edges will also with high probability end up on other servers as their connected nodes. The worst case which can occur is that in a relation (Bob) ⇒ (Alice), Bob will land on database server A, the relation itself will land on database server B and Alice will land on database server C, which then leads to a lot of required network requests. This is then finally reflected in a lack of query execution performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iCYWov4AwqlTqiD_.png" /></figure><h4>EnterpriseGraph</h4><p>The EnterpriseGraph is a more advanced type of graph in ArangoDB that is designed to support large-scale graph use cases in enterprise environments. EnterpriseGraphs will allow you to create graphs at scale with automated sharding key selection.</p><p>While the data itself is also “randomly sharded” (like in General Graphs), this specific graph type ensures that all edges adjacent to a vertex are co-located on the same server. This approach provides significant advantages as it minimizes the impact of having suboptimal sharding keys defined when creating the graph. It will give a vast performance benefit for all graphs sharded in an ArangoDB Cluster, reducing network hops substantially as more graph calculations can be done on the database nodes itself.</p><p>The only consequence here is, that you cannot define custom _key values on edges, as ArangoDB calculates its value based on the shard key — and this specific calculation is being done by the EnterpriseGraph graph type automatically. EnterpriseGraphs do store some additional meta information which leads to a slightly increased amount of disk usage compared to GeneralGraphs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kwZfQMUKfOQochT9.png" /></figure><h4>SmartGraph</h4><p>While EnterpriseGraphs are already improving the data distribution across multiple database servers, SmartGraphs are optimizing data distribution even further. Graphs in general know nothing of themselves. But, your application knows a lot about the graph. In many datasets there are highly interconnected communities, but few connections between these communities. For instance, a set covering your customers, regions or any other logic you apply to organize your graph at the application layer can in turn be used in sharding the graph through the cluster.</p><p>Think about a social network. In a social network, which connects people through relations, it is more likely that a person has more regional friends, relatives or followers. From this knowledge, we can choose the ideal property to define the graphs sharding. This property is called the smartGraphAttribute and needs to be defined for a SmartGraph in ArangoDB. This results in highly-connected communities landing on the same database server, which will improve query execution times compared to EnterpriseGraphs even more. A simple performance comparison between a General Graph and a SmartGraph example can be found <a href="https://www.arangodb.com/learn/graphs/using-smartgraphs-arangodb/">here</a>.</p><p>The storage size of SmartGraphs is comparable to the storage size of EnterpriseGraphs. At most it will be equal to the size of an EnterpriseGraph. At best it is close to the resource consumption of a General Graph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/0*gRWO1TJihBzcwMb-.png" /></figure><h3>Conclusion</h3><p>All the graph types ArangoDB offers have several advantages and disadvantages. All of them share the ability to scale your graph to multiple machines in a clustered environment. You can always start exploring using the <strong>General Graph</strong>.</p><p>If the performance does not fit your needs — try to use <strong>EnterpriseGraph</strong>, as you don’t need to manually take care of the graph data distribution. You should directly see better query execution times.</p><p>In case your requirements are still not met yet, you should start to organize your graph dataset in a smarter way. While you should always think about an ideal way of modeling your graph, the next graph type enforces you to define a property. Think about a good way to structure your graph into multiple assets with the primary goal to use <strong>SmartGraphs</strong> — which then will drastically improve the performance as then we can organize the data throughout all database servers in the best way possible.</p><p>Therefore, think carefully which graph type fits best to the schema of your graph and its dataset. Ideally, you choose the best fitting graph type for your use-case in advance, as converting to another graph type might cause extra effort (find details about graph migration <a href="https://www.arangodb.com/docs/stable/graphs-enterprise-graphs-getting-started.html#migrating-from-community-edition">here</a>).</p><h3>Next</h3><p>Want to even learn more about advanced graph sharding strategies? Stay connected. A more advanced blog post will follow soon about OneShard Graphs, Disjoint SmartGraphs, Hybrid SmartGraphs and Hybrid EnterpriseGraphs and SatelliteGraphs, which allow you to fine-tune graph sharding to the next level.</p><h3>Call to Action</h3><p>Thank you for reading!</p><p>Feel free to connect and ask us any questions @ <a href="https://www.arangodb.com/community/">https://www.arangodb.com/community/</a>.</p><p>Also don’t forget to check our free-trial on our cloud offering, ArangoGraph Insights Platform: <a href="https://www.arangodb.com/download/#try-cloud">https://www.arangodb.com/download/#try-cloud</a></p><p><em>Check out this and other exciting articles on </em><a href="https://www.arangodb.com/blog/"><em>ArangoDB websites.</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be5139483aaf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/arangodb/three-ways-to-scale-your-graph-be5139483aaf">Three Ways to Scale Your Graph</a> was originally published in <a href="https://medium.com/arangodb">ArangoDB</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Graph and Entity Resolution Against Cyber Fraud]]></title>
            <link>https://medium.com/arangodb/graph-and-entity-resolution-against-cyber-fraud-ecaa53bc11c4?source=rss-d2de8c290e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/ecaa53bc11c4</guid>
            <dc:creator><![CDATA[ArangoDB]]></dc:creator>
            <pubDate>Mon, 03 Apr 2023 14:38:51 GMT</pubDate>
            <atom:updated>2023-04-14T12:56:47.731Z</atom:updated>
            <content:encoded><![CDATA[<p>With the growing prevalence of the internet in our daily lives, the risks of malware, ransomware, and other cyber fraud are rising. The digital nature of these attacks makes it easy for fraudsters to scale by creating thousands of accounts, so even if one is identified, they can continue their attacks.<br>In this blog post, we will discuss how graph and entity resolution (ER) can help us battle these risks across different industries such as healthcare, finance, and e-commerce (for example, the US healthcare system alone can <a href="https://towardsdatascience.com/an-introduction-to-entity-resolution-needs-and-challenges-97fba052dde5">save $300 billion a year</a> with entity resolution). You will also receive hands-on experience with entity resolution on ArangoDB.</p><p><strong>What is Entity Resolution?</strong></p><p>Entity resolution, also called record linkage or deduplication, is a technique used to identify and merge similar or identical entities from multiple data sources into a single record. Imagine, for example, a fraudster creating many thousands of accounts across different services. Entity resolution can help match and resolve these virtual records into one entity or record.</p><p>Entity resolution can be used for various purposes, such as identifying duplicate customer records in a marketing database, matching medical records to patients, detecting fraudulent activities by identifying multiple identities of the same person or linking social media profiles to a single individual. Entity resolution can improve the accuracy and completeness of data by reducing redundancy, eliminating errors, and creating a unified view of the data. It can also facilitate data integration and analysis and support various applications such as recommendation systems, personalized marketing, and fraud detection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*VIiYTClcROF3IaZf" /></figure><p>User accounts: Merging multiple user entries into one entity.</p><p><strong>Graph to the Rescue</strong></p><p>Entity resolution is a challenge as there are typically no distinct keys identifying each entity.</p><p>Let us revisit the above example of user accounts and experience one of my daily frustrations with the German umlaut in my first name. Depending on the document or account, it is either spelled Joerg or Jörg (let us not get even get started on pronunciation). Matching different accounts is a challenge because the name is not a unique identifier. This is often the case even without weird letters; just imagine <em>J. Smith</em> vs <em>Joerg S</em>.</p><p>Luckily, in most real world scenarios we have additional contextual information, such as emails, phone numbers, addresses, and employers. Here is where graph comes into play, as this information can be easily assembled as a graph. Note that usually the context won’t 100% match, e.g., I have multiple email addresses and phone numbers, moved around quite a bit, and worked for different companies, so vary rarely will all entities have all potential information categories (e.g., the account <strong>Joerg S</strong>. on the right side does not have connected employer information). Still, we can treat each individual piece as evidence and then compute a similarity score.</p><p>In practice, this can be done by representing a user&#39;s neighborhood (i.e., a node in the graph) with similarity measures such as cosine similarity or Jaccard distance. Depending on the data, all nodes above a certain threshold are considered the same entity. Feel free to try yourself using this <a href="https://colab.research.google.com/github/arangodb/interactive_tutorials/blob/master/notebooks/EntityResolution.ipynb">Jupyter notebook</a>.</p><p>There are also other techniques, including <a href="https://www.arangodb.com/graph-analytics-at-enterprise-scale/">graph machine learning</a>, but we will cover that in more detail in another blog post.</p><p><strong>How to Battle Cyber Fraud with ER</strong></p><p>Entity resolution is used as a tool for cyber security by identifying and linking together various pieces of related information that can be used to catch fraudulent activity. This can include IP addresses, emails, or other means of device authentication. By analyzing these and other similar data points, entity resolution can help investigators detect the true identity of the fraudster and track their activities across multiple accounts or platforms.</p><p>There are several ways entity resolution can be used for cyber security. A few of these use cases include fraud detection, risk assessment, and identity verification. Some examples of entity resolution being put into play are identifying indicators of fraud, analyzing the risk associated with a particular transaction or user account, and verifying the identity of users through different identification methods.</p><p>Imagine a social media platform is trying to remove duplicate or fraudulent accounts. While there might be multiple accounts that go under the name <em>Jane Foster</em>, there would be a limited number of those accounts that would share a similar birth date. If you continue to narrow down your results, there may be 3 accounts under the name <em>Jane Foster</em> that use the same IP address and mobile device to connect to the duplicate accounts. This can also be used to identify spam accounts by associating devices and IP networks.</p><p><strong>Learn More About Entity Resolution with ArangoDB</strong></p><p>In this blog, we have discussed how graph and entity resolution can help with both cyber fraud and names with German umlauts! If you want to learn more about these topics take a look at the following additional resources.</p><p>Lunch + Learn Session:</p><p><a href="https://www.arangodb.com/resources/lunch-sessions/graph-beyond-lunch-break-15-entity-resolution/">https://www.arangodb.com/resources/lunch-sessions/graph-beyond-lunch-break-15-entity-resolution/</a></p><p>Entity Resolution in ArangoDB Blog Post:</p><p><a href="https://www.arangodb.com/2021/07/entity-resolution-in-arangodb/">https://www.arangodb.com/2021/07/entity-resolution-in-arangodb/</a></p><p>Fraud Detection with ArangoDB Webinar:</p><p><a href="https://hopin.com/events/fraud-detection-with-arangodb">https://hopin.com/events/fraud-detection-with-arangodb</a></p><p>Jupyter Notebook:</p><p><a href="https://colab.research.google.com/github/arangodb/interactive_tutorials/blob/master/notebooks/EntityResolution.ipynb">https://colab.research.google.com/github/arangodb/interactive_tutorials/blob/master/notebooks/EntityResolution.ipynb</a></p><p><em>Originally published at </em><a href="https://www.arangodb.com/2023/04/graph-and-entity-resolution-against-cyber-fraud/"><em>https://www.arangodb.com</em></a><em> on April 3, 2023.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ecaa53bc11c4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/arangodb/graph-and-entity-resolution-against-cyber-fraud-ecaa53bc11c4">Graph and Entity Resolution Against Cyber Fraud</a> was originally published in <a href="https://medium.com/arangodb">ArangoDB</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Combat Fraud with Graph]]></title>
            <link>https://medium.com/arangodb/combat-fraud-with-graph-cb947c6eb5e0?source=rss-d2de8c290e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb947c6eb5e0</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[graph-database]]></category>
            <category><![CDATA[arangodb]]></category>
            <category><![CDATA[fraud-detection]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[ArangoDB]]></dc:creator>
            <pubDate>Wed, 15 Mar 2023 12:00:04 GMT</pubDate>
            <atom:updated>2023-04-14T12:27:04.936Z</atom:updated>
            <content:encoded><![CDATA[<p>Fraud is one of the most significant issues facing businesses today. While companies have always faced fraud, detecting fraudulent activity has become even more challenging due to increased online transactions. Globally, fraud results in more than $3.7 trillion in annual losses ( <a href="https://news.cardnotpresent.com/news/each-1-of-fraud-loss-actually-costs-merchants-3.75-says-report">Murphy, 2022)</a>. Fraud comes in numerous forms, including but not limited to money laundering, identity theft, account takeover, and payment fraud. Due to the variety of ways companies can face fraud, they must have a system to protect themselves and their customers.</p><p>One technology that is increasingly being used to detect and prevent fraud is graph databases. This blog post will explain graph databases and how they can help with fraud detection.</p><p><strong>What are Graph Databases?</strong></p><p>A graph is a collection of nodes (points) and edges (lines) where the edges describe the relationship between the nodes. Graphs can be explored through graph theory, analytics, and database models. This database form is considered the next step for data and analytics to get the most out of their delivery. Graph databases give a way to organize and present data for use cases previously considered difficult or complicated to address appropriately.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/0*rIClnyB0nJaQCGk3" /></figure><p>A graph database stores the data and its natural relationships as a graph of nodes and edges instead of disconnected rows and columns in a table that you would see in a traditional relational database.</p><p>Graph databases have built-in algorithms for standard graph analytics functions such as <a href="https://www.arangodb.com/docs/stable/aql/graphs-shortest-path.html">Shortest path</a>, <a href="https://www.arangodb.com/docs/stable/aql/graphs-kshortest-paths.html">k shortest paths</a>, <a href="https://www.arangodb.com/docs/stable/aql/graphs-k-paths.html">k paths</a>, and <a href="https://www.arangodb.com/docs/stable/aql/graphs-all-shortest-paths.html">all shortest paths</a>.</p><p><strong>Graph Database vs. Fraud Detection</strong></p><p>Graph databases are ideal for fraud detection because they can quickly and efficiently identify patterns and connections between seemingly unrelated entities. Here are a few ways in which graph databases can help with fraud detection:</p><p><strong>Relationship Analysis:</strong> Fraudulent activity often involves multiple entities, such as a fraudster, a victim, and a middleman. Graph databases can identify behavior patterns indicative of fraud by analyzing the relationships between these entities. For example, if a fraudster uses multiple email addresses to create fake accounts, a graph database can identify them and link them to the same person.</p><p><strong>Real-time Detection: </strong>Graph databases can process large amounts of data in real time, making them ideal for detecting fraud as it happens. By continuously analyzing data streams, graph databases can detect behavior patterns indicative of fraud and trigger alerts or block transactions.</p><p><strong>Machine Learning: </strong>Graph databases can be integrated with machine learning algorithms to improve fraud detection. By training machine learning models on historical data, graph databases can identify behavior indicative of fraud and use those patterns to predict future fraud.</p><p>Learn more about Machine Learning with ArangoDB and ArangoGraphML <a href="https://www.arangodb.com/arangodb-for-machine-learning/">here</a>.</p><p><strong>Centralized Data Management: </strong>Graph databases can provide a single source of truth for fraud detection data. Companies can easily track and analyze fraud patterns across different systems and departments by collecting and storing all fraud detection data in a graph database.</p><p><strong>Entity Resolution for Fraud Detection</strong></p><p>Entity Resolution is a critical tool in combating fraud. Entity resolution finds duplicates of entities across multiple systems on-prem and in the cloud. This allows admins to have a clear view of data and to sort through different data types such as date, contact, email, address, email, device, or any additional unique identifier.</p><p>When applied to fraud detection, entity resolution allows admins to find duplicates of fraudsters within their system by cross-referencing these unique identifiers. This will enable fraudsters running different scams through similar networks to be flagged and taken down.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*UQ0ZRDFFRwhxSumk" /></figure><p><strong>ArangoDB as a Graph Database</strong></p><p>ArangoDB goes beyond graph by being a graph store that natively incorporates capabilities from other data models, including key-value, document, search, and more. The graph capabilities of ArangoDB are similar to a property graph database but add more flexibility in data modeling as vertices and edges are both full JSON documents.</p><p>Due to this natively integrated support, users can take the result of a JOIN operation, geospatial query, text search, or any other access pattern as a starting point for further graph analysis and vice versa — all in one query, if needed.</p><p>ArangoDB is the underlying database for <a href="https://www.arangodb.com/arangograph-enterprise/">ArangoGraph Insights Platform</a>, a cloud-based graph data, and analytics platform.</p><p><strong>Fraud Detection with ArangoDB</strong></p><p>Today’s criminals are developing new techniques to hide their activities by forming fraud networks with stolen or synthetic identities. Attacks are often launched from multiple vectors and can only be discovered by connecting diverse data sources to uncover difficult-to-detect patterns. Native graph technology is perfect for solving this challenge.</p><p>A graph database is a suitable solution as it decreases the time needed to process fraud detection queries against the database and delivers simple data visualizations to analysts. It also removes false positives as real customers wait for the money (customer satisfaction and lost revenue). Fraud detection is a great use case for a graph database, as relational databases are too slow and complex to query in real-time.</p><p>Try out our Fraud Detection guide on ArangoGraph Insights Platform → <a href="https://cloud.arangodb.com/home?utm_campaign=2023%20Fraud%20Campaigns&amp;utm_source=fraud%20detection%20blog">https://cloud.arangodb.com/home?utm_campaign=2023%20Fraud%20Campaigns&amp;utm_source=fraud%20detection%20blog</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YacRvXYMH75mG9Wd" /></figure><p><strong>Conclusion</strong></p><p>Fraud detection is a critical aspect of modern business, and graph databases can help companies detect and prevent fraud by identifying patterns and connections in complex and interconnected data. By leveraging the power of graph databases, companies can improve their fraud detection systems and better protect themselves and their customers.</p><p><strong>Additional Resources</strong></p><p>Read “Identifying Fraud at Scale with ArangoDB” <strong>→ </strong><a href="https://www.arangodb.com/resources/white-paper/fraud-detection/?utm_campaign=2023%20Fraud%20Campaigns&amp;utm_source=fraud%20detection%20blog">https://www.arangodb.com/resources/white-paper/fraud-detection/?utm_campaign=2023%20Fraud%20Campaigns&amp;utm_source=fraud%20detection%20blog</a></p><p>Did you miss our “Fraud Detection with ArangoDB” webinar? Watch it on demand today! → <a href="https://hopin.com/events/fraud-detection-with-arangodb?utm_source=Blog%20Post&amp;utm_campaign=Fraud%20Detection%20Campaign">https://hopin.com/events/fraud-detection-with-arangodb?utm_source=Blog%20Post&amp;utm_campaign=Fraud%20Detection%20Campaign</a></p><h4>About The Author:</h4><p><a href="https://www.arangodb.com/author/laura/">Laura Cope</a></p><p>Laura completed her MA in North American Studies at the University of Cologne in Germany. She has always been passionate about social media and the role it plays in how we communicate with each other.</p><p><em>Originally published at </em><a href="https://www.arangodb.com/2023/03/combat-fraud-with-graph/"><em>https://www.arangodb.com</em></a><em> on March 15, 2023.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb947c6eb5e0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/arangodb/combat-fraud-with-graph-cb947c6eb5e0">Combat Fraud with Graph</a> was originally published in <a href="https://medium.com/arangodb">ArangoDB</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is a Graph Database]]></title>
            <link>https://medium.com/arangodb/what-is-a-graph-database-c92e38b8dfbf?source=rss-d2de8c290e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/c92e38b8dfbf</guid>
            <category><![CDATA[graph-theory]]></category>
            <category><![CDATA[graph-database]]></category>
            <category><![CDATA[arangodb]]></category>
            <category><![CDATA[graph]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[ArangoDB]]></dc:creator>
            <pubDate>Fri, 05 Aug 2022 10:08:53 GMT</pubDate>
            <atom:updated>2022-08-05T10:08:53.243Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Graphs occur everywhere in everyday life: your network of friends, the network of roads you drive on, and the supply chain of factories, ships, and roads that brought you the device you’re reading this on. While it might be easy to connect the dots on how most things can be shown as a graph, what makes a database a graph database? That is the question you will have the answer to in this blog post, but to put it simply: a graph consists of nodes, edges, and properties representing the relationships within data.</p><p>In this article, we will discuss:</p><ul><li>What is a graph?</li><li>What is a graph database?</li><li>Different types of graph databases.</li><li>Graph database use cases.</li></ul><h3>What is a Graph?</h3><p>A graph is a collection of nodes and edges where the edges describe the relationship between the nodes. Graphs exist across multiple domains including graph theory, analytics, and database models. These three separate entities support each other and allow for connection through the specific abilities of each.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J-JbDNplqOVuBdzt6h3W1g.png" /><figcaption>Graph Database Example</figcaption></figure><p>According to <a href="https://en.wikipedia.org/wiki/Graph_theory#:~:text=In%20mathematics%2C%20graph%20theory%20is,also%20called%20links%20or%20lines).">Wikipedia</a>, graph theory is:</p><p><em>In mathematics, graph theory is the study of graphs, which are mathematical structures used to model pairwise relations between objects. A graph in this context is made up of vertices (also called nodes or points) which are connected by edges (also called links or lines).</em></p><p>In computing, it is considered an abstract data type that is good at representing connections or relations — unlike the tabular data structures of relational database systems, which are very limited in expressing concerns.</p><p>Graph analytics is not a new tool but is historically underutilized in data and analytics. Graph analytics is the process of analyzing data stored within a graph database. Data Scientists and Engineers can use a graph database to process nodes and edges to understand the relationship between the data collected. We cover some examples of graph analytics in the use cases section.</p><figure><img alt="Example of Nodes and Edges" src="https://cdn-images-1.medium.com/max/348/0*gThpN7k_9Bh7P6tq" /><figcaption>Example of Nodes and Edges</figcaption></figure><p>A good metaphor for graphs is to think of nodes as circles and edges as lines or arcs. The terms node and vertex are used interchangeably here. Usually, vertices are connected by edges, making up a graph. Vertices don’t have to be connected at all, but they may also be connected with more than one other vertex via multiple edges. You may also find vertices connected to themselves, as shown above.</p><h3>Graph Databases</h3><p>This database form is considered the next step for data and analytics to get the most out of their delivery. Graph databases give a way to organize and present data for use cases previously considered difficult to address appropriately.</p><p>A graph database lets users analyze large data sets that previously could have been associated with complex use cases. A graph database allows data to be stored, navigated, and displayed together instead of through separate databases. They are the gateway to empower developers to do graph analytics.</p><p><strong>A graph database stores the data and its natural relationships as a graph of nodes and edges instead of disconnected rows and columns in a table.</strong></p><p>Graph databases have built-in graph algorithms to perform standard <a href="https://www.arangodb.com/docs/stable/graphs-general-graphs-functions.html">graph functions</a> such as <a href="https://www.arangodb.com/docs/stable/aql/graphs-kshortest-paths.html">K Shortest Paths</a>, <a href="https://www.arangodb.com/docs/stable/aql/graphs-shortest-path.html">Shortest Paths</a>, and <a href="https://www.arangodb.com/docs/stable/graphs-general-graphs-functions.html">others</a>.</p><h3>Example of a Graph Database</h3><p>We currently live in an era where it is simpler than ever to connect with friends, family, and peers through digital communication. Social Media and Social Networking is a great example to showcase what a graph database can do.</p><p>Imagine this: You create an Instagram account for the first time. One of the first tasks that Instagram will ask you to do is connect your Facebook account or phone contacts. This way, Instagram can instantly find initial first contacts for you to follow. You can use a graph database through these first connections to find mutual connections through your followers and suggest other users for you to follow. These suggestions can range from celebrities whom many of your connections follow to connections with whom you might only share one mutual link. A graph database will let you view common interests, which can also impact the content that the algorithm will place in your feed. For example, if you are a sports fan constantly interacting with NHL content,you can use a graph databased to pull other content related to the NHL to showcase in your feed.</p><figure><img alt="Social Networking Use Case Example" src="https://cdn-images-1.medium.com/max/960/0*mE20SIyMvEIprzmd" /><figcaption>Social Networking Use Case Example</figcaption></figure><h3>Types of Graph Databases</h3><p>With many different kinds of databases on the market, it is essential to understand what separates the databases from one another. One of the key takeaways from a graph database is that a graph database allows you to store data more visually and interactively. Compared to a relational database, a graph database will enable you to store data on an individual level, while a relational database stores data in predefined tables. With a graph database, you can view the relationship between the data on a more fine-tuned level than in relational databases.</p><p><strong>COMPARING GRAPH NoSQL TO RELATIONAL</strong></p><figure><img alt="Relational Database Example" src="https://cdn-images-1.medium.com/max/862/0*AmRnkispTdbG6bD-" /><figcaption>Relational Database Example</figcaption></figure><figure><img alt="Graph Database Example" src="https://cdn-images-1.medium.com/max/1024/0*0qYFRYR9XwHhtSPX" /><figcaption>Graph Database Example</figcaption></figure><h3>Graph Database</h3><p>There are a few different types of graph models, including property graphs and RDF (Resource Description Framework) graphs, and a few different types are <a href="https://en.wikipedia.org/wiki/Graph_database#Graph_models">defined here</a>. ArangoDB is a property graph meaning the data you store can have an associated human-readable property. ArangoDB goes one step further by allowing you to keep these labels on the nodes and edges of your graph data. This allows for thoughtful data modeling that is intuitive to query and closer to the real-world relationship among data.</p><p>Storing the data in this way allows for an easier understanding of the data points and creates a way to analyze complex data sets. A lot of data naturally fits into a graph, so visualizing it as a graph can offer valuable insights, compared to visualizing data across countless tables or disconnected collections of documents, as you would with other databases. This creates a flexible way of viewing and utilizing your data.</p><p><strong>Flexibility allows graph databases to help solve more complex business needs for multiple industries.</strong></p><p>Graph databases can be used for various use cases, including fraud detection, supply chain networking, and 360 customer view.</p><p>As our understanding of data and technology expands, so does how we store data. Traditionally, there had been different databases, including relational, document, and key-value. Graph databases have become the go-to database solution as data becomes more complex and use cases have a higher demand in complexity.</p><h3>Use Cases for Graph Databases</h3><p>As mentioned earlier on in this post, graph databases can be used for more complex use cases, including but not limited to fraud detection, supply chain networking, and 360 customer view. Other famous use cases for graph databases include social networking, recommendation engine, ai knowledge graphs, and network/operations management.</p><h3>Fraud Detection</h3><p>Fraud Detection is one of the most popular use cases with a graph database. Fraud Detection is monitoring data from customers, accounts, devices, locations, etc., to identify money laundering and mule money accounts. This allows companies to identify fraudulent activity and fraud rings using stolen or synthetic identities. An example of a Graph Database used in a fraud detection use case is a company or any financial institution that wants to prevent illegal activity from complex fraud rings. A graph database is a suitable solution as it decreases the time needed to process fraud detection queries against the database and delivers simple data visualizations to analysts. They also allow for removing false positives as real customers are waiting for the money (customer satisfaction and lost revenue). Fraud Detection is a great use case for a graph database as relational DBs are too slow and complex to query in real-time.</p><h3>Supply Chain Management</h3><p>According to <a href="https://www.cips.org/knowledge/procurement-topics-and-skills/supply-chain-management/"><em>CIPS</em></a>, Supply Chain Management is the flow of goods and services from raw manufacturing to consumption by the consumer. This process requires an organization to have a network of suppliers (that serve as links in the chain) to move the product through each stage.</p><p>Supply chain management allows organizations to:</p><ul><li>Deliver more, quickly</li><li>Ensure products are available</li><li>Reduce quality issues</li><li>Navigate returns easily</li></ul><p>It is ultimately improving value for both the organization and the customers. A real-world application for supply chain management would be to detect the changes in consumer buying patterns, unpredictable costs for materials, warehousing and freight fraud, and counterfeit production of in-demand unstable supplier networks. A graph database can help with these use case examples through graph machine learning, which will help improve recommendations.</p><h3>Recommendation Engine</h3><p>A recommendation engine is a way in which a product or service can recommend similar products/goods based on previous purchases by customers and the similarities between products. Recommendation engines are helpful to companies as they gain insight into customer shopping habits, allowing them to create a unique customer experience for each of their users, in turn having a higher retention rate.</p><p>A solid recommendation engine comes with a few challenges, but the three main challenges are:</p><ul><li>Speed</li><li>Relevance</li><li>Computer-heavy queries</li></ul><p>A graph database can help solve all three of these problems. Regarding speed, a graph database can quickly surface all relationships between the data because those are stored as vertices and are how the data is organized. A graph database can also introduce more data points and remove some “fuzziness” from the results. Queries are simplified as a graph database can look at multiple queries compared to a relational database.</p><h3>360 Customer View</h3><p>Another use case for graph databases is 360 Customer View. 360 Customer View is the ability to aggregate and collect data from many sources and collectively store this in one place. 360 Customer View allows a business/organization to better understand their customer interactions. One particular use case ArangoDB solved was when a company needed to integrate many data sources containing semi-related data in different structures.</p><p>For this use case, this company required a fast, seamless data store solution to deal with growing amounts of unstructured data in their BI application. They decided to go with ArangoDB because of AQL (ArangoDB Query Language), which is easy to use and allows you to combine data models in queries. Another reason was our Foxx Microservices and our performance and scalability within a clustered environment.</p><h3>ArangoDB as a Graph Database</h3><p>ArangoDB goes graph and beyond by being a native graph store that natively incorporates capabilities from other data models, including key-value, document, search, and more. The graph capabilities of ArangoDB are similar to a property graph database but add more flexibility in data modeling as vertices and edges are both full JSON documents.</p><p>Due to this natively integrated support, users can take the result of a JOIN operation, geospatial query, text search, or any other access pattern as a starting point for further graph analysis and vice versa — all in one query, if needed.</p><p><em>Interested in learning more? Download our</em><a href="https://www.arangodb.com/resources/white-paper/white-paper-graph-and-beyond/"><em> Graph and Beyond White Paper</em></a></p><p><em>Reference:</em></p><p><em>Supply Chain Management | CIPS. </em><a href="https://www.cips.org/knowledge/procurement-topics-and-skills/supply-chain-management/"><em>https://www.cips.org/knowledge/procurement-topics-and-skills/supply-chain-management/</em></a></p><h3>Author</h3><p><a href="https://www.arangodb.com/author/laura/">Laura Cope</a></p><p>Laura completed her MA in North American Studies at the University of Cologne in Germany. She has always been passionate about social media and the role it plays in how we communicate with each other.</p><figure><img alt="Image- Laura Cope" src="https://cdn-images-1.medium.com/max/324/1*bnhDtfX1bbTM7y0DOas-OQ.jpeg" /><figcaption>Laura Cope</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c92e38b8dfbf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/arangodb/what-is-a-graph-database-c92e38b8dfbf">What is a Graph Database</a> was originally published in <a href="https://medium.com/arangodb">ArangoDB</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ArangoSync: A Recipe for Reliability]]></title>
            <link>https://medium.com/arangodb/arangosync-a-recipe-for-reliability-bf07f9d8128d?source=rss-d2de8c290e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf07f9d8128d</guid>
            <category><![CDATA[graph-database]]></category>
            <category><![CDATA[arangodb]]></category>
            <category><![CDATA[dcdc]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[ArangoDB]]></dc:creator>
            <pubDate>Wed, 06 Jul 2022 08:59:43 GMT</pubDate>
            <atom:updated>2022-07-06T08:59:43.200Z</atom:updated>
            <content:encoded><![CDATA[<h3>A detailed journey into deploying a DC2DC replicated environment</h3><p>When we thought about all the things we wanted to share with our users there were obviously a lot of topics to choose from. Our Enterprise feature; ArangoSync was one of the topics that we have talked about frequently and we have also seen that our customers are keen to implement this in their environments. Mostly because of the secure requirements of having an ArangoDB cluster and all of its data located in multiple locations in case of a severe outage.</p><p>This blog post will help you set up and run an ArangoDB DC2DC environment and will guide you through all the necessary steps. By following the steps described you’ll be sure to end up with a production grade deployment of two ArangoDB clusters communicating with each other with datacenter to datacenter replication.</p><p>All of the best practices that we use during our day-to-day operations regarding encryption and secure authentication have been used while writing this blog post and every step in the setup will be explained in detail, so there will be no need to doubt, research or ponder about which options to use and implement in any situation; Your home lab, your production grade database environment and basically anywhere you want to run a deployment like this.</p><p>A note of importance however is that the ArangoSync feature including the used encryption at rest are Enterprise features that we don’t offer in our Community version of ArangoDB. If you don’t have an available Enterprise license for this project, you can download an evaluation version that has all functionality at: <a href="https://www.arangodb.com/download-arangodb-enterprise/">https://www.arangodb.com/download-arangodb-enterprise/</a></p><p><strong>That’s a lot of words as an introduction but what actually is ArangoSync?</strong></p><p>ArangoSync is our Enterprise feature that enables you to seamlessly and asynchronously replicate the entire structure and content in an ArangoDB cluster in one location to a cluster in another location. Imagine different cloud provider regions or different office locations in your company.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/0*Ln_TExH491VUbVkT" /></figure><p>To run successfully, ArangoSync needs two fully functioning clusters and will not be useful when you’re only running a single instance of ArangoDB. So please also consider this when you’re making any plans to change or implement your database architecture.</p><p>In the above explanation I mentioned that ArangoSync works asynchronously. What this basically means is that when a client processes and writes data into the source datacenter, it will consider the request to be complete and finished before the data has been replicated to the other datacenter. The time needed to completely replicate changes to the other datacenter is typically in the order of seconds, but this can vary significantly depending on load, network &amp; computer capacity, so be mindful of what hardware you choose so it will have a positive and useful impact on your environment in terms of performance and suits your use case.</p><p>ArangoSync performs replication in a single direction only. That means that you can replicate data from cluster A to cluster B or from cluster B to cluster A, but never to and from both at the same time.</p><p>ArangoSync runs a completely autonomous distributed system of synchronisation workers. Once configured properly via this blog post or any related documentation, it is designed to run continuously without manual intervention from anyone.</p><p>This of course doesn’t mean that it doesn’t require any maintenance at all and as with any distributed system some attention is needed to monitor its operation and keep it secure (Think of certificate &amp; password rotation just to name two examples).</p><p>Once configured, ArangoSync will replicate both the structure and data of an entire cluster. This means that there is no need to make additional configuration changes when adding/removing databases or collections. Any data or metadata in the cluster will be automatically replicated.</p><p><strong>When to use it… and when not to use it</strong></p><p>ArangoSync is a good solution in all cases where you want to replicate data from one cluster to another without the requirement that the data is available immediately in the other cluster.</p><p>If you’re still doubting whether ArangoSync is the option for you then review the following list of no’s and if they apply to you or your organization.</p><ul><li>You want to use bidirectional replication data from cluster A to cluster B and vice versa.</li><li>You need synchronous replication between 2 clusters.</li><li>There is no network connection between cluster A and B.</li><li>You want complete control over which database, collection &amp; documents are replicated and which ones will not be.</li></ul><p><strong>Okay I’m done reading the official part, now let’s get started!</strong></p><p>To start off with the first ArangoDB cluster, you will need at least 3 nodes. In this blog post, we’re of course using 6 nodes for both data centers, meaning 3 nodes per datacenter.</p><p>As an example we’re using the hypothetical locations dc1 and dc2<strong> </strong>which can be located anywhere in the world but also in your test environment living in multiple VMs;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CLQLYJmKVfkFXDPvYnEu3A.png" /></figure><p>The three nodes with “dc1” are located in the first datacenter, the three nodes with “dc2” are located in the second datacenter. To test the location can of course be any local environment that supports running six nodes at once with sufficient resources.</p><p>In this blog post, we picked Ubuntu Linux as the OS but as we’re using the .tar.gz distribution with static executables, this means that you can choose whatever Linux distribution your organization runs and that you’re comfortable with. To control the ArangoDB installation, we use systemd, so the distribution should support systemd or you have to change things for automatic restarts after a reboot.<br>Currently, the most recent release of ArangoDB is version 3.8.0, so all of our examples mentioning file names will be using the arangodb3e-linux-3.8.0.tar.gz binary and the following ports need to be open/accessible on each node;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VbNPi4TdrWGHFEQhApyfwg.png" /></figure><p>Obviously, the process name next to the port name is for illustration purposes so you know what port belongs to which process.</p><p>We will roll out the clusters as the root user, but this is of course not necessary. In fact, our own packages create the <em>arangodb</em> user with the installation. It is considered good practice to keep file ownerships separate for services and should be done so in production environments. We could have used any normal user account, provided we have access to the nodes and their filesystem.</p><p>The only significant part where we need root access is to set up the systemd service. Another important prerequisite is that you have properly configured SSH access to all nodes.</p><p><strong>Setting up ArangoDB clusters — A detailed overview</strong></p><p>We will go through all of the detailed next steps. There are a bunch of them to follow so grab a cup of coffee or tea and sit back to work on this. As you might notice, the second half of the steps are repetitive as we’re setting up two similar clusters so we did not make a mistake to make you think you’ve misread. The settings for both clusters slightly differ from each other and therefore we need to separate the installation steps. All commands you need to follow are explained and written out in detail and can even be copied and pasted for your own future reference when you’d like to automate the installation steps in your own environment.</p><p><strong><em>Extract the downloaded binary in its target location:</em></strong><br>Assuming the archive file arangodb3e-linux-3.8.0.tar.gz is present on your local machine, we deploy it to each node in the first cluster with the following commands:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pnDeEqvqP-F3Pfgl9nU0vg.png" /></figure><p>To install ArangoDB, we run the following command on all cluster nodes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uIf1umbG22BxmGrbLlDx7w.png" /></figure><p><strong><em>A quick check to test the installation for functionality:</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eCBF5Ixx5WrHCctgMzMXvw.png" /></figure><p>This will launch a single server on each machine on port 8529, without any authentication, encryption, or anything. You can point your browser to the nodes on port 8529 to verify that the firewall settings are correct. If this does not work and you cannot reach the UI of the database, you should stop here and debug your firewall otherwise, you are bound to run into more difficult trouble later on, for example, because the processes in your cluster cannot reach each other over the network.</p><p>Afterward, simply press Control-C and run the following to clean up:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J2VS4kdTQI7UuViMWX_ieg.png" /></figure><p>Having tested basic functionality, let’s get to the actual deployment of the cluster;</p><p><strong><em>Create a shared secret for the first cluster</em></strong></p><p>The different processes in the ArangoDB cluster must authenticate themselves against each other. To this end, we require a shared secret, which is deployed to each cluster machine. Here is a simple way to create such a secret on your laptop and to deploy it to each of the cluster nodes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6b_PYpWvEOouvNaC3-yOcg.png" /></figure><p>Note that we are using the arangodb executable from the distribution to create a secret file secret.jwt. For this to work, you have to install ArangoDB on your laptop, too. If you want to avoid this, you can simply create all the secrets and keys on one of your cluster nodes and use arangodb there.</p><p>Please keep the file secret.jwt in a safe place, possession of the file grants unrestricted superuser access to the database.</p><p><strong><em>Create a CA and server keys, Then deploy them:</em></strong></p><p>All communications to the database as well as all communications within an ArangoDB cluster need to be encrypted via TLS. To this end, every process needs to have a pair of a private key and a corresponding public key (aka server certificate). During the steps of this blog post, we will create a self-signed CA key pair (the public CA key is signed by its own private key) and use that as the root certificate.</p><p>We use the following commands to create the CA keys tls-ca.key (private) and tls-ca.crt (public) as well as the server key and certificate in the keyfile files. A keyfile contains the private server key as well as the full chain of public certificates. Note how we add the server names into the server certificates:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TkY-jKQnNjRXi90STT3oug.png" /></figure><p>These commands are all executed on your local machine and deploy the server key to the cluster nodes. Keep the tls-ca.key file secure, it can be used to sign certificates, in particular, do not deploy it to your cluster! Furthermore, keep the sync-dc1-node*.keyfile files secure, since possession of them allows you to listen in to the communication with your servers.</p><p><strong><em>Create an encryption key for encryption at rest:</em></strong></p><p>ArangoDB can keep all the data on disk encrypted using the AES-256 encryption standard. This is a requirement for most secure database installations. To this end, we need a 32 byte key for the encryption. It can basically consist of random bytes. Use these commands to set up an encryption key:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YGzNsFQg1b8U7VxxRJDyxg.png" /></figure><p>Keep the encryption key secret, because possession allows one to open a database at rest, if one can get hold of the database files in the filesystem.</p><p><strong><em>Create a shared secret to use with ArangoSync:</em></strong></p><p>The data center to data center replication in ArangoDB is implemented as a set of external processes. This allows for scalability and minimal impact on the actual database operations. The executable for the ArangoSync system is called arangosync and is packaged with our Enterprise Edition. Similar to the above steps, we need to create a shared secret for the different ArangoSync processes, such that they can authenticate themselves against each other.</p><p>We produce the shared secret in a way that is very similar to the one for the actual ArangoDB cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dn1_EKvxHAziM3ebNPWmiw.png" /></figure><p>Keep the file syncsecret.jwt a secret, since its possession allows you to interfere with the ArangoSync system.</p><p><strong><em>Create a TLS encryption setup for ArangoSync:</em></strong></p><p>The same arguments about encrypted traffic and man-in-the-middle attacks apply to the ArangoSync system as explained above for the actual ArangoDB cluster. We choose to reuse the same CA key pair as above for the TLS certificate and key setup. For a change, we work with the same server keyfile for all three nodes.<br>Here we create the server keyfile, signed by the same CA key pair in tls-ca.key and tls-ca.crt:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RDSO3aCMGvaAJYfxiFjg5Q.png" /></figure><p>As usual, keep the file synctls.keyfile secure, since its possession allows it to listen to the encrypted traffic with the ArangoSync system.</p><p><strong><em>Set up client authentication setup for ArangoSync:</em></strong></p><p>There is one more secretive thing to set up before we can hit the launch button. The two ArangoSync systems in the two data centers need to authenticate each other. Actually, the second data center (“DC B”, the replica), needs to authenticate itself with the first data center (“DC A”, the original). Since this is cross data center traffic, the authentication is done via TLS client certificates.</p><p>We create and deploy the necessary files with the following commands on your local machine:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kpQwbQgpjnqD90N8a41AyQ.png" /></figure><p>Keep the file client-auth-ca.key secret, since it allows signing additional client authentication certificates. Do not store this on any of the cluster nodes.<br>Also, keep the file client-auth.keyfile secret, since it allows authentication with a syncmaster in either data center.</p><p><strong><em>The first cluster can now be launched:</em></strong></p><p>We launch the whole system by means of the ArangoDB starter, which is included in the ArangoDB distribution. We launch the starter via a systemd service file, which looks basically like the following snippet but feel free to adapt it to your needs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PpKo_d6CPoCc3wqBqSgv4g.png" /></figure><p>Apart from some infrastructural settings like the number of file descriptors and restart policies, the service file basically runs a single command. It refers to the starter program arangodb, which needs a few options to find all the secret files we have set up, these should be self-explanatory from what we have written above.</p><p>The network fabric of the cluster basically comes together since every instance of the starter gets told its own address (with the --starter.address option), as well as a list of all the participating starters (with the --starter.join option). We are achieving this by setting the actual server hostname in the line with Environment=SERVER=.... Then we can refer to this environment variable with the syntax ${SERVER} further down in the service file. This means that the above file has to be edited in just a single place for each individual machine, namely, you have to set the SERVER name.</p><p>Provided the above file has been given the name arango.service on your local machine, then you can deploy the service with the following commands on local machine:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KiK7Q7Iv1CvjBVKaepYY9A.png" /></figure><p>You then have to edit this file and adjust the server name, as described above. Then you launch the service with the following commands on each node in the cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rMrtGbDjMZCAVod3o7rKXw.png" /></figure><p>You can check the status of the service with:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sQ89g3ZPhZ8Ad5P6wCOMBQ.png" /></figure><p>Or investigate the live log file by running:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bKZpkF4VjhdiqXVVXnKvQQ.png" /></figure><p>Please note that all the data for the cluster resides in subdirectories of /arangodb/data. Every instance on each machine has a subdirectory there that contains its port in the directory name. You can find further log files in these subdirectories.</p><p>You should now be able to point your browser to port 8529 on any of the nodes. Before that, we recommend that you tell your browser to trust the tls-ca.crt certificate for server authentication. Since the public server keys are signed by the private CA key, your browser can then successfully prevent any man-in-the-middle attack.</p><p>An important step is now to change the root password, which will be empty in the beginning. You can use the UI for this.</p><p>We now set up the second cluster in a completely similar way. We simply show the commands used for that as they differ in some detail related to node names and such:</p><p><strong><em>Extract the downloaded binary in its target location:</em></strong></p><p>On your local machine run:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SGhGpIDosF2s5zlxn4EllQ.png" /></figure><p>Then on each machine of the second cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u2vQ6KCwtXbBuPWW491ZXw.png" /></figure><p><strong><em>Create a shared secret for the second cluster</em></strong></p><p>Perform these commands on your local machine:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zrkTXQx6Lhvvx6x2BFs3tA.png" /></figure><p>Warning: Keep the file secretdc2.jwt in a safe place, possession of the file grants unrestricted superuser access to the database.</p><p><strong><em>Create a CA and server keys, Then deploy them:</em></strong></p><p>Note that we are using the same pair of CA keys for the TLS setup here as before during the preparation of the first cluster, so we rely on the files tls-ca.key and tls-ca.crt on your local machine. Perform these commands:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BQUX9wSprwqqiXKhAlVD6g.png" /></figure><p>Keep the file sync-dc2-nodes.keyfile secure, since possession of it allows one to listen in to the communication with your servers.</p><p><strong><em>Create an encryption key for encryption at rest</em></strong></p><p>This is totally parallel to what we did for the first cluster. On your local machine run:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L7m6dpahUign6EjbMUBzvA.png" /></figure><p>Keep the encryption key secret, because possession allows one to open a database at rest, if one can get hold of the database files in the filesystem.</p><p><strong><em>Create a shared secret to use with ArangoSync:</em></strong></p><p>Run the following on your local machine:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ERmK32p3kkmJGR4EMNV_Lg.png" /></figure><p>Keep the file syncsecretdc2.jwt a secret, since its possession allows it to interfere with the ArangoSync system.</p><p><strong><em>Create a TLS encryption setup for ArangoSync:</em></strong></p><p>Again, we proceed exactly as for the first cluster. Do this on your local machine:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jldXnF4S6tHRjsGQnereIg.png" /></figure><p>As usual, keep the file synctlsdc2.keyfile secure, since its possession allows you to listen to the encrypted traffic with the ArangoSync system.</p><p><strong><em>Set up client authentication setup for ArangoSync:</em></strong></p><p>Note that for simplicity, we use the same client authority CA for DC B as we did for DC A. This is not necessary but avoids a bit of confusion. On your local machine run:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n0u2IDpCczMdz23WtAcRuQ.png" /></figure><p>Make sure to keep the file client-auth-ca.key and client–auth-par.keyfile secretly stored outside of the cluster.</p><p><strong><em>Now, we’re ready to launch the second cluster:</em></strong></p><p>We use a very similar service file like we did with the first cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DxJ1TYpjFmHS5V87jdHA1Q.png" /></figure><p>Provided the above file is named arango-dc2.service on your local machine, then you can deploy the service with the following commands on your local machine:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EcVmOX_yJR2D3y6FwDXmyQ.png" /></figure><p>Then, edit this file and adjust the server name, as described above. Then you need to launch the service with these commands on each machine in the cluster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3uLHz_YSRnc9JmfPzzqfgg.png" /></figure><p>You can query the status of the service with:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S4SjwgahyjX3mfGig6jCOA.png" /></figure><p>Or investigate the live log file by running:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pgiER0C46zwVw9fxjlFrYw.png" /></figure><p>Note that all the data for the cluster resides in subdirectories of /arangodb/data. Every instance on each machine has a subdirectory there which contains its port in the directory name. You can find further log files in these subdirectories.</p><p>You should now be able to point your browser to port 8529 on any of the nodes and connect to the ArangoDB UI.</p><p>Don’t forget to change the root password for your installation. This can be easily done via the ArangoDB UI.</p><p><strong><em>Enable ArangoSync synchronization and start it by using the CLI:</em></strong></p><p>ArangoSync is controlled via its CLI, this CLI will be installed together with ArangoDB so there’s no need to separately search for it in order to download and install it.</p><p>To configure DC to DC synchronisation from DC A to DC B, you now have to run this command on your local machine:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xe0TVKIsJae2Q2O6uKHfjg.png" /></figure><p>If you want (or need) to check if replication is running, you can run the following two commands:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nuto_Vm6xD7Oft9G-_sV_A.png" /></figure><p><strong><em>Get detailed information on running synchronization tasks:</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RhrCLg6kzo80esgEHzqrKA.png" /></figure><p><strong><em>Momentarily stop the synchronization process:</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G44UqUHp1_jMrT1W0sZC0Q.png" /></figure><p>This will briefly stop writes to DC A until both clusters are in perfect sync. It will then stop synchronization and switch DC B back to read/write mode. You can use the switch --ensure-in-sync=false if you do not want to wait for synchronization to be ensured.</p><p><strong><em>Abort synchronization:</em></strong></p><p>Losing connectivity between the two separate locations because of a network or other outage will mean you need to stop synchronization entirely with the following command:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_wBLV59vjt-vK52-3B3_rA.png" /></figure><p>This is the command to abort the synchronization on the target side (DC B). If there is no connectivity between the clusters, then this will naturally not abort the outgoing synchronization in DC A. Therefore, it is possible that you have to additionally send a corresponding command to the syncmaster in DC A, too.</p><p>It has a slightly different syntax:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qizJLEX1Hz87-XLpLkm0Kg.png" /></figure><p>The &lt;id&gt; will need to be replaced by the ID of the cluster in DC B. You can retrieve this ID via the arangosync get status output of DC A. There you can also tell if this is necessary, if you see an outgoing synchronization which does not have a corresponding incoming synchronization in the other DC, the ArangoSync abort outgoing sync command is needed.</p><p><strong><em>Restart synchronization in the opposite direction:</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w2ICcqyhC8EJGLyS2r4TAQ.png" /></figure><p>More details on the ArangoSync CLI and its options can be found in our documentation at:</p><p><a href="https://www.arangodb.com/docs/stable/administration-dc2-dc.html">https://www.arangodb.com/docs/stable/administration-dc2-dc.html</a></p><p><strong>Concluding words</strong></p><p>As I wrote at the very beginning of this blog post, we have thought long and hard about an ideal topic and we’re very excited that a lot of our users want to use ArangoSearch either to test or in production. We truly hope that this is a great start for those looking into a rock-solid replicated database environment and wish you serious heaps of fun rolling it out and check out the benefits!</p><h3>Author</h3><p><a href="https://www.arangodb.com/author/edgar/">Edgar Lanting</a></p><p>Edgar is a seasoned distributed systems &amp; database engineer who loves to dive under the hood of systems to get to the bottom of a problem and the inner workings of components. When not working, Edgar likes to spend time with his wife and dog traveling Europe in their campervan as well as photographing, mountain biking &amp; snowboarding.</p><figure><img alt="Edgar Lanting" src="https://cdn-images-1.medium.com/max/190/1*f9TGIo_Pdf6QxToouaTzlg.jpeg" /><figcaption>Edgar Lanting</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf07f9d8128d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/arangodb/arangosync-a-recipe-for-reliability-bf07f9d8128d">ArangoSync: A Recipe for Reliability</a> was originally published in <a href="https://medium.com/arangodb">ArangoDB</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Who’s Who in Data Science]]></title>
            <link>https://medium.com/arangodb/whos-who-in-data-science-669a16263e6?source=rss-d2de8c290e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/669a16263e6</guid>
            <category><![CDATA[arangodb]]></category>
            <category><![CDATA[graph-technology]]></category>
            <category><![CDATA[graph-database]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[ArangoDB]]></dc:creator>
            <pubDate>Fri, 24 Jun 2022 14:06:52 GMT</pubDate>
            <atom:updated>2022-06-24T14:06:52.237Z</atom:updated>
            <content:encoded><![CDATA[<p>Multiple data science personas participate in the daily operations of data logistics and intelligent business applications. Management and employees need to understand the big picture of data science to maximize collaboration efforts for these operations. This article will highlight the specialized roles and skillsets needed for the different data science tasks and the best tools to empower data-driven teams. You will come away from this article with a better understanding of how to support your own data science teams, and it is valuable for both managers and team members alike.</p><p>This article will cover some of the different data science personas that exist primarily within the engineering space, aiming to answer the questions:</p><ul><li>What are the different views of Enterprise data?</li><li>What are the different data science personas?</li><li>What is each persona’s relationship to the data?</li><li>What tools do they use?</li><li>What challenges do they face both independently and collaboratively?</li></ul><p>We will first need to understand how data is transformed for valuable data science applications and its lifecycle within an enterprise. We will quickly cover some terms that can have different meanings depending on who you are talking to about them. When talking about ‘buzzy’ terms like Enterprise, Big, Operational, and Analytic Data, we must make sure we are talking about the same thing. As these are ever-evolving things, it is equally important to clarify what we perceive these terms to mean, introducing the personas along the way.</p><p>There is a lot of information to absorb in this article. So we decided to wrap up with an easily digestible infographic that summarizes the personas we cover here. We won’t blame you for skipping to the bottom if you want the downloadable TLDR.</p><h3>Enterprise Data</h3><p>We will follow Fred trying to get a new loan application processed throughout the article. The loan processing includes multiple steps within the enterprise, including the operational and analytic data stores.</p><p>Data can have a complex life cycle involving multiple teams and data science personas in a large-scale enterprise environment. The life cycle involves aggregating data from various business (operational) processes, cleaning, and transforming it. The data will be in a form suitable for analysis and performing required analytical tasks to develop data science-driven business applications.</p><p>The following diagram shows how data is moved around or accessed within an organization. Data could be potentially stored in any or all of the following:</p><ul><li>Analytical Data Warehouse</li><li>Operational Data Store</li><li>External Data Partners/Platforms</li></ul><p>Data will move across different sectors of the enterprise for various tasks relating to:</p><ul><li>Transformation and storage based on business rules</li><li>ETL tasks for loading and modeling data across departments</li><li>Feature engineering and machine learning</li><li>Analytic tasks for technical and non-technical resources</li><li>Production and monitoring</li></ul><figure><img alt="Example of data movement within an organisation" src="https://cdn-images-1.medium.com/max/1024/0*5aNkUTTUZezlHDvW" /><figcaption>Example of data movement within an organisation</figcaption></figure><h3>Challenges of Big Data</h3><p>Big data workloads in the enterprise occur in both operational and analytic environments. Operational workloads capture data from day-to-day business operations. This data is cleaned and later moved to an analytic data warehouse to develop analytic applications.</p><p>Understanding the specifics of Enterprise and Big Data is important because understanding the challenges of scaling data is key to understanding the personas that solve these challenges and then utilize the data from there.</p><p>See the <a href="https://www.iso.org/obp/ui/#iso:std:iso-iec:20546:ed-1:v1:en">ISO specification</a> for a formal definition of Big Data. In less specification-laden terms, you are working with big data when you must scale beyond the limits of vertical scalability, possibly scaling horizontally as a result of:</p><ul><li>The amount of data being stored and analyzed is increasing. (volume)</li><li>The processing speed of this data detrimentally decreases. (velocity)</li><li>The stored data contains multiple model types and complicates processing, requiring more compute resources. (variety)</li><li>The combination of the previous three points, the inability to keep up with the dynamic nature of the data coming into the system, and the specific needs for accessing the data frequently change. (variability)</li></ul><h3>Data Analyst</h3><p>Primarily concerned with analyzing and presenting data to enable the business to make decisions. They are interested in evaluating business hypotheses to enhance a product feature or understand and assess business conditions. They evaluate a business hypothesis and prepare reports that capture KPIs of business performance.</p><p>Data analysts require friction-free data access and an advanced understanding of the database query language. Since the data analysts’ primary focus is business intelligence, they mainly work with reporting tools and limited data science libraries. They are the primary resource to present data to the more non-technical contacts of the business. Thus, they need to access the data required consistently and flexibly to deliver the best view of information to their contacts.</p><p>Next up, we dig a little deeper into operational and analytic data.</p><h3>Operational Data</h3><p>Operational Big Data systems provide operational capabilities to run real-time, interactive workloads that ingest and store data. These stores offer a unified access point for business applications. Having data housed in these big data stores results in a single source of precious data. Rather than needing to make multiple trips across multiple different data stores, the application can retrieve all necessary information at once.</p><figure><img alt="Example of data movement within an organisation" src="https://cdn-images-1.medium.com/max/1024/0*sRhTMJel7bCWblsT" /><figcaption>Example of data movement within an organisation</figcaption></figure><p>For the operational data store to maintain its value, it must also be able to scale effectively. One approach for scaling is also what keeps operational data stores so valuable. The operational data store is also behind a business rules application that ensures that only the most valued data is stored in this operational data store. Typically, the raw uncleaned data is kept in a data warehouse to be used later, likely with analytical applications that might need unknown data.</p><p>In our loan application scenario, the operational data store integrates data services for multiple parts of the loan application process. There may be a credit score application, risk analysis, general bookkeeping, and property value assessment applications that all eventually store or pull data from the operational data store.</p><h3>Data Engineers</h3><p>Data engineers are primarily concerned with engineering the data pipelines and the infrastructure needed to support the needs of the other three participants in this table. They collaborate extensively with the other participants and understand the needs motivating the data requirements of the other participants.</p><p>Data engineers need to perform experiments to evaluate if a pipeline fulfills the needs of the consumers. They require proficiency with data querying as well as data integration tools. The data engineer will spend a lot of their time doing tasks such as data wrangling and are in charge of systems controlling data governance and archival.</p><h3>Data Processing</h3><p>A large portion of data science work involves dealing with the data from our daily big data operations. Big data platforms are in charge of collecting digital exhaust, interaction analytics, logs, and non-business data. These systems typically handle tasks such as <a href="https://www.gartner.com/en/information-technology/glossary/clickstream-analysis">clickstreams </a>and interface with systems such as Hadoop and Spark, which feed into other systems, including Impala and Hive. These systems are the primary domain for a data engineer.</p><p>There needs to be an <a href="https://en.wikipedia.org/wiki/Extract,_transform,_load">ETL</a> layer within these systems that ensures that the data is prepared for the eventual machine learning or analytics work.</p><figure><img alt="Example of data movement within an organisation" src="https://cdn-images-1.medium.com/max/1024/0*391tIcrmjklSmUFg" /><figcaption>Example of data movement within an organisation</figcaption></figure><p>Our diagram shows that the data storage and processing systems bridge our operational data store and the ML and analytics processes. This diagram further highlights the importance of the data engineer and how vital it is that they can effectively collaborate with the other participants.</p><h3>Analytic Data</h3><p>Suppose we continue with the loan application example from the operational data store section. In that case, analytic data involves the process of aggregating data from multiple parts of the operational applications and eventually running, typically read-only, queries to infer new information. So the analytical side would be more interested in the default rate for loans granted over five years and the associated credit scores to determine if the business should update the loan approval process.</p><p>Typically, operational data stores operate in an environment where they respond to events and persist data. They are more involved in the day-to-day operations of the application, while the analytic data stores are involved in the analysis of the days’, months’, or years’ work.</p><p>Processing the business data for analytical purposes could involve removing personally identifiable information and performing other regulatory checks, for example, verifying that all required information has been consolidated. (in a loan, this could be things like late charges). Once this is done, the data is suitable for analysis to support business decisions, for example:</p><p>1. Understanding customer aggregate behavior, ex: grouping customers into good credit, lower risk, and customers who have defaulted.</p><p>2. Developing cross-selling applications, ex: target customers with good credit ratings for opportunities to apply for new loans.</p><p>3. Identification of a set of activities that are possibly suspicious or require further investigation.</p><h3>Data Scientist</h3><p>A data scientist is primarily concerned with experimentation and hypothesis evaluation on business data. These activities support the improvement of existing business applications or motivate the development of new ones. They answer the question: “What is the best way to solve this problem with a model?” Once they have answered that question, they are in charge of creating and defining the real-world solution.</p><p>A data scientist needs to be able to obtain the data necessary to evaluate their hypotheses. This process is directly tied to the work of the data engineers. They must also be able to document their findings for reproducibility and monitoring, should the model be moved to production. The capturing of machine learning pipeline metadata and model productionizing is the responsibility of the machine learning engineer.</p><p>While a data scientist may be a database query guru, they need a database with a robust and easy to query language to facilitate getting the necessary data for their external data science libraries. The bulk of their query work will be light analytics and data exploration of the data stored in the database.</p><p>They will primarily work with data science libraries and utilities such as NetworkX, PyTorch, Tensorflow, and visualization libraries. The data scientist wil be in charge of figuring out the algorithm to use to solve the problem they are working on but this solution usually needs a bit of refining and this is where the Machine Learning Engineer comes in.</p><h3>Machine Learning Engineer</h3><p>Finally, the machine learning pipelines are tightly coupled with the analytics systems and require someone to manage this relationship; this is the job of the machine learning engineer. A machine learning engineer optimizes and refines established data science models. They are primarily concerned with productionizing an optimized machine learning model. As shown in our diagram, the model generation performed by the data scientists is what the ML engineer will deploy, monitor, and productionize. They will follow up with the data analyst for quality assurance and ensure friction-free access to the resulting data. They will also communicate model performance metrics back to the data scientist.</p><p>The ML engineer will need expertise in both the field of operations and machine learning to deploy the most optimal models; finding these optimizations is usually the result of experimentation. The ML Engineer usually works with model information provided by the data scientist and attempts to clean it up and make it as efficient and production-ready as possible.</p><figure><img alt="Example of data movement within an organisation" src="https://cdn-images-1.medium.com/max/1024/0*Eefs4Iet5kP3QS5x" /><figcaption>Example of data movement within an organisation</figcaption></figure><h3>Data Science Personas Infographic</h3><p>This infographic summarizes the data included in this article. It briefly describes the role of each persona, what they need to achieve their day-to-day tasks and the tools and features they use or desire.</p><p>(Click to open larger size)</p><figure><img alt="Who’s Who in Data Science" src="https://cdn-images-1.medium.com/max/1000/0*9lx_oGDbWBUV3xFc.png" /><figcaption>Who’s Who in Data Science</figcaption></figure><h3>Conclusion</h3><p>This article covered four of the different data science personas that make up many data science teams. You should now understand more about these personas, the different views of Enterprise data, and each persona’s relationship to the data.</p><p>Hopefully, this article will empower you to either understand your own role within your team better or to better understand how to better support your data science teams.</p><h3>Author</h3><p><a href="https://www.arangodb.com/author/chris/">Christopher Woodward</a></p><p>Chris has over 10 years experience at all angles of technology including service, support, and development. He is also passionate about learning and right now he is focused on improving the learning experience for the ArangoDB community. Chris believes the future is native multi-model and wants to help tell the world.</p><figure><img alt="Christopher Woodward" src="https://cdn-images-1.medium.com/max/190/1*z3SkumwNUi77a3YBIj7N9g.png" /><figcaption>Christopher Woodward</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=669a16263e6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/arangodb/whos-who-in-data-science-669a16263e6">Who’s Who in Data Science</a> was originally published in <a href="https://medium.com/arangodb">ArangoDB</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Detecting Complex Fraud Patterns with ArangoDB]]></title>
            <link>https://medium.com/arangodb/detecting-complex-fraud-patterns-with-arangodb-4d6bb206360a?source=rss-d2de8c290e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d6bb206360a</guid>
            <category><![CDATA[aql]]></category>
            <category><![CDATA[arangodb]]></category>
            <category><![CDATA[graph-database]]></category>
            <category><![CDATA[fraud-detection]]></category>
            <category><![CDATA[graph-traversal]]></category>
            <dc:creator><![CDATA[ArangoDB]]></dc:creator>
            <pubDate>Fri, 10 Jun 2022 13:25:00 GMT</pubDate>
            <atom:updated>2022-06-10T13:25:00.305Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>This article presents a case study of using AQL queries for detecting complex money laundering and financial crime patterns. While there have been multiple publications about the advantages of graph databases for fraud detection use cases, few of them provide concrete examples of implementing detection of complex fraud patterns that would work in real-world scenarios.</p><p>This case study is based on a third-party transaction data generator, which is designed to simulate realistic transaction graphs of any size. The generator disguises complex financial fraud patterns of two kinds:</p><ul><li>Circular money flows: a big amount of money is going through different nodes and comes back to the source node.</li><li>Indirect money transfers: a big amount of money is sent from source node to a target node over a multi-layered network of intermediate accounts.</li></ul><p>In this post, we demonstrate that AQL provides declarative, yet efficient means for detecting such patterns and showcase a set of ArangoDB features:</p><ul><li>Complex graph traversal and visualization</li><li>Advanced AQL filtering and aggregation</li><li>Query Parallelism</li><li>The <a href="https://www.arangodb.com/docs/3.8/aql/operations-window.html">WINDOW</a> function introduced in ArangoDB 3.8</li></ul><h3>Dataset</h3><p>In the case study, the dataset we use is produced by a third-party generator of financial transaction data available at:</p><p><a href="https://github.com/MGrin/transactions-graph-generator">https://github.com/MGrin/transactions-graph-generator</a></p><p>The generator can produce any data size and thus is invaluable for testing fraud detection at a large scale. A sample dataset with 1.5 million transactions that we used for testing can also be downloaded from the following link and imported using arangorestore:</p><p><a href="https://drive.google.com/file/d/1P8XFFzEn3nOHaQUwJcExp5uYuwOIAZJO/view?usp=sharing">https://drive.google.com/file/d/1P8XFFzEn3nOHaQUwJcExp5uYuwOIAZJO/view?usp=sharing</a></p><figure><img alt="Three Vertex Collection Diagram" src="https://cdn-images-1.medium.com/max/788/0*Nnz1IsDV6DbGzhGe" /><figcaption>Three Vertex Collection Diagram</figcaption></figure><p>In ArangoDB, we represent the generated dataset by 3 vertex collections, one for each entity type:</p><ul><li>clients</li><li>companies</li><li>atms.</li></ul><p>Each entity is identified by a UUID-based key and described by a JSON document. For example, we have a client entity with the key d942a1fe-ad32–46da-9630–4622c4c01354 and describe it as:</p><pre>{<br>  &quot;first_name&quot;: &quot;Sharice&quot;,<br>  &quot;last_name&quot;: &quot;Quinn&quot;,<br>  &quot;age&quot;: 36,<br>  &quot;email&quot;: &quot;conservatives1812@gmail.com&quot;,<br>  &quot;occupation&quot;: &quot;Tool Maker&quot;,<br>  &quot;political_views&quot;: &quot;Libertarian&quot;,<br>  &quot;nationality&quot;: &quot;Ecuadorian&quot;,<br>  &quot;university&quot;: &quot;Florida Polytechnic University (Florida Poly or FPU)&quot;,<br>  &quot;academic_degree&quot;: &quot;None&quot;,<br>  &quot;address&quot;: &quot;1348 San Felipe Square&quot;,<br>  &quot;postal_code&quot;: 62492,<br>  &quot;country&quot;: &quot;United States&quot;,<br>  &quot;city&quot;: &quot;Prior Lake&quot;,<br>  &quot;label&quot;: &quot;Sharice Quinn&quot;<br>}</pre><p>Transactions are modeled by an edge collection connecting the vertices representing the sources and the destinations of the transactions. Each document in the collection has the special attributes _from and _to, which act as references to the source and destination nodes, and a set of further attributes describing the time, the amount and the currency of the transaction, e.g.:</p><pre>{<br>  &quot;_from&quot;: &quot;clients/fddbced9-0d3f-4dbf-84dd-61ea27a718e9&quot;,<br>  &quot;_to&quot;: &quot;clients/72cef499-ba34-4360-ab65-c9a8e01e5060&quot;,<br>  &quot;date&quot;: &quot;2015-12-10&quot;,<br>  &quot;time&quot;: &quot;04:35:05.632384&quot;,<br>  &quot;amount&quot;: 96.01706505950199,<br>  &quot;currency&quot;: &quot;USD&quot;<br>}</pre><h3>Cycle Detection</h3><p>One of the money laundering patterns is a circular money flow. Money is being transferred from one account to another and eventually lands back in the originator’s account.</p><p>To detect such a circular flow, we begin at a particular transaction from account A to account B and monitor transactions that eventually link back to account A. We check for transactions from account B at a later time but with a similar amount of money. We continue following the transactions from account to account to check if we return to account A. A sequence of transactions showing this money laundering pattern would result in a graph such as this one, which results from the AQL query in the following section.</p><figure><img alt="Money Laundering Pattern Graph" src="https://cdn-images-1.medium.com/max/914/0*dfujmT3RwQLmUKPL" /><figcaption>Money Laundering Pattern Graph</figcaption></figure><h3>Detecting a Circular Money Flow</h3><p>A very simple AQL query can detect if there is a circle of transactions starting at a given transaction @firstTrans:</p><pre>LET t = DOCUMENT(@firstTrans)<br>FOR v, e, p IN 2..10 OUTBOUND t._to transactions<br>   FILTER e._to == t._from<br>   LET cycle = APPEND([t], p.edges)<br>   RETURN cycle</pre><p>The query attempts a deep graph traversal up to level 10 and uses the filter `e._to == t._from` to check that we end up at the account where we started.</p><p>However, such a query won’t work in practice, because in a realistic data set we will have thousands of transactions per account and a traversal of depth 3 would already need to visit billions of paths. Also, there will be a lot of accidental circular paths in the transaction graph that do not necessarily represent a circular money flow.</p><p>In a circular money flow, money is retransferred after it was received, so we can filter the transactions that happen within a certain time window after the first one. Also, the amount of money being retransferred is similar to the original amount. So we can optimize our query by applying corresponding filters on the dates and amount attributes of all transactions on the path:</p><pre>// Example:<br>LET t = DOCUMENT(@firstTrans)<br>LET beginDate = t.date<br>LET endDate = DATE_ADD(t.date, 100, &quot;day&quot;)<br>LET lowerAmount = t.amount * 0.95<br>   LET upperAmount = t.amount * 1.01<br>   FOR v, e, p IN 2..10 OUTBOUND t._to transactions<br>      FILTER (p.edges[*].date ALL &gt;= beginDate) AND <br>             (p.edges[*].date ALL &lt;= endDate) AND<br>             (p.edges[*].amount ALL &gt;= lowerAmount) AND <br>             (p.edges[*].amount ALL &lt;= upperAmount) AND<br>             (e._to == t._from)<br>      LET cycle = APPEND([t], p.edges)<br>// All of the variables are returned as a performance measure<br>      RETURN { beginDate, endDate, lowerAmount, upperAmount, cycle }</pre><p>The AQL optimizer recognizes that these conditions must be satisfied by all edges in the path and performs an optimization that avoids traversing the edges that do not satisfy the filter. As a result, this query can perform a deep traversal on a large dataset within milliseconds.</p><p>For example, within just 10ms we can find a cycle of length 8 beginning at <br>@firstTrans = ‘transactions/0976884d-b460–406e-8d18–7ddf4e56fc92’</p><h3>Generate a Graph Visualization</h3><p>Thanks to the built-in graph viewer of the WebUI we can easily generate a graph with our results by extracting and returning the edges from our results:</p><pre>// Example:<br>// @firstTrans = &#39;transactions/0976884d-b460-406e-8d18-7ddf4e56fc92&#39;<br>LET cycles = (<br>   LET t = DOCUMENT(@firstTrans)<br>   LET beginDate = t.date<br>   LET endDate = DATE_ADD(t.date, 100, &quot;day&quot;)<br>   LET lowerAmount = t.amount * 0.95<br>   LET upperAmount = t.amount * 1.01<br>   FOR v, e, p IN 2..10 OUTBOUND t._to transactions<br>      FILTER (p.edges[*].date ALL &gt;= beginDate) AND <br>             (p.edges[*].date ALL &lt;= endDate) AND<br>             (p.edges[*].amount ALL &gt;= lowerAmount) AND <br>             (p.edges[*].amount ALL &lt;= upperAmount) AND<br>             (e._to == t._from)<br>      LET cycle = APPEND([t], p.edges)<br>	// All of the variables are returned as a performance measure<br>      RETURN { beginDate, endDate, lowerAmount, upperAmount, cycle })<br>FOR c IN cycles<br>   FILTER c.beginDate != null AND c.endDate != null <br>          AND c.lowerAmount != null AND c.upperAmount != NULL<br>   FOR e IN c.cycle<br>      RETURN e</pre><figure><img alt="Money Laundering Pattern Graph" src="https://cdn-images-1.medium.com/max/914/0*USsU0u7ftQJb7o-8" /><figcaption>Money Laundering Pattern Graph</figcaption></figure><h3>Checking Suspicious Clients</h3><p>As a result of flagging a transaction as suspicious, we may decide it is necessary to investigate all transactions of the client in question for other fraudulent behavior. This is accomplished with a small change to our initial FOR loop. We change the initial transaction lookup to a graph traversal that starts at a specified client and traverses all of their transactions using the same criteria from the first query.:</p><pre>// @client=&quot;clients/8bf6c520-b0e3-470d-9470-f8d3c3ce1f52&quot;<br>LET cycles = (<br>   FOR c, t IN OUTBOUND @client transactions<br>      LET beginDate = t.date<br>      LET endDate = DATE_ADD(t.date, 30, &quot;day&quot;)<br>      LET lowerAmount = t.amount * 0.95<br>      LET upperAmount = t.amount * 1.01<br>      FOR v, e, p IN 2..10 OUTBOUND t._to transactions<br>         FILTER (p.edges[*].date ALL &gt;= beginDate) AND <br>                (p.edges[*].date ALL &lt;= endDate) AND<br>                (p.edges[*].amount ALL &gt;= lowerAmount) AND <br>                (p.edges[*].amount ALL &lt;= upperAmount) AND<br>                (e._to == t._from)<br>         LET cycle = APPEND([t], p.edges)<br>         RETURN { beginDate, endDate, lowerAmount, upperAmount, cycle })<br>FOR c IN cycles<br>   FILTER c.beginDate != null AND c.endDate != null <br>      AND c.lowerAmount != null AND c.upperAmount != null<br>   FOR e IN c.cycle<br>      RETURN e</pre><figure><img alt="Money Laundering Pattern Graph" src="https://cdn-images-1.medium.com/max/748/0*ajiuKAe3t0G8I9nu" /><figcaption>Money Laundering Pattern Graph</figcaption></figure><h3>Checking Every Transaction</h3><p>Performing analysis on a single transaction or client is very important, but analyzing all of the available transactions to start getting a picture of just how widespread fraud is in your dataset is where some serious business value is gained.</p><p>We can scan the entire data set for possible cycles, e.g., check all transactions with a large enough amount or are recent enough. We can significantly reduce the execution of such queries by employing parallel traversals. As a result, it takes only 15s to detect all cyclic money flows in our sample data set on a 4-CPU core machine. Traversal parallelization was introduced in ArangoDB 3.7 and allows for executing nested traversals in parallel on a single server and a clustered environment. The amount of value received from this optimization is dependent on the number of available cores. See the <a href="https://www.arangodb.com/docs/3.7/release-notes-new-features37.html#traversal-parallelization-enterprise-edition">release notes</a> and <a href="https://www.arangodb.com/docs/3.7/aql/graphs-traversals.html#working-with-named-graphs">graph traversal documentation</a> for more information.</p><pre>FOR t IN transactions<br>   FILTER t.amount &gt; 10000<br>   LET beginDate = t.date<br>   LET endDate = DATE_ADD(t.date, 100, &quot;day&quot;)<br>   LET lowerAmount = t.amount * 0.95<br>   LET upperAmount = t.amount * 1.01<br>   FOR v, e, p IN 2..10 OUTBOUND t._to transactions<br>      OPTIONS { parallelism : 4 }<br>      FILTER (p.edges[*].date ALL &gt;= beginDate) AND <br>             (p.edges[*].date ALL &lt;= endDate) AND<br>             (p.edges[*].amount ALL &gt;= lowerAmount) AND <br>             (p.edges[*].amount ALL &lt;= upperAmount) AND<br>             (e._to == t._from)<br>      LET cycle = APPEND([t], p.edges)<br>      RETURN { beginDate, endDate, lowerAmount, upperAmount, cycle }</pre><p>As before, we can easily generate a graph with our results by extracting the edges from our results:</p><pre>LET res =<br>(FOR t IN transactions<br>   FILTER t.amount &gt; 10000<br>   LET beginDate = t.date<br>   LET endDate = DATE_ADD(t.date, 100, &quot;day&quot;)<br>   LET lowerAmount = t.amount * 0.95<br>   LET upperAmount = t.amount * 1.01<br>   FOR v, e, p IN 2..10 OUTBOUND t._to transactions<br>      OPTIONS { parallelism : 4 }<br>      FILTER (p.edges[*].date ALL &gt;= beginDate) AND <br>             (p.edges[*].date ALL &lt;= endDate) AND<br>             (p.edges[*].amount ALL &gt;= lowerAmount) AND <br>             (p.edges[*].amount ALL &lt;= upperAmount) AND<br>             (e._to == t._from)<br>      LET cycle = APPEND([t], p.edges)<br>      RETURN { beginDate, endDate, lowerAmount, upperAmount, cycle })<br>FOR r IN res<br>   FILTER r.beginDate != null AND r.endDate != null AND r.lowerAmount != null AND r.upperAmount != NULL<br>   FOR e IN r.cycle<br>      RETURN e</pre><p>Now we can see an excellent visualization of all the money cycles in the dataset:</p><figure><img alt="Money Transfer cycles in Dataset" src="https://cdn-images-1.medium.com/max/1024/0*Dg-fArLpk4SSOLzR" /><figcaption>Money Transfer cycles in Dataset</figcaption></figure><h3>Money Flow Pattern Detection</h3><p>Another money laundering pattern hidden in the generated data set is indirect money transfers over multiple accounts. Money is split into smaller amounts and sent to the destination over multiple layers of intermediate accounts. We can try to detect such a pattern by looking for a sequence of transactions from one account to multiple other accounts that are relatively close in time and are followed by retransfers of a similar amount of money to further accounts.</p><figure><img alt="Illustration- Transaction over multiple layers" src="https://cdn-images-1.medium.com/max/1024/0*FMRX29T2z_VKuM9D" /><figcaption>Illustration- Transaction over multiple layers</figcaption></figure><p>The problem of detecting a money flow network is more complicated than detecting circular money flows because in this case, our results are not just separate paths in the graph, but rather an entire subgraph. Our idea of detecting such graphs is based on solving the problem in two phases:</p><ul><li>First detecting and marking suspicious transactions that are potentially part of such money flows</li><li>Traverse the graph of suspicious transactions to detect if they build a connected network with a single source and destination</li></ul><h3>Identifying Suspicious Transactions</h3><p>For the first part, we can use the <a href="https://www.arangodb.com/docs/3.8/aql/operations-window.html">WINDOW</a> function introduced in ArangoDB 3.8 to aggregate over accounts in a specific time range efficiently. We look at the time window from 5 days before and up to 30 days after the current transaction and check if it is the first one in this range so that it begins the pattern:</p><pre>FOR acc1, t1 IN OUTBOUND @clientId transactions<br>   LET beginDate = DATE_TIMESTAMP(t1.date)<br>   WINDOW beginDate WITH { preceding : &quot;P5D&quot;, following : &quot;P30D&quot; }<br>      AGGREGATE seq = UNIQUE(t1)<br>   FILTER seq[0]._key == t1._key <br>   RETURN { seq: seq[*]._key }</pre><p>Of course, there will likely be a lot of such transaction sequences for each account. For detecting potentially fraudulent behavior, it is also important to check if the money is being transferred to multiple accounts and whether the recipient of the money retransfers it further. So we add a subquery, which checks whether each recipient has transferred at least 90% of the received money within the following 30 days.</p><pre>/*<br>clientId: &quot;​​clients/8fc3b93f-a27e-428c-b5a4-3ec2c242e31b&quot;<br>*/<br>FOR acc1, t1 IN OUTBOUND @clientId transactions<br>   LET beginDate = DATE_TIMESTAMP(t1.date)<br>   WINDOW beginDate WITH { preceding : &quot;P5D&quot;, following : &quot;P30D&quot; }<br>      AGGREGATE seq = UNIQUE(t1)<br>   FILTER seq[0]._key == t1._key <br>   FILTER COUNT_UNIQUE(seq[*]._to) &gt; 1 <br>   LET retransferCnt = COUNT(<br>      FOR t2 IN seq<br>         LET receivedAmount = <br>                SUM(FOR n, t3 IN OUTBOUND t2._to transactions<br>                       FILTER t3.date &gt;= t2.date <br>                         AND t3.date &lt;= DATE_ADD(t2.date, &quot;P30D&quot;)<br>                       RETURN t3.amount)<br>         FILTER receivedAmount &gt; t2.amount * 0.9<br>         RETURN 1)<br>   FILTER retransferCnt == COUNT(seq)<br>   RETURN { seq: seq[*]._key }</pre><p>The query identifies patterns of potential participation of the client in the money flows. Still, to visualize the pattern from the beginning to the end through all intermediate layers, we will perform such analysis for all clients. In practice, since the transactions constituting a pattern are close in time, such analysis can be applied incrementally, each time processing only relatively recent transactions.</p><p>We can also make our search more efficient by applying additional filters:</p><ul><li>Ignore all small transactions, i.e. with an amount &lt; 100</li><li>The money transfers in a sequence are of similar size</li><li>The overall transferred amount in the sequence exceeds 1000</li><li>The average time between the money transfers in the sequence does not exceed 5 days</li></ul><p>Also instead of returning the suspicious patterns, as a result, we instead mark the suspicious edges, so we can traverse over them afterward to visualize entire money flows. More specifically, in each suspicious edge, we write an attribute outboundLead pointing to the transaction that leads the pattern it belongs to:</p><pre>FOR c IN clients<br>  LET suspPatterns = <br>     (FOR acc1, t1 IN OUTBOUND c transactions<br>         FILTER t1.amount &gt; 100 AND !t1.atm<br>         LET beginDate = DATE_TIMESTAMP(t1.date)<br>         WINDOW beginDate WITH { preceding : &quot;P5D&quot;, following : &quot;P30D&quot; }<br>         AGGREGATE seq1 = UNIQUE(t1)<br>         LET seq = seq1[* FILTER ABS(CURRENT.amount - t1.amount) <br>                            &lt; t1.amount * 0.1]<br>         FILTER seq[0]._key == t1._key <br>         FILTER COUNT_UNIQUE(seq[*]._to) &gt; 1 <br>         FILTER SUM(seq[*].amount) &gt; 1000<br>         LET endDate = MAX(seq[*].date), seqLen = COUNT(seq)<br>         FILTER DATE_DIFF(beginDate, endDate, &quot;day&quot;) / seqLen &lt;= 5<br>         LET retransferCnt =<br>            COUNT(FOR t2 IN seq<br>               LET transferAmount = <br>                  SUM(FOR n, t3 IN OUTBOUND t2._to transactions<br>                         FILTER t3.date &gt;= t2.date <br>                           AND t3.date &lt;= DATE_ADD(t2.date, &quot;P30D&quot;)<br>                         RETURN t3.amount)<br>               FILTER transferAmount &gt; t2.amount * 0.9 RETURN 1)<br>         FILTER retransferCnt / seqLen &gt;= 0.8<br>         RETURN { lead: t1._key, seq: seq[*]._key })<br>    FOR p IN suspPatterns<br>       FOR t IN p.seq<br>          UPDATE t WITH { outboundLead : p.lead } IN transactions</pre><p>Next, we detect suspicious inbound money flow patterns in an analogous way and annotate the suspicious edges with the attribute inboundLead:</p><pre>FOR c IN clients<br>  LET patterns = <br>     (FOR acc1, t1 IN INBOUND c transactions<br>         FILTER t1.amount &gt; 100 AND !t1.atm<br>         LET beginDate = DATE_TIMESTAMP(t1.date)<br>         WINDOW beginDate WITH { preceding : &quot;P5D&quot;, following : &quot;P30D&quot; }<br>         AGGREGATE seq1 = UNIQUE(t1)<br>         LET seq = seq1[* FILTER ABS(CURRENT.amount - t1.amount) <br>                            &lt; t1.amount * 0.1]<br>         FILTER seq[0]._key == t1._key <br>         FILTER COUNT_UNIQUE(seq[*]._from) &gt; 1 <br>         FILTER SUM(seq[*].amount) &gt; 1000<br>         LET endDate = MAX(seq[*].date), seqLen = COUNT(seq)<br>         FILTER DATE_DIFF(beginDate, endDate, &quot;day&quot;) / seqLen &lt;= 5<br>         LET retransferCnt =<br>            COUNT(FOR t2 IN seq<br>               LET receivedAmount = <br>                  SUM(FOR n, t3 IN INBOUND t2._from transactions<br>                         FILTER t3.date &lt;= t2.date <br>                           AND t3.date &gt;= DATE_SUBTRACT(t2.date, &quot;P30D&quot;)<br>                         RETURN t3.amount)<br>               FILTER receivedAmount &gt; t2.amount * 0.9 RETURN 1)<br>         FILTER retransferCnt / seqLen &gt;= 0.8<br>         RETURN { lead: t1._key, seq: seq[*]._key })<br>    FOR p IN patterns<br>       FOR t IN p.seq<br>          UPDATE t WITH { inboundLead : p.lead } IN transactions</pre><h3>Visualizing the Money Flows</h3><p>Now we are ready to link the pieces together. Our goal is to run graph traversals for each money flow pattern from the initial source to the final destination:</p><ul><li>First, we identify the leading transactions of money flow patterns. Each money flow begins with a sequence of outbound transfers, which are not retransferred of received money. We collect such transactions into the leads array.</li><li>Each transaction t1 in the leads array begins one independent money flow pattern, which originates in the client t1._from. To collect the entire money flow, we perform a graph traversal starting from t1._from that follows suspicious transactions.</li><li>In each money flow path, we expect each transaction to be happening within a limited time frame after the previous one. By adding a corresponding condition in the PRUNE clause, we heavily limit the number of possible paths to be traversed.</li><li>We can reduce the combinatorial explosion of traversing various paths in the graph traversal by using only the lead edges to reach the next layer of the pattern.</li><li>PRUNE causes the pruned paths to be returned as part of the result. Since we are not interested in the pruned paths, we have to repeat a part of the PRUNE condition in the FILTER to get rid of them.</li></ul><pre>LET leads = (<br>   FOR t IN transactions<br>      FILTER t.outboundLead == t._key AND t.inboundLead == null<br>      LIMIT 1<br>      RETURN t)</pre><pre>FOR t1 IN leads<br>   LET client = t1._from<br>   LET leadKey = t1._key<br>   FOR n, e, p IN 2..6 OUTBOUND client transactions<br>      PRUNE LENGTH(p.edges) == 1 ? e.outboundLead != leadKey : <br>        (LENGTH(p.edges) &gt; 1 AND <br>           (e.date &lt; p.edges[LENGTH(p.edges)-2].date OR <br>            e.date &gt; DATE_ADD(p.edges[LENGTH(p.edges)-2].date, &quot;P30D&quot;) OR<br>            e.inboundLead != e._key)<br>        )<br>      FILTER e.inboundLead != null <br>        AND e.date &gt;= p.edges[LENGTH(p.edges)-2].date <br>        AND e.date &lt;= DATE_ADD(p.edges[LENGTH(p.edges)-2].date, &quot;P30D&quot;)<br>      RETURN p</pre><figure><img alt="Graph Traversal for Money flow" src="https://cdn-images-1.medium.com/max/924/0*6afauYyBRmaL5Xax" /><figcaption>Graph Traversal for Money flow</figcaption></figure><p>To see more patterns just increase the limit in the first subquery:</p><figure><img alt="Graph Traversal for Money flow" src="https://cdn-images-1.medium.com/max/1024/0*lTrOsfpwYKLHtdCR" /><figcaption>Graph Traversal for Money flow</figcaption></figure><h3>Conclusion</h3><p>As you can see now, the bad guys can get pretty tricky with how they move around money but not tricky enough to get past a well-done AQL query!</p><p>The ability to nest graph traversals while aggregating data all in a single query allows for efficiently gaining insights that may have otherwise been impossible to obtain and shows the power of the ArangoDB query language. Although we developed relatively complex queries, they are very declarative and constitute just a small fraction of the complexity that would be necessary to implement a corresponding algorithm using a programming language.</p><p>As promised, we covered a lot in this article, and if you would like to continue learning more about fraud detection, be sure to check out our Fraud Detection white paper:</p><p><a href="https://www.arangodb.com/resources/white-paper/fraud-detection/">Scalable Fraud Detection With Multi-Model</a></p><p>If you would like to see fraud detection in action at scale, we also have a fraud detection example guide available through Oasis, which you can get started with for free at <a href="http://cloud.arangodb.com/">cloud.arangodb.com</a>!</p><p>If you want to get hands-on with other ArangoDB learning content, be sure to check out our ever-growing library of <a href="https://github.com/arangodb/interactive_tutorials/">interactive tutorial notebooks</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d6bb206360a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/arangodb/detecting-complex-fraud-patterns-with-arangodb-4d6bb206360a">Detecting Complex Fraud Patterns with ArangoDB</a> was originally published in <a href="https://medium.com/arangodb">ArangoDB</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing the new ArangoDB Datasource for Apache Spark]]></title>
            <link>https://medium.com/arangodb/introducing-the-new-arangodb-datasource-for-apache-spark-da3df4d72028?source=rss-d2de8c290e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/da3df4d72028</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[imdb]]></category>
            <category><![CDATA[graph-database]]></category>
            <category><![CDATA[arangodb]]></category>
            <category><![CDATA[integration]]></category>
            <dc:creator><![CDATA[ArangoDB]]></dc:creator>
            <pubDate>Tue, 05 Apr 2022 12:33:45 GMT</pubDate>
            <atom:updated>2022-04-05T12:33:45.378Z</atom:updated>
            <content:encoded><![CDATA[<p>We are proud to announce the general availability of ArangoDB Datasource for Apache Spark: a new generation Spark connector for ArangoDB.</p><p>Nowadays, Apache Spark is one of the most popular analytics frameworks for large-scale data processing. It is designed to process in parallel data that is too large or complex for traditional databases, providing high performances by optimizing query execution, caching data in-memory and controlling the data distribution.</p><p>It exposes a pluggable DataSource API that can be implemented to allow interaction with external data providers in order to make the data accessible from Spark Dataframe and Spark SQL.</p><figure><img alt="Introducing the new ArangoDB Datasource for Apache Spark" src="https://cdn-images-1.medium.com/max/507/0*AEzgpV0cFxGsNryJ" /></figure><p>ArangoDB Spark Datasource is an implementation of <a href="https://jaceklaskowski.gitbooks.io/mastering-spark-sql/content/spark-sql-data-source-api-v2.html">DataSource API V2</a> and enables reading and writing from and to ArangoDB in batch execution mode.</p><p>Its typical use cases are:</p><ul><li>ETL (Extract, Transform, Load) data processing</li><li>data analytics and machine learning</li><li>uniform access to data from multiple datasources</li><li>interactive data exploration with Spark (SQL) shell</li><li>batch import and export</li><li>integration of ArangoDB in existing Spark workflows</li></ul><p>ArangoDB Datasource is available in different variants, each one compatible with different Spark versions. At the time of writing Spark 2.4 and 3.1 are supported. It works with all the <a href="https://www.arangodb.com/subscriptions/end-of-life-notice/">non-EOLed ArangoDB versions</a>. While internally implemented in Scala, it can be used from any Spark high-level API language, namely: Java, Scala, Python and R.</p><h3>Batch Reading</h3><p>The connector can read data from an ArangoDB <strong>collection</strong> or from a user provided <strong>AQL query</strong>.</p><p>When reading data from a <strong>collection</strong>, the reading job is split into many tasks, one for each shard in the source collection. The reading tasks can be executed in parallel in the Spark cluster and the resulting Spark DataFrame has the same number of partitions as the number of shards in the collection.</p><p>The reading requests are load balanced across all the available ArangoDB Coordinators. Each query hits only one DB-Server, the one holding the target shard, where it can be executed locally. This allows partitioning the read workload reflecting the ArangoDB collection sharding and distributing the reading tasks heavenly.</p><p>When reading data from an <strong>AQL query</strong>, the job cannot be partitioned and parallelized on the Spark side. This mode is recommended for executing queries in ArangoDB that produce only a small amount of data to be transferred to Spark.</p><h3>Mapping</h3><p>The reading schema can be provided by the user or inferred automatically by the connector prefetching a sample of the target collection data. Automatic schema inference can be useful in case of interactive data exploration, i.e. from the Spark Shell, nonetheless user provided schema is recommended for production use, since it allows for more detailed control.</p><p>Since ArangoDB does not internally enforce a schema for the stored data, it could be possible that the read data does not match the target Spark SQL type required by the read schema. In this case the connector will make its best effort to translate the data into the correct format, i.e. applying implicit deserialization casts. In case this is not possible, corrupt records will be handled according to one of the configurable strategies: drop them, put them in a dedicated column or make the entire task fail.</p><p>The mapping implementation leverages the Spark JSON mapping module, which is based on Jackson Databind API. In this way the connector reuses the state-of-the-art implementation of JSON mapping already existing in the Spark SQL project, which is used by the Spark JSON files datasource. Velocypack support has been plugged in by using <a href="https://github.com/arangodb/jackson-dataformat-velocypack">jackson-dataformat-velocypack</a>, a Jackson Databind API implementation for Velocypack, therefore compatible to be used from within the JSON mapping module. Thanks to this approach, the mapping behavior is identical to the Spark JSON files one and includes support for all Spark SQL types.</p><h3>Filter and Projection Pushdown</h3><p>Given the lazy nature of Dataframe transformations, Spark can optimize query execution with an end to end awareness, therefore requesting to the datasources only the subset of data strictly needed to perform the ongoing job. This helps to reduce the amount of transferred data and is achieved by pushing down to the datasource filter predicates and column projections, so that only the required fields of the matching documents will be returned. ArangoDB Datasource can dynamically generate AQL queries that contain the required filters and projections.</p><h3>Batch Write</h3><p>The connector can be used to save Dataframe records as documents in an ArangoDB collection. Write tasks are load balanced across the available ArangoDB Coordinators and the saved data is distributed in the ArangoDB cluster according to the target collection sharding strategy, so potentially differently from the Spark DataFrame partitioning. The amount of parallelizable writing tasks is equal to the number of partitions in the original Spark Dataframe.</p><p>The connector can optionally create new collections, append data to existing ones or overwrite them. Conflicts arising from documents with the same id can be handled with several strategies: updating, replacing, ignoring or throwing exceptions. The Spark Datasource API is designed to offer a transactional behavior of writing tasks. This means that Spark supervises all the writing tasks and makes its best effort to make sure that either all of them are successful or the entire job is aborted, thus trying to roll back the already executed tasks.</p><p>Depending on the configuration, the connector can fulfill the transactional API and offer more resilient behavior. For example if the provided configuration allows for idempotent operations, the connector can and retry failing requests or entire tasks, and be compatible with speculative executions of tasks.</p><p>In case of connectivity problems connections can be failed over to a different Coordinator.</p><h3>Usage Example</h3><p>Next we will see how to use the connector to write and read to and from an ArangoDB collection. All the code snippets are extracted from the official <a href="https://github.com/arangodb/arangodb-spark-datasource/tree/main/demo">ArangoDB Datasource Demo</a> and based on the <a href="https://github.com/arangodb/example-datasets/tree/master/Graphs/IMDB">IMDB example dataset</a>.</p><p>The connector can be imported from Maven Central using the coordinates:</p><p>com.arangodb:arangodb-spark-datasource-3.1_2.12:&lt;version&gt;</p><p>Given an existing Spark Dataframe moviesDF we can save it to an ArangoDB collection movies in this way:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7xUFTkpBdjG1MMASOD16NA.png" /></figure><p>where we perform these steps:</p><ol><li>specify the datasource id</li><li>provide configuration parameters to establish a connection to ArangoDB</li><li>provide configuration parameters for the writing job</li><li>perform the writing job</li></ol><p>This will also create the collection if it does not exist already.</p><p>Now we can read back the data saved in the movies collection. First we create a schema for the data going to be read:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_BQ6kxX-DOII5ZvzwkSl3Q.png" /></figure><p>Then we create a Spark SQL view backed by the ArangoDB collection movies:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_u6dTnee-J4o_-ikt1bVOA.png" /></figure><p>In the snippet above we performed the following steps:</p><ol><li>specify the datasource id</li><li>provide configuration parameters to establish a connection to ArangoDB</li><li>provide configuration parameters to identify the source collection</li><li>specify the collection schema</li><li>create a Dataframe</li><li>create the Spark SQL view named movies</li></ol><p>From now on movies is visible to our Spark session, so we can reference it from Spark SQL queries, for example, we can read some data from it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vsO2z0VRUS0ri7tg2zNxvw.png" /></figure><p>This will print to console:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pkz2gNuGGNc4a0z0dUeD8g.png" /></figure><p>Let’s perform some filtering and column selection to find all action or animation movies about dinosaurs and showing only some fields (title, releaseDate, genre):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jFZPtCfb3Tlmhxxfpz0SWQ.png" /></figure><p>This will show:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qk-tsnwx_gcBM3Ao4uU9zA.png" /></figure><p>As we can see, we get in the console logs a feedback about the pushed down filters:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mpQErKYGn-QRnAH2KP72Sg.png" /></figure><p>and about the generated AQL:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qnQ42R9O3fkNBHo7p8hZVg.png" /></figure><p>The log above is printed for every query task into which the job is partitioned, therefore so many times as the number of shards of the movie collection. Each query is performed against a different source collection shard.</p><p>Next we would like to perform some aggregate calculation on the data, for example getting the count of movies by genre. Since Spark Dataframe API does not push down aggregate clauses (before Spark 3.2), we can express it as a user AQL query that will be executed on the db side:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jHEe9Y3AVgIcb7bXIYgHiQ.png" /></figure><p>which will show:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6JftnNrtiXf_ATtU7n2npQ.png" /></figure><h3><strong>Conclusion</strong></h3><p>In this article, we saw an architectural overview of ArangoDB Datasource for Apache Spark and some introductory steps to get started with it. Further usage examples can be found in the<a href="https://github.com/arangodb/arangodb-spark-datasource/tree/main/demo"> ArangoDB Datasource Demo</a>project.</p><p>The Spark Datasource V2 API is still evolving and being extended at every Spark new release, in Spark 3.2 aggregate push-down capability has been added and many improvements are still yet to come (<a href="https://issues.apache.org/jira/browse/SPARK-22386">Data Source V2 improvements</a>).</p><p>In future releases of ArangoDB Datasource, we can expect to see support for Spark 3.2 API, streaming execution mode, and better observability, i.e. tasks metrics.</p><h3>Reference links</h3><ul><li><a href="https://github.com/arangodb/arangodb-spark-datasource">GitHub page</a></li><li><a href="https://github.com/arangodb/arangodb-spark-datasource/issues">Issue Tracker</a></li><li><a href="https://www.arangodb.com/docs/stable/drivers/spark-connector-new.html">Documentation</a></li><li><a href="https://github.com/arangodb/arangodb-spark-datasource/tree/main/demo">Demo</a></li></ul><h3><strong>Author</strong></h3><p><a href="https://www.arangodb.com/author/michele/">Michele Rastelli</a></p><p>Michele is a software engineer with experience in algorithms, web architectures, and data modeling. He has been working as a technical leader for web startups, designing and implementing scalable and resilient microservices applications. He currently works at ArangoDB on the development of the Java driver and frameworks integrations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/184/1*71h3P0zgde7xa-srrj37eQ.png" /><figcaption>Michele Rastelli</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da3df4d72028" width="1" height="1" alt=""><hr><p><a href="https://medium.com/arangodb/introducing-the-new-arangodb-datasource-for-apache-spark-da3df4d72028">Introducing the new ArangoDB Datasource for Apache Spark</a> was originally published in <a href="https://medium.com/arangodb">ArangoDB</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing the ArangoDB-DGL Adapter]]></title>
            <link>https://medium.com/arangodb/introducing-the-arangodb-dgl-adapter-4fadf3aafbfd?source=rss-d2de8c290e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/4fadf3aafbfd</guid>
            <category><![CDATA[graph-database]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[integration]]></category>
            <category><![CDATA[arangodb]]></category>
            <category><![CDATA[graph]]></category>
            <dc:creator><![CDATA[ArangoDB]]></dc:creator>
            <pubDate>Fri, 04 Mar 2022 09:46:22 GMT</pubDate>
            <atom:updated>2022-03-04T09:46:22.385Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Introducing the ArangoDB-DGL Adapter" src="https://cdn-images-1.medium.com/max/1024/1*yHDDlk0VAUpyWQzdwd84AQ.png" /><figcaption>Introducing the ArangoDB-DGL Adapter</figcaption></figure><p>We are proud to announce the GA 1.0 release of the <a href="https://pypi.org/project/adbdgl-adapter/">ArangoDB-DGL Adapter</a>!</p><p><em>The ArangoDB-DGL Adapter exports Graphs from ArangoDB, a multi-model Graph Database, into Deep Graph Library (DGL), a python package for graph neural networks, and vice-versa.</em></p><p>On December 30th, 2021, we introduced to the ArangoML community our first release of the DGL Adapter for ArangoDB. We worked closely with our existing <a href="https://pypi.org/project/adbnx-adapter/">ArangoDB-NetworkX Adapter</a> implementation to aim for a consistent UX across our (growing) Adapter Family. You can expect the same developer-friendly options, along with a helpful getting-started guide via Google Colab. And as always, it is open source!</p><p>This blog post will serve as a walkthrough of the ArangoDB-DGL Adapter, via its official Jupyter Notebook.</p><p>We will cover the following use cases:</p><ol><li>ArangoDB to DGL</li></ol><blockquote>Via an ArangoDB graph</blockquote><blockquote>Via a set of ArangoDB collections</blockquote><blockquote>Via a user-defined metagraph</blockquote><blockquote>Unique cases in attribute transfer</blockquote><p>2. DGL to ArangoDB</p><blockquote>Homogeneous graphs</blockquote><blockquote>Heterogeneous graphs</blockquote><blockquote>Unique cases in attribute transfer</blockquote><p><a href="https://github.com/arangoml/dgl-adapter/blob/master/examples/outputs/ArangoDB_DGL_Adapter_output.ipynb">Check it out on github</a></p><blockquote>ArangoDB DGL Adapter Getting Started Guide</blockquote><p><a href="https://colab.research.google.com/github/arangoml/dgl-adapter/blob/master/examples/ArangoDB_DGL_Adapter.ipynb">Google Colaboratory</a></p><h3>Author</h3><p><a href="https://www.arangodb.com/author/anthony/">Anthony Mahanna</a></p><p>Anthony is an Honours Computer Science student at the University of Ottawa, Canada. He first discovered ArangoDB’s multi-model services while working on his image repository side project. After presenting his side project in an ArangoDB Community Pioneer session, Anthony transitioned to working with the Core &amp; ML teams as an SWE intern.</p><figure><img alt="Anthony Mahanna" src="https://cdn-images-1.medium.com/max/320/1*k5plUamlcQ8yukyQ6b7jQw.jpeg" /><figcaption>Anthony Mahanna</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4fadf3aafbfd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/arangodb/introducing-the-arangodb-dgl-adapter-4fadf3aafbfd">Introducing the ArangoDB-DGL Adapter</a> was originally published in <a href="https://medium.com/arangodb">ArangoDB</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing the ArangoDB-NetworkX Adapter]]></title>
            <link>https://medium.com/arangodb/introducing-the-arangodb-networkx-adapter-6f55b7a6f20c?source=rss-d2de8c290e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f55b7a6f20c</guid>
            <category><![CDATA[networkx]]></category>
            <category><![CDATA[integration]]></category>
            <category><![CDATA[arangodb]]></category>
            <category><![CDATA[graph-database]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[ArangoDB]]></dc:creator>
            <pubDate>Thu, 24 Feb 2022 13:32:45 GMT</pubDate>
            <atom:updated>2022-02-24T13:32:45.176Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Introducing the ArangoDB-NetworkX Adapter" src="https://cdn-images-1.medium.com/max/1024/1*2-h7WyLyVqOgcH6AdjA3Gw.png" /></figure><p>We are proud to announce the GA 3.0 release of the <a href="https://pypi.org/project/adbnx-adapter/">ArangoDB-NetworkX Adapter</a>!</p><p><em>The ArangoDB-Networkx Adapter exports Graphs from ArangoDB, a multi-model Graph Database, into NetworkX, the swiss army knife for graph analysis with python, and vice-versa.</em></p><p>Back in November 2021, we (quietly) released its 1.0 distribution, which overhauled the adapter in its entirety. We refactored its existing feature of converting ArangoDB graphs to NetworkX, and introduced the ability to convert NetworkX graphs to ArangoDB, via a range of developer-friendly options. It’s also open source!</p><p>This latest major release complements previous overhaul &amp; restoration changes by optimizing adapter functionality, polishing its codebase, and improving documentation. For more information on what’s changed between the 2.0 &amp; 3.0 releases, see <a href="https://github.com/arangoml/networkx-adapter/releases/tag/3.0.0">networkx-adapter/releases/tag/3.0.0</a>.</p><p>This blog post will serve as a walkthrough of the ArangoDB-NetworkX Adapter, via its official Jupyter Notebook.</p><p>We will cover the following use cases:</p><ol><li>ArangoDB to NetworkX</li></ol><blockquote>Via an ArangoDB graph</blockquote><blockquote>Via a set of ArangoDB collections</blockquote><blockquote>Via a user-defined metagraph</blockquote><p>2. NetworkX to ArangoDB</p><blockquote>Homogeneous graphs</blockquote><blockquote>Heterogeneous graphs</blockquote><p>3. Full Cycles</p><blockquote>ArangoDB → NetworkX → ArangoDB</blockquote><blockquote>NetworkX → ArangoDB → NetworkX</blockquote><p><a href="https://github.com/arangoml/networkx-adapter/blob/master/examples/outputs/ArangoDB_NetworkX_Adapter_output.ipynb">Check it out on github</a></p><blockquote>ArangoDB NetworkX Adapter Getting Started Guide</blockquote><p><a href="https://colab.research.google.com/github/arangoml/networkx-adapter/blob/master/examples/ArangoDB_NetworkX_Adapter.ipynb">Google Colaboratory</a></p><h3>Author</h3><p><a href="https://www.arangodb.com/author/anthony/">Anthony Mahanna</a></p><p>Anthony is an Honours Computer Science student at the University of Ottawa, Canada. He first discovered ArangoDB’s multi-model services while working on his image repository side project. After presenting his side project in an ArangoDB Community Pioneer session, Anthony transitioned to working with the Core &amp; ML teams as an SWE intern.</p><figure><img alt="Image- Anthony Mahanna" src="https://cdn-images-1.medium.com/max/320/1*SJteZeDGlO3tQUL5kvOn7A.jpeg" /><figcaption>Image- Anthony Mahanna</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f55b7a6f20c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/arangodb/introducing-the-arangodb-networkx-adapter-6f55b7a6f20c">Introducing the ArangoDB-NetworkX Adapter</a> was originally published in <a href="https://medium.com/arangodb">ArangoDB</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>