<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Arshdeep Kaur on Medium]]></title>
        <description><![CDATA[Stories by Arshdeep Kaur on Medium]]></description>
        <link>https://medium.com/@arshdeepkaur?source=rss-2094fd6f5143------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*MM7chV_b42iIypGTaNjp2w.jpeg</url>
            <title>Stories by Arshdeep Kaur on Medium</title>
            <link>https://medium.com/@arshdeepkaur?source=rss-2094fd6f5143------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 01:53:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@arshdeepkaur/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Knowledge Graphs for RAG using Neo4j — chat with SEC data — Part 1]]></title>
            <link>https://arshdeepkaur.medium.com/knowledge-graphs-for-rag-using-neo4j-chat-with-sec-data-part-1-f1c37985ee10?source=rss-2094fd6f5143------2</link>
            <guid isPermaLink="false">https://medium.com/p/f1c37985ee10</guid>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[neo4j]]></category>
            <category><![CDATA[cypher]]></category>
            <category><![CDATA[knowledge-graph]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Arshdeep Kaur]]></dc:creator>
            <pubDate>Thu, 09 May 2024 21:03:11 GMT</pubDate>
            <atom:updated>2024-05-09T21:03:11.359Z</atom:updated>
            <content:encoded><![CDATA[<h3>Knowledge Graphs for RAG using Neo4j — chat with SEC data — Part 1</h3><h3>CHAT WITH SEC — PLAN OF ACTION</h3><ol><li>We will work with a single 10-K <strong>form</strong> for a company called NetApp.<br>(Sections/items <strong>1</strong>, <strong>1a</strong>, <strong>7</strong>, <strong>7a</strong> were extracted from the10-K file and converted to <a href="https://github.com/neo4j-examples/sec-edgar-notebooks/tree/main/data/single/form10k">.json object</a>. )</li><li>Split form sections (i.e. items: 1, 1a, 7, 7a) into <strong>chunks</strong> using a Langchain text splitter.</li><li>Create a graph where each <strong>chunk</strong> is a <strong>node</strong>. Add chunk <strong>metadata</strong> as <strong>properties</strong>.</li><li>Create a <strong>vector index</strong>.</li><li>Create a text <strong>embedding vector for each chunk</strong> and <strong>populate</strong> the index.</li></ol><h3>CONSTRUCTING A KNOWLEDGE GRAPH FROM TEXT</h3><ol><li>For each specified item (section) in the Form 10-K filing (item1, item1a, item7, item7a), the function performs the following steps:</li></ol><ul><li>Retrieves the text data for the <strong>item</strong> from the JSON object.</li><li>Splits the text data into chunks using a text_splitter.split_text() function.</li><li>Initializes a sequence ID (chunk_seq_id) for the chunks.</li><li>Iterates through the first 20 chunks of the item’s text data. (We limit the number of chunks in each section to 20 to speed things up.)</li><li>Constructs a dictionary representing each chunk along with its associated metadata.</li><li>To summarize: each item (‘item1’ ’item1a’ ’item7&#39; ’item7a&#39;) is split into 20 chunks of 2000 characters each and assigned a unique chunkId. I.e. a <strong>chunkId is uniquely identified by </strong><strong>form_id, </strong><strong>item, </strong><strong>chunk_seq_id within the </strong><strong>item</strong>.</li></ul><pre>def split_form10k_data_from_file(file):<br>    chunks_with_metadata = [] # use this to accumlate chunk records<br>    file_as_object = json.load(open(file)) # open the json file<br>    for item in [&#39;item1&#39;,&#39;item1a&#39;,&#39;item7&#39;,&#39;item7a&#39;]: # pull these keys from the json<br>        print(f&#39;Processing {item} from {file}&#39;) <br>        item_text = file_as_object[item] # grab the text of the item<br>        item_text_chunks = text_splitter.split_text(item_text) # split the text into chunks<br>        chunk_seq_id = 0<br>        for chunk in item_text_chunks[:20]: # only take the first 20 chunks<br>            form_id = file[file.rindex(&#39;/&#39;) + 1:file.rindex(&#39;.&#39;)] # extract form id from file name<br>            # finally, construct a record with metadata and the chunk text<br>            chunks_with_metadata.append({<br>                &#39;text&#39;: chunk, <br>                # metadata from looping...<br>                &#39;f10kItem&#39;: item,<br>                &#39;chunkSeqId&#39;: chunk_seq_id,<br>                # constructed metadata...<br>                &#39;formId&#39;: f&#39;{form_id}&#39;, # pulled from the filename<br>                &#39;chunkId&#39;: f&#39;{form_id}-{item}-chunk{chunk_seq_id:04d}&#39;,<br>                # metadata from file...<br>                &#39;names&#39;: file_as_object[&#39;names&#39;],<br>                &#39;cik&#39;: file_as_object[&#39;cik&#39;],<br>                &#39;cusip6&#39;: file_as_object[&#39;cusip6&#39;],<br>                &#39;source&#39;: file_as_object[&#39;source&#39;],<br>            })<br>            chunk_seq_id += 1<br>        print(f&#39;\tSplit into {chunk_seq_id} chunks&#39;)<br>    return chunks_with_metadata</pre><p>2. Next, we <strong>create (</strong>upsert: update/insert i.e. if the match fails<strong>) </strong>graph nodes using chunks.</p><pre>merge_chunk_node_query = &quot;&quot;&quot;<br>MERGE(mergedChunk:Chunk {chunkId: $chunkParam.chunkId})<br>    ON CREATE SET <br>        mergedChunk.names = $chunkParam.names,<br>        mergedChunk.formId = $chunkParam.formId, <br>        mergedChunk.cik = $chunkParam.cik, <br>        mergedChunk.cusip6 = $chunkParam.cusip6, <br>        mergedChunk.source = $chunkParam.source, <br>        mergedChunk.f10kItem = $chunkParam.f10kItem, <br>        mergedChunk.chunkSeqId = $chunkParam.chunkSeqId, <br>        mergedChunk.text = $chunkParam.text<br>RETURN mergedChunk<br>&quot;&quot;&quot;</pre><p>3. Loop through the chunks and call function: merge_chunk_node_query, and create nodes.</p><pre>node_count = 0<br>for chunk in first_file_chunks:<br>    print(f&quot;Creating `:Chunk` node for chunk ID {chunk[&#39;chunkId&#39;]}&quot;)<br>    kg.query(merge_chunk_node_query, <br>            params={<br>                &#39;chunkParam&#39;: chunk<br>            })<br>    node_count += 1<br>print(f&quot;Created {node_count} nodes&quot;)</pre><p>4. <strong>23</strong> nodes get created for thesechunkIds:</p><ul><li>20 nodes for item1: 0000950170–23–027948-item1-chunk0000— — 0000950170–23–027948-item1-chunk0019</li><li>1 node for each of the other items item1a, item7, item7a:<br> 0000950170–23–027948-item1a-chunk0000, 0000950170–23–027948-item7-chunk0000, 0000950170–23–027948-item7a-chunk0000.</li></ul><p>5. Sanity check: verify 23 nodes:</p><pre>kg.query(&quot;&quot;&quot;<br>         MATCH (n)<br>         RETURN count(n) as nodeCount<br>         &quot;&quot;&quot;)</pre><p>Output</p><pre>[{&#39;nodeCount&#39;: 23}]</pre><p>So far so good. ✅</p><h3>VECTOR INDEX</h3><ol><li>Create vector index</li></ol><pre>kg.query(&quot;&quot;&quot;<br>         CREATE VECTOR INDEX `form_10k_chunks` IF NOT EXISTS<br>          FOR (c:Chunk) ON (c.textEmbedding) <br>          OPTIONS { indexConfig: {<br>            `vector.dimensions`: 1536,<br>            `vector.similarity_function`: &#39;cosine&#39;    <br>         }}<br>&quot;&quot;&quot;)</pre><p>Make sure the vector index is online</p><pre>kg.query(&quot;SHOW INDEXES&quot;)</pre><p>Output: ‘state’: ‘ONLINE’,✅</p><pre>[{&#39;id&#39;: 5,<br>  &#39;name&#39;: &#39;form_10k_chunks&#39;,<br>  &#39;state&#39;: &#39;ONLINE&#39;,<br>  &#39;populationPercent&#39;: 100.0,<br>  &#39;type&#39;: &#39;VECTOR&#39;,<br>  &#39;entityType&#39;: &#39;NODE&#39;,<br>  &#39;labelsOrTypes&#39;: [&#39;Chunk&#39;],<br>  &#39;properties&#39;: [&#39;textEmbedding&#39;],<br>  &#39;indexProvider&#39;: &#39;vector-1.0&#39;,<br>  &#39;owningConstraint&#39;: None,<br>  &#39;lastRead&#39;: None,<br>  &#39;readCount&#39;: None}]</pre><p>2. Calculate embedding vectors for chunks and populate index. This query calculates the embedding vector and stores it as a property called textEmbedding on each Chunk node.</p><pre>kg.query(&quot;&quot;&quot;<br>    MATCH (chunk:Chunk) WHERE chunk.textEmbedding IS NULL<br>    WITH chunk, genai.vector.encode(<br>      chunk.text, <br>      &quot;OpenAI&quot;, <br>      {<br>        token: $openAiApiKey, <br>        endpoint: $openAiEndpoint<br>      }) AS vector<br>    CALL db.create.setNodeVectorProperty(chunk, &quot;textEmbedding&quot;, vector)<br>    &quot;&quot;&quot;, <br>    params={&quot;openAiApiKey&quot;:OPENAI_API_KEY, &quot;openAiEndpoint&quot;: OPENAI_ENDPOINT} )</pre><p>3. Create a generic function that takes a question as an input and performs <strong>similarity search</strong>:</p><pre>def neo4j_vector_search(question):<br>  &quot;&quot;&quot;Search for similar nodes using the Neo4j vector index&quot;&quot;&quot;<br>  vector_search_query = &quot;&quot;&quot;<br>    WITH genai.vector.encode(<br>      $question, <br>      &quot;OpenAI&quot;, <br>      {<br>        token: $openAiApiKey,<br>        endpoint: $openAiEndpoint<br>      }) AS question_embedding<br>    CALL db.index.vector.queryNodes($index_name, $top_k, question_embedding) yield node, score<br>    RETURN score, node.text AS text<br>  &quot;&quot;&quot;<br>  similar = kg.query(vector_search_query, <br>                     params={<br>                      &#39;question&#39;: question, <br>                      &#39;openAiApiKey&#39;:OPENAI_API_KEY,<br>                      &#39;openAiEndpoint&#39;: OPENAI_ENDPOINT,<br>                      &#39;index_name&#39;:VECTOR_INDEX_NAME, <br>                      &#39;top_k&#39;: 10})<br>  return similar</pre><p>4. Perform similarity search</p><pre>search_results = neo4j_vector_search(<br>    &#39;In a single sentence, tell me about Netapp.&#39;<br>)</pre><p>Output:</p><pre>search_results[0]<br><br>{&#39;score&#39;: 0.9358915090560913,<br> &#39;text&#39;: &#39;&gt;Item 1.  \nBusiness\n\n\nOverview\n\n\nNetApp, Inc. (NetApp, we, us or the Company) is a global cloud-led, data-centric software company. We were incorporated in 1992 and are headquartered in San Jose, California. Building on more than three decades of innovation, we give customers the freedom to manage applications and data across hybrid multicloud environments. Our portfolio of cloud services, and storage infrastructure, powered by intelligent data management software, enables applications to run faster, more reliably, and more securely, all at a lower cost.\n\n\nOur opportunity is defined by the durable megatrends of data-driven digital and cloud transformations. NetApp helps organizations meet the complexities created by rapid data and cloud growth, multi-cloud management, and the adoption of next-generation technologies, such as AI, Kubernetes, and modern databases. Our modern approach to hybrid, multicloud infrastructure and data management, which we term ‘evolved cloud’, provides customers the ability to leverage data across their entire estate with simplicity, security, and sustainability which increases our relevance and value to our customers.\n\n\nIn an evolved cloud state, the cloud is fully integrated into an organization’s architecture and operations. Data centers and clouds are seamlessly united and hybrid multicloud operations are simplified, with consistency and observability across environments. The key benefits NetApp brings to an organization’s hybrid multicloud environment are:\n\n\n•\nOperational simplicity: NetApp’s use of open source, open architectures and APIs, microservices, and common capabilities and data services facilitate the creation of applications that can run anywhere.\n\n\n•\nFlexibility and consistency: NetApp makes moving data and applications between environments seamless through a common storage foundation across on-premises and multicloud environments.&#39;}</pre><p>So far we have only performed Vector Search. To create a Chatbot (question &amp; answering), we will build a <strong>RAG</strong> system using LangChain.👇</p><p>5. Set up a LangChain RAG workflow to chat with the form using RetrievalQAWithSourcesChain to carry out question answering. Under the hood, Neo4j will use Cypher to perform vector <em>similarity</em> searches.<br>chain_type=”stuff”, means we are doing prompt stuffing — i.e. combine everything and send to LLM.</p><pre>neo4j_vector_store = Neo4jVector.from_existing_graph(<br>    embedding=OpenAIEmbeddings(),<br>    url=NEO4J_URI,<br>    username=NEO4J_USERNAME,<br>    password=NEO4J_PASSWORD,<br>    index_name=VECTOR_INDEX_NAME,<br>    node_label=VECTOR_NODE_LABEL,<br>    text_node_properties=[VECTOR_SOURCE_PROPERTY],<br>    embedding_node_property=VECTOR_EMBEDDING_PROPERTY,<br>)<br><br>retriever = neo4j_vector_store.as_retriever()<br><br>chain = RetrievalQAWithSourcesChain.from_chain_type(<br>    ChatOpenAI(temperature=0), <br>    chain_type=&quot;stuff&quot;, <br>    retriever=retriever<br>)<br><br>def prettychain(question: str) -&gt; str:<br>    &quot;&quot;&quot;Pretty print the chain&#39;s response to a question&quot;&quot;&quot;<br>    response = chain({&quot;question&quot;: question},<br>        return_only_outputs=True,)<br>    print(textwrap.fill(response[&#39;answer&#39;], 60))</pre><p>6. Ask question</p><pre>question = &quot;What is Netapp&#39;s primary business?&quot;<br>prettychain(question)</pre><p>Output</p><pre>NetApp&#39;s primary business is enterprise storage and data<br>management, cloud storage, and cloud operations.</pre><h3>ADDING RELATIONSHIPS TO THE SEC KNOWLEDGE GRAPH</h3><p>Now we will add relationships to preserve the structure of the documents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*zR9JTC9Ljp-tpP7Elvd11g.png" /></figure><ol><li>Create form node: Form</li></ol><pre>cypher = &quot;&quot;&quot;<br>    MERGE (f:Form {formId: $formInfoParam.formId })<br>      ON CREATE <br>        SET f.names = $formInfoParam.names<br>        SET f.source = $formInfoParam.source<br>        SET f.cik = $formInfoParam.cik<br>        SET f.cusip6 = $formInfoParam.cusip6<br>&quot;&quot;&quot;<br><br>kg.query(cypher, params={&#39;formInfoParam&#39;: form_info})</pre><p>Sanity check</p><pre>kg.query(&quot;MATCH (f:Form) RETURN count(f) as formCount&quot;)</pre><pre>[{&#39;formCount&#39;: 1}]</pre><p>✅</p><p>Now we will connect everything together. <strong>Our goal is to add relationships to improve the context around each chunk</strong>. The result will reflect the original structure of the document.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*KYt9eRzanLH9GTXSXBO1PQ.png" /><figcaption>Connect form and chunk nodes</figcaption></figure><p>2. Write query that 1. retrieves chunks from the <strong>same</strong> <strong>form</strong>, <strong>same</strong> <strong>section</strong> (e.g. item1), 2. uses collect to collect everything in a list. <br>Loop through all sections and call this query to link these chunks using relationship: NEXT.</p><pre>cypher = &quot;&quot;&quot;<br>  MATCH (from_same_section:Chunk)<br>  WHERE from_same_section.formId = $formIdParam<br>    AND from_same_section.f10kItem = $f10kItemParam<br>  WITH from_same_section<br>    ORDER BY from_same_section.chunkSeqId ASC<br>  WITH collect(from_same_section) as section_chunk_list<br>    CALL apoc.nodes.link(<br>        section_chunk_list, <br>        &quot;NEXT&quot;, <br>        {avoidDuplicates: true}<br>    ) <br>  RETURN size(section_chunk_list)<br>&quot;&quot;&quot;<br><br>for form10kItemName in [&#39;item1&#39;, &#39;item1a&#39;, &#39;item7&#39;, &#39;item7a&#39;]:<br>  kg.query(cypher, params={&#39;formIdParam&#39;:form_info[&#39;formId&#39;], <br>                           &#39;f10kItemParam&#39;: form10kItemName})</pre><p>3. Connect chunks to their parent form with a relationship: PART_OF.</p><pre>cypher = &quot;&quot;&quot;<br>  MATCH (c:Chunk), (f:Form)<br>    WHERE c.formId = f.formId<br>  MERGE (c)-[newRelationship:PART_OF]-&gt;(f)<br>  RETURN count(newRelationship)<br>&quot;&quot;&quot;<br><br>kg.query(cypher)</pre><p>4. Create a relationship: SECTION on first chunk of each section</p><pre>cypher = &quot;&quot;&quot;<br>  MATCH (first:Chunk), (f:Form)<br>  WHERE first.formId = f.formId<br>    AND first.chunkSeqId = 0<br>  WITH first, f<br>    MERGE (f)-[r:SECTION {f10kItem: first.f10kItem}]-&gt;(first)<br>  RETURN count(r)<br>&quot;&quot;&quot;<br><br>kg.query(cypher)</pre><p>5. Play with queries. E.g. Return a window of three chunks:</p><pre>cypher = &quot;&quot;&quot;<br>    MATCH (c1:Chunk)-[:NEXT]-&gt;(c2:Chunk)-[:NEXT]-&gt;(c3:Chunk) <br>        WHERE c2.chunkId = $chunkIdParam<br>    RETURN c1.chunkId, c2.chunkId, c3.chunkId<br>    &quot;&quot;&quot;<br>​<br>kg.query(cypher,<br>         params={&#39;chunkIdParam&#39;: next_chunk_info[&#39;chunkId&#39;]})</pre><pre>[{&#39;c1.chunkId&#39;: &#39;0000950170-23-027948-item1-chunk0000&#39;,<br>  &#39;c2.chunkId&#39;: &#39;0000950170-23-027948-item1-chunk0001&#39;,<br>  &#39;c3.chunkId&#39;: &#39;0000950170-23-027948-item1-chunk0002&#39;}]</pre><p>6. Modify NEXT relationship to have variable length. And return the window with the longest path.</p><pre>cypher = &quot;&quot;&quot;<br>  MATCH window=<br>      (:Chunk)-[:NEXT*0..1]-&gt;(c:Chunk)-[:NEXT*0..1]-&gt;(:Chunk)<br>    WHERE c.chunkId = $chunkIdParam<br>  WITH window as longestChunkWindow <br>      ORDER BY length(window) DESC LIMIT 1<br>  RETURN length(longestChunkWindow)<br>  &quot;&quot;&quot;<br><br>kg.query(cypher,<br>         params={&#39;chunkIdParam&#39;: first_chunk_info[&#39;chunkId&#39;]})</pre><pre>[{&#39;length(longestChunkWindow)&#39;: 1}]</pre><p>Now we have learnt how to modify the context around a chunk using a window.</p><p>7. Create chatbot: define a <strong>window retrieval query</strong> to get consecutive chunks, create a vector store that retrieves from retrieval_query_window</p><pre>retrieval_query_window = &quot;&quot;&quot;<br>MATCH window=<br>    (:Chunk)-[:NEXT*0..1]-&gt;(node)-[:NEXT*0..1]-&gt;(:Chunk)<br>WITH node, score, window as longestWindow <br>  ORDER BY length(window) DESC LIMIT 1<br>WITH nodes(longestWindow) as chunkList, node, score<br>  UNWIND chunkList as chunkRows<br>WITH collect(chunkRows.text) as textList, node, score<br>RETURN apoc.text.join(textList, &quot; \n &quot;) as text,<br>    score,<br>    node {.source} AS metadata<br>&quot;&quot;&quot;<br><br># Create a vector store that retrieves from retrieval_query_window<br>vector_store_window = Neo4jVector.from_existing_index(<br>    embedding=OpenAIEmbeddings(),<br>    url=NEO4J_URI,<br>    username=NEO4J_USERNAME,<br>    password=NEO4J_PASSWORD,<br>    database=&quot;neo4j&quot;,<br>    index_name=VECTOR_INDEX_NAME,<br>    text_node_property=VECTOR_SOURCE_PROPERTY,<br>    retrieval_query=retrieval_query_window<br>)<br><br># Create a retriever from the vector store<br>retriever_window = vector_store_window.as_retriever()<br><br># Create a chatbot Question &amp; Answer chain from the retriever<br>chain_window = RetrievalQAWithSourcesChain.from_chain_type(<br>    ChatOpenAI(temperature=0), <br>    chain_type=&quot;stuff&quot;, <br>    retriever=retriever_window<br>)</pre><p>For comparison create a regular vector store that retrieves a single node:</p><pre># Create a regular vector store that retrieves a single node<br>neo4j_vector_store = Neo4jVector.from_existing_graph(<br>    embedding=OpenAIEmbeddings(),<br>    url=NEO4J_URI,<br>    username=NEO4J_USERNAME,<br>    password=NEO4J_PASSWORD,<br>    index_name=VECTOR_INDEX_NAME,<br>    node_label=VECTOR_NODE_LABEL,<br>    text_node_properties=[VECTOR_SOURCE_PROPERTY],<br>    embedding_node_property=VECTOR_EMBEDDING_PROPERTY,<br>)<br># Create a retriever from the vector store<br>windowless_retriever = neo4j_vector_store.as_retriever()<br><br># Create a chatbot Question &amp; Answer chain from the retriever<br>windowless_chain = RetrievalQAWithSourcesChain.from_chain_type(<br>    ChatOpenAI(temperature=0), <br>    chain_type=&quot;stuff&quot;, <br>    retriever=windowless_retriever<br>)</pre><p>Compare the 2 chains: windowless_chain, chain_window</p><pre>question = &quot;In a single sentence, tell me about Netapp&#39;s business.&quot;</pre><pre>answer = windowless_chain(<br>    {&quot;question&quot;: question},<br>    return_only_outputs=True,<br>)<br>print(textwrap.fill(answer[&quot;answer&quot;]))</pre><pre>NetApp is a global cloud-led, data-centric software company that<br>provides customers with the freedom to manage applications and data<br>across hybrid multicloud environments, focusing primarily on<br>enterprise storage and data management, cloud storage, and cloud<br>operations markets.</pre><pre>answer = chain_window(<br>    {&quot;question&quot;: question},<br>    return_only_outputs=True,<br>)<br>print(textwrap.fill(answer[&quot;answer&quot;]))</pre><pre>NetApp&#39;s business focuses on providing storage-as-a-service solutions<br>and global support for hybrid cloud environments through a<br>multichannel distribution strategy and partnerships with leading cloud<br>providers and other industry partners.</pre><h3>Acknowledgements &amp; References</h3><ol><li><a href="https://medium.com/u/592ce2a67248">Andrew Ng</a> at <a href="https://medium.com/u/5d4101757bbb">DeepLearning.AI</a>,<br><a href="https://www.linkedin.com/in/akollegger/">Andreas Kollegger</a> at <a href="https://medium.com/u/26bec2028463">Neo4j</a><br><a href="https://learn.deeplearning.ai/courses/knowledge-graphs-rag/lesson/1/introduction">Short course on Knowledge Graphs for RAG</a><br><a href="https://github.com/neo4j-examples/sec-edgar-notebooks">https://github.com/neo4j-examples/sec-edgar-notebooks</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f1c37985ee10" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Knowledge Graphs using Neo4j — Movie Recommendation]]></title>
            <link>https://arshdeepkaur.medium.com/knowledge-graphs-using-neo4j-movie-recommendation-26b33789e6f6?source=rss-2094fd6f5143------2</link>
            <guid isPermaLink="false">https://medium.com/p/26b33789e6f6</guid>
            <category><![CDATA[neo4j]]></category>
            <category><![CDATA[cypher]]></category>
            <category><![CDATA[knowledge-graph]]></category>
            <dc:creator><![CDATA[Arshdeep Kaur]]></dc:creator>
            <pubDate>Thu, 09 May 2024 18:03:51 GMT</pubDate>
            <atom:updated>2024-05-09T18:29:27.572Z</atom:updated>
            <content:encoded><![CDATA[<h3><a href="https://medium.com/p/e3ab3b3f1904?source=post_stats_page-------------------------------------">Knowledge Graphs using Neo4j</a> — Movie Recommendation</h3><p>Movie recommendation: We will write GSQL queries to recommend movies.</p><h3><strong>MOVIE RECOMMENDATION: PLAN OF ATTACK</strong></h3><p>RAG systems use vector representations of text to match the prompt to the relevant sections within the unstructured data.</p><ol><li><strong>Create vector indices/embeddings</strong> on attribute: tagline in vertex: movie</li><li><strong>Populate vector indices</strong>: Calculate vector representation for each movie. tagline using OpenAI. Add vector to the vertex:movie as attribute:tagline_embedding.</li><li><strong>Perform similarity search</strong> using cosine similarity<br>- Calculate embedding for input question.<br>- Identify matching movies based on similarity of question and tagline_embedding vectors.</li></ol><h3><strong>MOVIE RECOMMENDATION: IMPLEMENTATION</strong></h3><ol><li><strong>Create a vector index: </strong>The first step in enabling vector search is to create a vector index. We will create vector index on node:m, with label:Movie. We we specify OpenAI to use the cosine similarity function.</li></ol><pre>kg.query(&quot;&quot;&quot;<br>  CREATE VECTOR INDEX movie_tagline_embeddings IF NOT EXISTS<br>  FOR (m:Movie) ON (m.taglineEmbedding) <br>  OPTIONS { indexConfig: {<br>    `vector.dimensions`: 1536,<br>    `vector.similarity_function`: &#39;cosine&#39;<br>  }}&quot;&quot;&quot;<br>)</pre><p>2. <strong>Verify</strong> that the vector index is created.</p><pre>kg.query(&quot;&quot;&quot;<br>  SHOW VECTOR INDEXES<br>  &quot;&quot;&quot;<br>)</pre><p>Output</p><pre>[{&#39;id&#39;: 3,<br>  &#39;name&#39;: &#39;movie_tagline_embeddings&#39;,<br>  &#39;state&#39;: &#39;POPULATING&#39;,<br>  &#39;populationPercent&#39;: 0.0,<br>  &#39;type&#39;: &#39;VECTOR&#39;,<br>  &#39;entityType&#39;: &#39;NODE&#39;,<br>  &#39;labelsOrTypes&#39;: [&#39;Movie&#39;],<br>  &#39;properties&#39;: [&#39;taglineEmbedding&#39;],<br>  &#39;indexProvider&#39;: &#39;vector-1.0&#39;,<br>  &#39;owningConstraint&#39;: None,<br>  &#39;lastRead&#39;: None,<br>  &#39;readCount&#39;: None}]</pre><p>3. <strong>Populate the vector index</strong> (calculate embeddings for each movie.tagline). We will use function: genai.vector.encode and pass it 3 parameters: 1. the value we want to encode (movie.tagline), 2. the embedding model we want to use (OpenAI), 3. the OpenAI API key.</p><pre>kg.query(&quot;&quot;&quot;<br>    MATCH (movie:Movie) WHERE movie.tagline IS NOT NULL<br>    WITH movie, genai.vector.encode(<br>        movie.tagline, <br>        &quot;OpenAI&quot;, <br>        {<br>          token: $openAiApiKey,<br>          endpoint: $openAiEndpoint<br>        }) AS vector<br>    CALL db.create.setNodeVectorProperty(movie, &quot;taglineEmbedding&quot;, vector)<br>    &quot;&quot;&quot;, <br>    params={&quot;openAiApiKey&quot;:OPENAI_API_KEY, &quot;openAiEndpoint&quot;: OPENAI_ENDPOINT} )</pre><p>4. <strong>Verify</strong> that the vector indices are populated</p><pre>result = kg.query(&quot;&quot;&quot;<br>    MATCH (m:Movie) <br>    WHERE m.tagline IS NOT NULL<br>    RETURN m.tagline, m.taglineEmbedding<br>    LIMIT 1<br>    &quot;&quot;&quot;<br>)<br>result[0][&#39;m.tagline&#39;]<br>result[0][&#39;m.taglineEmbedding&#39;][:10]</pre><p>Output</p><pre>result[0][&#39;m.tagline&#39;]<br>&#39;Welcome to the Real World&#39;<br><br>result[0][&#39;m.taglineEmbedding&#39;][:10]<br>[0.01738535612821579,<br> -0.005492697935551405,<br> -0.002040519379079342,<br> -0.02559983730316162,<br> -0.01443757489323616,<br> 0.01673029363155365,<br> -0.017123330384492874,<br> 0.0005064451252110302,<br> -0.02524610422551632,<br> -0.02953021228313446]</pre><p>5. <strong>Verify</strong> the length of the embeddings</p><pre>len(result[0][&#39;m.taglineEmbedding&#39;])</pre><p>Output</p><pre>1536</pre><p>So far so good. ✅</p><h3>MOVIE RECOMMENDATION: <strong>SIMILARITY SEARCH</strong></h3><p>Now comes the fun part: We will 1. calculate embedding for the question, 2. identify matching movies based on similarity of question and taglineEmbedding vectors.</p><ol><li>Define input <strong>question</strong>:</li></ol><pre>question = &quot;What movies are about love?&quot;</pre><p>2. <strong>Calulate embedding for the question, perform similarity search</strong></p><pre>kg.query(&quot;&quot;&quot;<br>    WITH genai.vector.encode(<br>        $question, <br>        &quot;OpenAI&quot;, <br>        {<br>          token: $openAiApiKey,<br>          endpoint: $openAiEndpoint<br>        }) AS question_embedding<br>    CALL db.index.vector.queryNodes(<br>        &#39;movie_tagline_embeddings&#39;, <br>        $top_k, <br>        question_embedding<br>        ) YIELD node AS movie, score<br>    RETURN movie.title, movie.tagline, score<br>    &quot;&quot;&quot;, <br>    params={&quot;openAiApiKey&quot;:OPENAI_API_KEY,<br>            &quot;openAiEndpoint&quot;: OPENAI_ENDPOINT,<br>            &quot;question&quot;: question,<br>            &quot;top_k&quot;: 5<br>            })</pre><p>3. Output 🍿</p><pre>[{&#39;movie.title&#39;: &#39;Joe Versus the Volcano&#39;,<br>  &#39;movie.tagline&#39;: &#39;A story of love, lava and burning desire.&#39;,<br>  &#39;score&#39;: 0.9063377380371094},<br> {&#39;movie.title&#39;: &#39;As Good as It Gets&#39;,<br>  &#39;movie.tagline&#39;: &#39;A comedy from the heart that goes for the throat.&#39;,<br>  &#39;score&#39;: 0.9022750854492188},<br> {&#39;movie.title&#39;: &#39;Snow Falling on Cedars&#39;,<br>  &#39;movie.tagline&#39;: &#39;First loves last. Forever.&#39;,<br>  &#39;score&#39;: 0.9013444781303406},<br> {&#39;movie.title&#39;: &#39;Sleepless in Seattle&#39;,<br>  &#39;movie.tagline&#39;: &#39;What if someone you never met, someone you never saw, someone you never knew was the only someone for you?&#39;,<br>  &#39;score&#39;: 0.8945244550704956},<br> {&#39;movie.title&#39;: &#39;When Harry Met Sally&#39;,<br>  &#39;movie.tagline&#39;: &#39;Can two friends sleep together and still love each other in the morning?&#39;,<br>  &#39;score&#39;: 0.8942663669586182}]</pre><p>4. Try another question</p><pre>question = &quot;What movies are about adventure?&quot;</pre><pre>kg.query(&quot;&quot;&quot;<br>    WITH genai.vector.encode(<br>        $question, <br>        &quot;OpenAI&quot;, <br>        {<br>          token: $openAiApiKey,<br>          endpoint: $openAiEndpoint<br>        }) AS question_embedding<br>    CALL db.index.vector.queryNodes(<br>        &#39;movie_tagline_embeddings&#39;, <br>        $top_k, <br>        question_embedding<br>        ) YIELD node AS movie, score<br>    RETURN movie.title, movie.tagline, score<br>    &quot;&quot;&quot;, <br>    params={&quot;openAiApiKey&quot;:OPENAI_API_KEY,<br>            &quot;openAiEndpoint&quot;: OPENAI_ENDPOINT,<br>            &quot;question&quot;: question,<br>            &quot;top_k&quot;: 5<br>            })</pre><pre>[{&#39;movie.title&#39;: &#39;RescueDawn&#39;,<br>  &#39;movie.tagline&#39;: &quot;Based on the extraordinary true story of one man&#39;s fight for freedom&quot;,<br>  &#39;score&#39;: 0.8997483253479004},<br> {&#39;movie.title&#39;: &#39;Cast Away&#39;,<br>  &#39;movie.tagline&#39;: &#39;At the edge of the world, his journey begins.&#39;,<br>  &#39;score&#39;: 0.898583173751831},<br> {&#39;movie.title&#39;: &#39;Ninja Assassin&#39;,<br>  &#39;movie.tagline&#39;: &#39;Prepare to enter a secret world of assassins&#39;,<br>  &#39;score&#39;: 0.8880103826522827},<br> {&#39;movie.title&#39;: &#39;Joe Versus the Volcano&#39;,<br>  &#39;movie.tagline&#39;: &#39;A story of love, lava and burning desire.&#39;,<br>  &#39;score&#39;: 0.8869995474815369},<br> {&#39;movie.title&#39;: &#39;As Good as It Gets&#39;,<br>  &#39;movie.tagline&#39;: &#39;A comedy from the heart that goes for the throat.&#39;,<br>  &#39;score&#39;: 0.8855770826339722}]</pre><h3>Acknowledgements &amp; References</h3><ol><li><a href="https://medium.com/u/592ce2a67248?source=post_page-----e3ab3b3f1904--------------------------------">Andrew Ng</a> at <a href="https://medium.com/u/5d4101757bbb?source=post_page-----e3ab3b3f1904--------------------------------">DeepLearning.AI</a>,<br><a href="https://www.linkedin.com/in/akollegger/">Andreas Kollegger</a> at <a href="https://medium.com/u/26bec2028463?source=post_page-----e3ab3b3f1904--------------------------------">Neo4j</a><br><a href="https://learn.deeplearning.ai/courses/knowledge-graphs-rag/lesson/1/introduction">Short course on Knowledge Graphs for RAG</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=26b33789e6f6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Knowledge Graphs for RAG using TigerGraph (version: 3.10) — part 1]]></title>
            <link>https://arshdeepkaur.medium.com/knowledge-graphs-for-rag-using-tigergraph-version-3-10-part-1-e3ab3b3f1904?source=rss-2094fd6f5143------2</link>
            <guid isPermaLink="false">https://medium.com/p/e3ab3b3f1904</guid>
            <category><![CDATA[knowledge-graph]]></category>
            <category><![CDATA[gsql]]></category>
            <category><![CDATA[similarity-search]]></category>
            <category><![CDATA[tigergraph]]></category>
            <category><![CDATA[graph-algorithms]]></category>
            <dc:creator><![CDATA[Arshdeep Kaur]]></dc:creator>
            <pubDate>Mon, 29 Apr 2024 05:00:47 GMT</pubDate>
            <atom:updated>2024-05-09T21:08:25.936Z</atom:updated>
            <content:encoded><![CDATA[<h3>Knowledge Graphs using TigerGraph (version: 3.10)</h3><h3>TigerGraph NOMENCLATURE</h3><ol><li><strong>Vertex: </strong>Represents entities/<strong>nodes</strong> in the graph, such as people, places, or things.</li><li><strong>Edge: </strong>Represents <strong>relationships</strong> between vertices.</li><li><strong>Attribute</strong>: Both vertex and edges have <strong>properties</strong>/attributes.</li><li><strong>Path</strong>: Measured by the number of relationships in the path. E.g. for a 3 node path, the length would be 2.</li></ol><h3>TigerGraph CLOUD &amp; GraphStudio SERVER SETUP</h3><ol><li>Go to <a href="https://tgcloud.io/">https://tgcloud.io/</a>. Sign up &amp; login.</li><li>Go to <a href="https://tgcloud.io/app/clusters">https://tgcloud.io/app/clusters</a> and click on Create Cluster. <br>Create a FREE cluster, and wait for 5 minutes.🍿</li><li>Click on the newly created cluster. <br>-&gt; Tools<strong> <br>-</strong>&gt;<strong> </strong>GraphStudio</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*XQySaQP8R_BTuDQbX4o0Lw.gif" /><figcaption>TigerGraph GraphStudio</figcaption></figure><h3>ACCESS MANAGEMENT</h3><ol><li>Go to <a href="https://tgcloud.io/app/clusters">https://tgcloud.io/app/clusters</a> <br>-&gt; select your newly created cluster <br>-&gt; select Access Management in the top bar</li><li>Select Database Access<br>-&gt; and click on Add Database User. <br>Add <em>username</em> and <em>password</em>.</li><li>Go to Role Management.<br>And give appropriate roles to this user.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*a1ovuZNE6jUZg8To3rQPbA.gif" /></figure><h3>GSQLShell — GSQL WEB SHELL</h3><ul><li>Define vertex and edge schema</li></ul><pre># Create vertices and edges<br>use global <br>CREATE VERTEX person (<br>  PRIMARY_ID name STRING, born DATETIME <br>)<br>WITH primary_id_as_attribute=&quot;true&quot;<br><br>CREATE VERTEX movie (<br>  PRIMARY_ID title STRING, tagline STRING, released DATETIME<br>)<br>WITH primary_id_as_attribute=&quot;true&quot;<br><br>CREATE UNDIRECTED EDGE ACTED_IN (FROM person, TO movie)<br><br>CREATE UNDIRECTED EDGE DIRECTED (FROM person, TO movie)<br><br>CREATE UNDIRECTED EDGE FRIEND (FROM person, TO person)</pre><ul><li>Create graph</li></ul><pre># Create graph<br>CREATE GRAPH movie_graph(person, movie, ACTED_IN, DIRECTED, FRIEND)<br><br># View graph<br>LS</pre><p>Now, we will repeat the above steps and add data using python package: pyTigerGraph.</p><h3>LAB CONFIGURATION</h3><p>In this section we will install the pyTigerGraph package and set up connection to GraphStudio server.</p><p>Fetch packages</p><pre>!pip install pyTigerGraph<br>!pip install python-dotenv<br>import pyTigerGraph as tg<br>from dotenv import load_dotenv<br>import os<br>import pandas as pd<br>import json<br>import requests</pre><p>Grab data files, create data frames from .csv files, and prepare for upsertion</p><pre>!git clone https://github.com/ArshdeepKaur/knowledge_graph_using_tigergraph<br><br>df_movies = pd.read_csv(&quot;/content/knowledge_graph_using_tigergraph/data/movies.csv&quot;, sep=&quot;;&quot;)<br>df_actors = pd.read_csv(&quot;/content/knowledge_graph_using_tigergraph/data/actors.csv&quot;, sep=&quot;;&quot;)<br>df_acted_in = pd.read_csv(&quot;/content/knowledge_graph_using_tigergraph/data/acted_in.csv&quot;, sep=&quot;;&quot;)<br><br>print(df_movies.info())<br>print(df_actors.info())<br>print(df_acted_in.info())</pre><p>Look for attributes with NaN/Null values</p><p>df_movies</p><pre>null_df = df_movies[df_movies.isna().any(axis=1)]<br>print(null_df)<br><br>                     title tagline  released<br>33  Something&#39;s Gotta Give     NaN      2003</pre><p>df_actors</p><pre>null_df = df_actors[df_actors.isna().any(axis=1)]<br>print(null_df)<br><br>                 name  born<br>103     Naomie Harris   NaN<br>129       Paul Blythe   NaN<br>130      Angela Scope   NaN<br>131  Jessica Thompson   NaN<br>132    James Thompson   NaN</pre><p>Replace all NaN attributes with ‘’ empty strings or 0 values depending on their data type. Otherwise we will get an error: TigerGraphException: (‘Null JSON value is not supported.’, ‘REST-30200’)</p><pre>df_movies = df_movies.fillna(&#39;&#39;) #tagline (str) has NaN<br>df_actors = df_actors.fillna(0) #born (int) has NaN</pre><p>Create host string using subdomain</p><pre>load_dotenv(&#39;/content/env&#39;, override=True)<br><br>TG_IP = os.getenv(&#39;TG_IP&#39;)<br>TG_CLUSTER = os.getenv(&#39;TG_CLUSTER&#39;)<br>TG_SUBDOMAIN = os.getenv(&#39;TG_SUBDOMAIN&#39;)<br>TG_DOMAIN=f&quot;{TG_SUBDOMAIN}.i.tgcloud.io&quot;<br>TG_SUBNET = os.getenv(&#39;TG_SUBNET&#39;)<br>TG_USERNAME = os.getenv(&#39;TG_USERNAME&#39;)<br>TG_PASSWORD = os.getenv(&#39;TG_PASSWORD&#39;)<br>TG_GRAPH = os.getenv(&#39;TG_GRAPH&#39;)<br>host_string = f&quot;https://{TG_DOMAIN}&quot;</pre><p>Setup connection</p><pre>conn = tg.TigerGraphConnection(host=host_string, username=TG_USERNAME, password=TG_PASSWORD)<br>print(&quot;Connected&quot;)<br>print(conn)</pre><h3>CREATING GRAPH SCHEMA USING pyTigerGraph</h3><p>At this point we will drop the whole schema (previously created using GSQLShell) and create everything afresh using pyTigerGraph</p><pre>#Dangerous!<br>conn.gsql(&#39;&#39;&#39;<br>USE GLOBAL<br>DROP ALL<br>&#39;&#39;&#39;)</pre><p>Create vertices and edges (again!)</p><pre>conn.gsql(&#39;&#39;&#39;<br>USE GLOBAL<br>CREATE VERTEX person (<br>  PRIMARY_ID name STRING, born INT<br>)<br>WITH primary_id_as_attribute=&quot;true&quot;<br><br>CREATE VERTEX movie (<br>  PRIMARY_ID title STRING, tagline STRING, released INT<br>)<br>WITH primary_id_as_attribute=&quot;true&quot;<br><br>CREATE UNDIRECTED EDGE ACTED_IN (FROM person, TO movie)<br>CREATE UNDIRECTED EDGE DIRECTED (FROM person, TO movie)<br>CREATE UNDIRECTED EDGE FRIEND (FROM person, TO person)<br>&#39;&#39;&#39;)</pre><p>Create Graph</p><pre>conn.gsql(&#39;&#39;&#39;<br>CREATE GRAPH movie_graph(person, movie, ACTED_IN, DIRECTED, FRIEND)<br>&#39;&#39;&#39;)</pre><p>Fetch access token</p><pre>conn.graphname = TG_GRAPH<br>secret = conn.createSecret()<br>token = conn.getToken(secret, setToken=True)<br><br>print(conn, conn.graphname)</pre><p>Verify the schema using conn object</p><pre>print(conn.getSchema())<br>print(conn.gsql(&quot;LS&quot;))<br>print(conn.getVertexTypes())<br>print(conn.getEdgeTypes())</pre><h3>UPSERTING DATA TO KNOWLEDGE GRAPH USING pyTigerGraph</h3><p>Upsert (insert + update) themovie vertex</p><pre>v_movies = conn.upsertVertexDataFrame(df_movies, &#39;movie&#39;, v_id=&#39;title&#39;, attributes={&#39;title&#39;:&#39;title&#39;, &#39;tagline&#39;: &#39;tagline&#39;, &#39;released&#39;: &#39;released&#39;})<br>print(str(v_movies) + &quot; movie VERTICES upserted&quot;)</pre><p>Upsert the person vertex</p><pre>v_actors = conn.upsertVertexDataFrame(df_actors, &#39;person&#39;, v_id=&#39;name&#39;, attributes={&#39;name&#39;:&#39;name&#39;, &#39;born&#39;: &#39;born&#39;})<br>print(str(v_actors) + &quot; person VERTICES upserted&quot;)</pre><p>Upsert ACTED_IN edge</p><pre>e_ACTED_IN = conn.upsertEdgeDataFrame(df=df_acted_in, sourceVertexType=&#39;person&#39;, edgeType=&quot;ACTED_IN&quot;, targetVertexType=&#39;movie&#39;, from_id=&#39;actor&#39;, to_id=&#39;movie&#39;, attributes={}<br>)<br>print(str(e_ACTED_IN) + &quot; ACTED_IN EDGES upserted&quot;)</pre><h3>QUERYING KNOWLEDGE GRAPHS USING pyTigerGraph</h3><p><strong>A simple 1-hop parameterized query (without accumulators):</strong></p><p>The query starts by seeding a vertex set start with the person vertex identified by parameter <em>p</em> passed in from the query call.</p><p>Next, the SELECT statement describes a 1-hop traversal according to the pattern described in the FROM clause:</p><p>start:s -(ACTED_IN:e)- movie:tgt</p><pre># simple select query<br>query = &#39;&#39;&#39;<br>    USE GRAPH movie_graph<br>    DROP QUERY select_all_movies_for_person<br>    CREATE QUERY select_all_movies_for_person (VERTEX&lt;person&gt; p) {<br>        start = {p};<br>        result = SELECT tgt<br>            FROM start:s -(ACTED_IN:e)- movie:tgt;<br>        PRINT result[result.title];<br>    }<br>INSTALL QUERY select_all_movies_for_person<br>&#39;&#39;&#39;<br><br>results = conn.gsql(query)<br>print(results)<br><br>parameters = {<br>    &quot;p&quot;: &quot;Tom Hanks&quot;<br>}<br>results = conn.runInstalledQuery(&quot;select_all_movies_for_person&quot;, params=parameters)<br>print(json.dumps(results, indent=2))</pre><p><strong>2-hop parameterized query with accumulators:</strong><br>Accumulators serve as the runtime attributes (properties) attachable to each vertex visited during our traversal on the graph.</p><p><strong>ACCUM</strong> executes its statement(s) once for each matched <strong>edge</strong> (or <strong>path</strong>) of the FROM clause pattern. Further, ACCUM parallelly executes its statements for all the matches.</p><p><strong>POST-ACCUM</strong> executes its statement(s) once for each involved <strong>vertex</strong>. Note that each statement within the POST-ACCUM clause can refer to either source vertices or target vertices but not both.</p><p>In this example we will explore different accumulator variables (ListAccum, SumAccum, AvgAccum, OrAccum), data types (INT, STRING, BOOL), scopes (@@&amp;@: @@ means the scope is global; @ means the scope is local/vertex-attached.)</p><ol><li>The list of movies that Tom Hanks has worked in:<br> ListAccum&lt;STRING&gt; @movies;</li><li>The count of movies that Tom Hanks has worked in:<br> SumAccum &lt;INT&gt; @@count_movies;</li><li>The list costars that he has worked in:<br> ListAccum&lt;STRING&gt; @costars;</li><li>The count of costars that he has worked in:<br> SumAccum &lt;INT&gt; @@count_costars;</li><li>Their average age:<br>AvgAccum @@avg_age;</li></ol><pre>query = &#39;&#39;&#39;<br>USE GRAPH movie_graph<br><br>DROP QUERY movies_of_costars<br><br>CREATE QUERY movies_of_costars (VERTEX&lt;person&gt; p, INT current_year = 2024) {<br>    OrAccum  @visited = FALSE;<br>    SumAccum &lt;INT&gt; @@count_movies;<br>    SumAccum &lt;INT&gt; @@count_costars;<br>    AvgAccum @@avg_age;<br>    ListAccum&lt;STRING&gt; @movies;<br>    ListAccum&lt;STRING&gt; @costars;<br>    <br>    start = {p};<br><br>    first_neighbors = SELECT tgt_m<br>        FROM start:s -(ACTED_IN:e)- movie:tgt_m<br>        ACCUM tgt_m.@visited += TRUE, s.@visited += TRUE, tgt_m.@movies += tgt_m.title<br>        POST-ACCUM (tgt_m) @@count_movies += 1;<br><br>    second_neighbors = SELECT tgt_p <br>        FROM person:tgt_p -(:e)- first_neighbors<br>        WHERE tgt_p.@visited == FALSE <br>        POST-ACCUM (tgt_p) @@count_costars += 1, @@avg_age += (current_year - tgt_p.born), tgt_p.@costars += tgt_p.name;<br><br>    PRINT &quot;titles&quot;;<br>    PRINT first_neighbors[first_neighbors.title];<br>    PRINT &quot;titles using @movies accumulator&quot;;<br>    PRINT first_neighbors[first_neighbors.@movies];<br>    PRINT &quot;costars&quot;;<br>    PRINT second_neighbors[second_neighbors.name];<br>    PRINT &quot;costars using @costars accumulator&quot;;<br>    PRINT second_neighbors[second_neighbors.@costars];<br>    PRINT &quot;count of movies&quot;;<br>    PRINT @@count_movies;<br>    PRINT &quot;count of costars&quot;;<br>    PRINT @@count_costars;<br>    PRINT &quot;average age of costars&quot;;<br>    PRINT @@avg_age;<br>}<br><br>INSTALL QUERY movies_of_costars<br>&#39;&#39;&#39;<br><br>results = conn.gsql(query)<br>print(results)<br><br>parameters = {<br>    &quot;p&quot;: &quot;Tom Hanks&quot;<br>}<br>results = conn.runInstalledQuery(&quot;movies_of_costars&quot;, params=parameters)<br>print(json.dumps(results, indent=2))</pre><h3>Acknowledgements &amp; References</h3><ol><li><a href="https://medium.com/u/a3858c1316db">Jon Herke</a> at <a href="https://medium.com/u/4ba88d6cae61">TigerGraph</a><br><a href="https://github.com/TigerGraph-DevLabs/AMLSim_Python_Lab/blob/main/notebooks/TigerGraph_AMLSim_Demo.ipynb">Setup Connection to TigerGraph 3.0</a></li><li><a href="https://medium.com/u/592ce2a67248">Andrew Ng</a> at <a href="https://medium.com/u/5d4101757bbb">DeepLearning.AI</a>,<br><a href="https://www.linkedin.com/in/akollegger/">Andreas Kollegger</a> at <a href="https://medium.com/u/26bec2028463">Neo4j</a><br><a href="https://learn.deeplearning.ai/courses/knowledge-graphs-rag/lesson/1/introduction">Short course on Knowledge Graphs for RAG</a></li><li><a href="https://dev.tigergraph.com/">TigerGraph community</a>: <a href="https://dev.tigergraph.com/forum/t/need-correct-json-syntax-to-upsert-a-vertex-with-no-attributes/563"><br>Need correct Json syntax to upsert a vertex with no attributes</a></li><li><a href="https://docs.tigergraph.com/home/">TigerGraph documentation</a>:<br><a href="https://docs.tigergraph.com/gsql-ref/current/tutorials/gsql-101/parameterized-gsql-query">Develop Parameterized Queries — GSQL 101</a><br><a href="https://docs.tigergraph.com/gsql-ref/current/tutorials/accumulators-tutorial">Accumulators Tutorial</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3ab3b3f1904" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The power of Transfer Learning]]></title>
            <link>https://arshdeepkaur.medium.com/the-power-of-transfer-learning-e2272fdd31a8?source=rss-2094fd6f5143------2</link>
            <guid isPermaLink="false">https://medium.com/p/e2272fdd31a8</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Arshdeep Kaur]]></dc:creator>
            <pubDate>Tue, 26 Feb 2019 10:44:02 GMT</pubDate>
            <atom:updated>2019-02-26T10:44:02.743Z</atom:updated>
            <content:encoded><![CDATA[<p>Transfer Learning is a technique in machine learning that focuses on storing knowledge gained from one task and applying it to a different but related task altogether. E.g., the knowledge gained while learning to recognize cars could apply when trying to recognize trucks.</p><p>How does that work? Remember that the earlier layers of a deel learning model learn simple shapes. Later layers identify more complex visual patterns. And the very last layers make predictions. So most layers from a pre-trained model are useful in new applications because most computer vision problems involve similar low level visual patterns. A pre-trained Convolutional Neural Network is nothing but a feature extractor — one that extracts a rich set of disciminating features, and Transfer Learning is the ability to use this pre-trained model (for instance a pre-trained ResNet model) as a way to learn patterns from data it was not originally trained on. The ResNet model has many layers. We keep all the layers and just remove the last layer that was used to make predictions. The features extracted previously can be “reused” for other classification tasks.</p><p>Transfer Learning is especially used when we don’t have a lot of data to begin with.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e2272fdd31a8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[“Reject the null hypothesis if p-value is less than 0.05.” Why exactly?]]></title>
            <link>https://arshdeepkaur.medium.com/reject-the-null-hypothesis-if-p-value-is-less-than-0-05-why-exactly-6363592c8319?source=rss-2094fd6f5143------2</link>
            <guid isPermaLink="false">https://medium.com/p/6363592c8319</guid>
            <category><![CDATA[product-management]]></category>
            <dc:creator><![CDATA[Arshdeep Kaur]]></dc:creator>
            <pubDate>Wed, 30 Jan 2019 05:49:48 GMT</pubDate>
            <atom:updated>2019-01-30T07:39:18.407Z</atom:updated>
            <content:encoded><![CDATA[<p>Let’s dig in deeper into the math behind this concept that helps us choose or reject hypothesis.</p><p>In simple terms, the p-value tells us how inline the sample set is with the null hypothesis. Hence, a large p-value tells us that that the sample set is more inline with the null hypothesis, and that we do not need to reject it. It basically means how likely are we to get a result like this if the null hypothesis is true.</p><p>Technically speaking: p-value is the probability of finding the observed, results when the <strong>null hypothesis (Ho)</strong> of a study question is true</p><p>Let’s say the mean number of visitors to my website were 50 (<strong>μo=50</strong>) before I added a new feature to my website. I want to evaluate the performance of my new feature. In order to evaluate the effectiveness of this change:</p><ol><li>We will first set up the null hypothesis Ho which says that: <br>There is no change in the mean value after the change. Hence in our case the mean number of visitors is still 50. Or in other words:<br><strong>Ho : μ = μo = 50</strong></li><li>Next, we define our <strong>alternative hypothesis (Ha)</strong>:<br>The mean is now different after adding the new feature to the website. Let’s say the new mean number of visitors is not 20. Let’s say it’s 25.<br>More specifically Ha could be 1 of these:<br><strong>Ha: μ != μo </strong>(two tailed) ; μ=100 or μ=10<strong><br>Ha: μ &lt; μo </strong>(left tailed) ; μ=10<br><strong>Ha: μ &gt;μo </strong>(right tailed) ; μ=100</li><li>Next, we define a <strong>significance level (⍺</strong>): <br>Let’s say <strong>⍺ = 5% = 0.05</strong> (This is the most common significance level)</li><li>We will now take a sample of the people visiting my website, and calculate the descriptive statistics of this sample (mean, variance, standard variation).</li><li>The evidence about Ho is the <strong>distance of the sample mean from teh null hypothesis value μo relative to the standard error — </strong>which means if Ho is true, then the new sample mean should be close to μo. Intuitively, if Ho is true, then it is unlikely that the sample mean would be very far from the original mean μo.</li><li>Hence, the question now is: If the Ho is true, what is the <strong>probability of getting a sample with the statistics</strong> that we get?<strong> </strong>Calculate this probability. Remember, p-value stands for the probability value.<br>In other words: what is the probability of getting a sample mean let’s say =55, given the original mean was 50?</li><li>If this <strong>probability is less than the defined significance level</strong>, then we reject the Ho, and say that we have evidence for Ha.</li></ol><p>Now the big question is how do we calculate the probability? The test statistic used for this purpose is the Z-statistic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/0*2yk2komVWNJaAYU-" /></figure><p>Z-score tells us how many standard deviations are we away from the mean. The farther the observed value of sample mean falls from μo, the larger is the absolute value z of Z-statistic is. The larger the value of |z|, the smaller the <strong>area under curve </strong>(calculated using the Z Score Table), i.e. the farther away we are from μo (remember left-tailed, right-tailed, both-tailed graphs) — the <strong>lesser the probability </strong>of observing such an observation given Ho is true, and hence greater the evidence against Ho. And, hence we reject the null hypothesis and keep the alternate hypothesis.</p><p>Alternatively, if the p-value is greater than the significance value, we keep the null hypothesis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vdu5tiZ0HmkbZPYD.jpg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6363592c8319" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Neural Network & Hyperparameter Tuning In A Nutshell]]></title>
            <link>https://arshdeepkaur.medium.com/neural-network-hyperparameter-tuning-in-a-nutshell-8c2b9a14a2c0?source=rss-2094fd6f5143------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c2b9a14a2c0</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[hyperparameter]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Arshdeep Kaur]]></dc:creator>
            <pubDate>Mon, 02 Jul 2018 04:29:08 GMT</pubDate>
            <atom:updated>2018-07-19T18:31:22.498Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Parameterization</strong>: Defining the necessary parameters of a NN — data, scoring function, loss function, weights &amp; biases.</p><p><strong>Objective of a NN:</strong> Use the scoring &amp; loss functions (&amp; gradient descent) to optimize the weights &amp; biases in order to obtain a high classification accuracy. Weights &amp; Biases are initialized randomly, however they cannot be initialized to zero because in each iteration of backpropagation algorithm, the weights are updated by multiplying the existing weight by a delta determined by backpropagation. If the initial weight value is 0, multiplying it by any value for delta won’t change the weight which means each iteration has no effect on the weights you’re trying to optimize. Another reason is that in a basic fully connected, feed-forward network, if each layer is initialized with the same weights, then all of the nodes follow the same path and are redundant.</p><p><strong>Hyperparameters</strong>: For a Neural Network to achieve optimal performance we need to tune the Hyperparameters. Some of the most common Hyperparameters are:</p><ol><li><strong>Value of k</strong> in k-NN algorithm</li><li><strong>Distance metric</strong> — Euclidean or Manhattan</li><li><strong>Loss function</strong> — standard Hinge loss/the squared variation loss/Cross Entropy loss. Loss landscape is usually imagined like a ceral bowl , but the difference between a loss landscape and a cereal bowl is that the cereal bowl only exists in three dimensions, while the loss landscape exists in many dimensions, perhaps tens, hundreds, or thousands of dimensions.</li><li><strong>α (alpha) </strong>— the Learning Rate used to control Gradient Descent algorithms. GD is attempting to optimize our parameters for low loss and high classification accuracy via an iterative process of taking a step in the direction that minimizes loss.Vanilla GD performs 1 weight update per epoch — making it slow. Stochastic GD applies multiple weight updates per epoch. SGD is the workhorse of deep learning — most NN are trained using SGD. It also allows us to take more steps along the gradient (one step per each batch versus one step per epoch), ultimately leading to faster convergence and no negative effects to loss and classification accuracy. Back to α. Tune α to get it JUST RIGHT. <strong>Very high learning rates</strong> (i.e. if α is too large) will drop loss initially but dramatically increase soon after — this means that he algorithm will bounce around the loss landscape &amp; not learn any patterns — never actually “descending” to the bottom of the basin (unless random bouncing takes it there by pure luck). <strong>Low learning rates</strong> will be approximately linear in their loss over time — this means that the algorithm will take a painfully long amount of computational time to achieve even a reasonable loss. <strong>High learning rates</strong> will drop quickly, but level-off. Finally, <strong>good learning rates</strong> will decrease at a rate faster than linear, but at a lower exponential, allowing us to navigate the loss landscape. <strong>Momentum</strong> and <strong>Nesterov acceleration</strong> are the bells &amp; whistles of GD.</li><li><strong>λ (lambda) </strong>— the Regularization Penalty. This method updates our weights based on the gradient multiplied by a learning rate α. Some of the pertinent questions to get it right are if regularization should be applied/which method of regularization (L1/L2/Elastic Net) should be used/what the proper value of λ is. <br>Regularization: increase testing accuracy at the expense of training accuracy. It lessens the effects of overfitting (variance), and enhances the model’s ability to generalize. With too much regularization, we run the risk of underfitting (bias). Types of regularization: <strong>L1</strong>, <strong>L2</strong> (aka Weight Decay), <strong>Elastic Net</strong>, explicit: <strong>Dropout</strong>, and implicit: <strong>Data Augmentation</strong>, and <strong>Early Stopping</strong>.<br><strong>Weight decay</strong>: The sum of squares in the L2 regularization penalty discourages large weights in the weights matrix, preferring smaller ones. Why might we want to discourage large weight values? In short, by penalizing large weights, we can improve the ability to generalize, and thereby reduce overfitting. the larger a weight value is, the more influence it has on the output prediction. Dimensions with larger weight values can almost singlehandedly control the output prediction of the classifier (provided the weight value is large enough, of course) which will almost certainly lead to overfitting.<br><strong>Dropout</strong>: Here we insert a layer that randomly disconnects nodes from the previous layer to the next layer, thereby ensuring that no single node is responsible for learning how to represent a given class.<br><strong>Data Augmentation</strong>: Data augmentation can be used to overcome small dataset limitations. It purposely perturbs training examples, changing their appearance slightly, before passing them into the network for training. The end result is that a network consistently sees “new” training data points generated from the original training data, partially alleviating the need for us to gather more training data (though in general, gathering more training data will rarely hurt your algorithm). data augmentation can help dramatically reduce overfitting, all the while ensuring that our model generalizes better to new input samples. Particularly helpful when the image dataset is small — such as the Flower quite small, having only 80 images per class for a total of 1,360 images. A general rule of thumb when applying deep learning to computer vision tasks is to have 1,000–5,000 examples per class, so we are certainly at a huge deficit here which can be overcome using Data Augmentation.</li></ol><p>While the Test set is used in evaluating the performance of the network, the <strong>Validation set</strong> (10–20% of the Training set) is used to identify &amp; tweak &amp; tune the set of parameters that work best. And the final accuracy measures are tested on the <strong>Test set</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vCjuDIofrymzImk1X9z58Q.png" /><figcaption>How α (Learning Rate) can affect Loss</figcaption></figure><p><strong>Acknowledgements</strong>:<br>Blogs, books, videos, and wisdom by <a href="https://www.linkedin.com/in/adrian-rosebrock-59b8732a/">Adrian Rosebrock</a>, <a href="https://www.linkedin.com/in/andrewyng/">Andrew NG</a>, <a href="https://www.linkedin.com/in/ian-goodfellow-b7187213/">Ian Goodfellow</a>, <a href="https://www.linkedin.com/in/rishi-sidhu-237239a/">Rishi Sidhu</a> and some other kind people in the field of Data Science.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c2b9a14a2c0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Carbon VS Silicon | Homo Deus]]></title>
            <link>https://arshdeepkaur.medium.com/carbon-vs-silicon-homo-deus-2a6e6e02e3ef?source=rss-2094fd6f5143------2</link>
            <guid isPermaLink="false">https://medium.com/p/2a6e6e02e3ef</guid>
            <category><![CDATA[books]]></category>
            <dc:creator><![CDATA[Arshdeep Kaur]]></dc:creator>
            <pubDate>Sat, 30 Jun 2018 02:24:45 GMT</pubDate>
            <atom:updated>2018-06-30T02:24:45.465Z</atom:updated>
            <content:encoded><![CDATA[<p>“1. Medicine is undergoing a tremendous conceptual revolution. 20th-century medicine aimed to heal the sick. 21st-century medicine is increasingly aiming to upgrade the healthy. If some form of upgrade becomes so cheap and common that everyone enjoys it, it will simply be considered the new baseline, which the next generation of treatments will strive to surpass.</p><p>2. Unlike in the 20th-century, when the elite had a stake in fixing the problems of the poor because they were militarily and economically vital, in the 21st-century the most efficient strategy may be to let go of the useless third-class carriages, and dash forward with the first class only. In order to compete with Japan, Brazil might need a handful of upgraded superhumans far more than millions of healthy ordinary workers.”<br>– Homo Deus: A Brief History of Tomorrow — by Yuval Noah Harari</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*wmLtavkxkavjlcQ8k-1FYQ.jpeg" /></figure><p>Humans are but algorithms shaped by natural selection — and apparently machines are getting there — only too fast! This is inline with the Substitution Effect in Economics — and as per this snippet from Homo Deus — in a few years — when the consumer is a Superhuman — the question would probably be: “Will Superhumans/Supercomputers completely wipe off normal-Humans (Homo Sapiens) from the Machines-VS-Humans graph, and thereby from the Indifference-Curve of our planet altogether? — It might not be ECONOMICALLY VIABLE for Superhumans to invest in the betterment of masses anymore, and hence Homo Sapiens could become completely irrelevant!?!?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*gaEPyf7Pnp_t5zclkOMNQg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a6e6e02e3ef" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>