<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jitendra Yadav on Medium]]></title>
        <description><![CDATA[Stories by Jitendra Yadav on Medium]]></description>
        <link>https://medium.com/@jitendrayadav_43136?source=rss-a00fbad32834------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*I_JCJcZnQI3AMdNO</url>
            <title>Stories by Jitendra Yadav on Medium</title>
            <link>https://medium.com/@jitendrayadav_43136?source=rss-a00fbad32834------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 15:45:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jitendrayadav_43136/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Implementing Data Products on Google Data Lakehouse Architecture]]></title>
            <link>https://medium.com/google-cloud/implementing-data-products-on-google-data-lakehouse-architecture-df5c5f4cc07e?source=rss-a00fbad32834------2</link>
            <guid isPermaLink="false">https://medium.com/p/df5c5f4cc07e</guid>
            <category><![CDATA[lakehouse-architecture]]></category>
            <category><![CDATA[data-quality-management]]></category>
            <category><![CDATA[data-product-creation]]></category>
            <category><![CDATA[data-governance-framework]]></category>
            <category><![CDATA[data-governance]]></category>
            <dc:creator><![CDATA[Jitendra Yadav]]></dc:creator>
            <pubDate>Sat, 21 Sep 2024 01:35:52 GMT</pubDate>
            <atom:updated>2024-09-21T01:35:52.576Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Authors:</blockquote><blockquote><a href="https://uk.linkedin.com/in/mihaelasavastre">Mihaela Savastre</a>, EMEA Data Analytics Solution Lead, Google Cloud</blockquote><blockquote><a href="http://in.linkedin.com/pub/jitendra-yadav/21/48b/881">Jitendra Yadav</a>, Data &amp; AI/ML Specialist, Google Cloud</blockquote><p>Organizations today require a robust and scalable data foundation to support their growing data needs. A Lakehouse architecture offers a modern approach, combining the scalability and flexibility of a data lake with the transactional capabilities, performance, and security of a data warehouse.</p><p>This approach allows business users to access data with the speed and reliability they need, without relying on disparate tools and technologies. It addresses common challenges such as data silos, inconsistent data quality, and slow access to information.</p><p>Furthermore, a Lakehouse architecture provides the technical capabilities for storing, processing, and accessing data at scale, regardless of its structure or format. This supports self-service data access and data sharing, key aspects of a modern data strategy. By utilizing a single storage layer for all data, organizations can leverage different data processing engines based on the specific use case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ac9VnxBYNMA3CPd7MdiHdg.png" /></figure><p>A Data Mesh operating model is the ideal complement to this architecture, unlocking its full potential. Data mesh principles guide how data is managed, accessed, and governed across different domains within an organization. This ensures that data is treated as a valuable asset, fostering collaboration and reducing data silos</p><p>A data mesh is an architectural and organizational framework that treats data as a product. Teams that best understand their data and who follow an organization wide set of data governance standards develop data products most effectively. Once the products are deployed, distributed teams can discover and access data that is relevant to their needs more quickly and efficiently.</p><p>To achieve a well-functioning data mesh, the high level architectural components and organizational roles must be established first.</p><p>Key data mesh principles:</p><ul><li><strong>Domain-oriented decentralized data ownership &amp; architecture: </strong>Data assets are organized, and data ownership assigned, based on domains, with de-centralized ownership of data across business specific data domains which are centrally managed and monitored. This organisational structure creates an ecosystem producing and consuming data which can scale out as the number of sources of data, the number of use cases, and the diversity of access models to the data, increases.</li><li><strong>Data as a product: </strong>A logical collection of data resources that enables data users to easily discover, understand and securely use high quality data.</li></ul><p>Data products can be expanded to be classified as source-aligned, aggregated and consumer aligned, and templated to record the intent of each data product in a consistent way. Data products can then be discovered, accessed, consumed, managed and monitored against the agreed SLAs.</p><ul><li><strong>Self-serve data infrastructure as a platform: </strong>Enables the domain teams to<em> create and consume data products autonomously</em> using Google Cloud Platform abstractions, hiding the complexity of building, executing and maintaining secure and interoperable data products.</li><li><strong>Federated computational governance</strong></li></ul><p>A standard set of tools and utilities for domain teams to leverage, along with standardised metadata elements and formats which the product teams will use during the build and management of each data product.</p><p>This approach can be visualised like so:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/0*TGaH2-BiO8BPDp6B" /></figure><p>Converting the simplistic approach above to a more architectural and functional view with aforementioned team personas mapped in order produces the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2i8J148LiAf1JoXn" /></figure><p>However, an appropriate data mesh architecture depends heavily on the internal organisation of the teams. As alluded to before, there is no one size fits all solution, and a decision will therefore need to be made by organisations on how centralised and decentralised certain functions should be.</p><p>As an example, a fully decentralised architecture is presented here, including both source-oriented and consumer-oriented data products:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ie7nvla6f6O3qvsX" /></figure><p>Google Cloud has the necessary services and controls to enable organisation to deliver their required balance between centralisation and decentralisation, but to continue the fully decentralised theme, the following is an example implementation of the above diagram’s approach:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ToTganzgGqQbFpfr" /></figure><p>Regardless of the amount of centralisation, the capabilities from the previous diagram are always required, and are shown here:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*f-cdpveHv2zRbhhn" /></figure><p>A pure Google Cloud implementation of the above capabilities would look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K25Va9Up83W8J5BJ" /></figure><p>Apart from the specialised services for ingesting, storing, processing and sharing data a key component in this architecture is <strong>Dataplex</strong>, which plays the role of the governance backbone enabling distributed data management, policy-as-code data quality, data classification, lifecycle management, metadata collection, data discovery and data interoperability between domains.</p><h3>Automation and Self-Service</h3><p>To be truly effective, the previously described Google Cloud implementation needs to be as automated and self-service as possible, for data platform, data governance, data producer and data consumer teams.</p><p>With the aforementioned approach:</p><p>A data governance team can self-serve their own reporting needs, exploring access logging, control framework adherence, data lineage, management information etc. by owning their own</p><p>A data platform team could describe all of the necessary infrastructure building blocks in Terraform Infrastructure-as-Code templates, with CICD pipelines to deploy and configure all of the Google Cloud services needed to build a data product in an approved way.</p><p>A data producer team can then use these components as a factory to generate repeatable data products quickly and consistently, knowing that all security controls, observability, integration with governance etc. is automatically baked into the data product. A data consumer can quickly and easily discover, request access, and start to use data products without being required to interact with a data producer team, aside from perhaps an access approval workflow.</p><p>The goal with this approach is that organisations can reduce the amount of conversations and hand-offs between teams and thus reduce the implicit delays that such things incur, while still being able to provide teams with what they need in a well-governed manner.</p><p>An example self-service data platform approach can be visualised like so:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DDVWsHkSsOzBVL9f" /></figure><h3>Common Services — Data Operations and Governance</h3><p>Dataplex is the backbone of the data mesh implementation described above in Google Cloud.</p><p>Dataplex enables automated data discovery, metadata harvesting and enrichment of structured, semi-structured and unstructured data stored in Google Cloud and beyond, all without data movement, with Dataplex Data Catalog.</p><p>Dataplex also offers centralised security policies as code, monitoring and auditing for data authorisation, retention and classification to help answer questions such as ‘Who did what when?’.</p><p>Dataplex also enables automatic collection of lineage from BigQuery, Dataproc, VertexAI and other data processing services and stores, and the built-in AI-driven intelligence across data classification, data quality, data deduplication and lifecycle management helps to answer the question of ‘Who produced the data, what was the source and what transformations have been applied to the data since its inception?’.</p><p>In diagram form this looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/0*-zLKAYiOuBWSpk43" /></figure><h3>Common Services — Observability</h3><p>The ability to understand, visualise and report upon data products is important to allow data product managers to gain valuable insight into the use of their products, for security teams to understand access control and user behaviour, and for business teams to generate management information.</p><p>Google Cloud provides capability in all of these areas, building upon the monitoring and auditing in Dataplex. In Dataplex, and across all Google Cloud services, Audit Logging provides information regarding admin activities, system changes, data access and data updates. This is used extensively within tons of Google Cloud customers already, and connects to security tooling across the estate for threat detection and forensic analysis.</p><p>As mentioned, Dataplex already provides information regarding auditing for data authorisation, and Dataplex will see greater improvements to its surfacing of logging and monitoring information. However, for Management or Business Information or other business driven processes, Organization business users may prefer the flexibility and ease-of-use of a Looker dashboard.</p><p>Dataplex and BigQuery audit logs can be sent to an aggregated sink and a Looker dashboard can be connected to that sink, enabling a data product manager, auditor or access control analyst to quickly see what datasets are being accessed when, and by whom.</p><p>Google has some BigQuery Audit Logging Views available on GitHub: <a href="https://github.com/GoogleCloudPlatform/bigquery-utils/tree/master/views/audit">https://github.com/GoogleCloudPlatform/bigquery-utils/tree/master/views/audit</a>.</p><p>An example Looker dashboard could be an access report:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jcnIeTymwJKUtzVg" /></figure><p>Looker can also show pre-built reporting blocks for BigQuery performance, concurrency and throughput, and cost.</p><p>For example, a Looker dashboard for usage trends, spend trends, chargeback per project and job performance can help a data product manager to understand cost to performance for their consumers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*C2LsHBJto9EYsPHH" /></figure><p>Looker can also receive data from other Google services like Analytics Hub or Apigee.</p><p>For example, Analytics Hub has usage metrics for shared data, allowing data product managers to track how subscribers are using datasets. And sending Apigee’s analytics to Looker will create visualisations of API usage.</p><p>By combining different data sources into Looker dashboards, Organizations could create highly customisable, self-serve reporting on their data product ecosystem that would help to demonstrate data points such as:</p><ul><li>The percentage of automatically generated vs. manually input metadata</li><li>The percentage of data at each level of data confidentiality classification</li><li>Who, what and when data is being accessed</li><li>How is data being accessed</li><li>Which transformations data has been through</li><li>The quality [ accuracy, completeness, consistency and freshness ] of data</li><li>The percentage of adherence to data governance controls</li><li>Cost vs. performance for data</li></ul><p>These reports can be in near real-time or on a schedule. For example, an auditor or access control analyst may wish to see who could access a dataset today and also see who has accessed the data in the past month.</p><h3>Common Frameworks — Data Governance</h3><p>Data Governance is key to a successful data project and adoption. It is a broad term concerning security and sensitive data management, but also how to ensure end users understand what data is available in the business and how they could request access to it. For Data Governance, Dataplex is the unified governance layer on top of your data platform on Google Cloud.</p><p>A proper data governance model often requires:</p><ul><li>Fostering a collaboration between IT, data and business teams</li><li>Building a framework that enables every stakeholder to address his need with a good level of autonomy, while adhering to the best practices defined by the IT department</li><li>Enabling self-service analytics and fast time-to-insights</li><li>Implementing best practices in terms of security, DevOps/DataOps, compliance and regulations</li><li>Enabling data science teams to quickly build and deliver production-grade models</li></ul><p>However setting up a proper data governance model at scale is not trivial. Traditional approaches, consisting of relying on a central data team to handle most of the data life cycle, from data collection, data transformation, data quality and data security, up to the business KPIs definition have failed to scale and deliver the expected outcomes in most organisations, as depicted in the illustration below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PyjYkXlkLUVTB_yq" /></figure><p>Most of the challenges mentioned here often come from the level of autonomy given to the business teams and the ownership model for the different steps of the data life cycle.</p><p>Returning to the key expectations that organisations have given, some of the challenges can be described as:</p><ul><li>Enabling other directions to be fully autonomous to analyse data and build reports</li><li>Enabling a self-service analytics experience for the business team while keeping a strong governance model</li><li>Be able to have a clear readability of the data lineage</li><li>Be able to properly secure data at large scale</li><li>Deploy AI/ML models in production with technology good practices</li></ul><p>Google’s approach to set up a rigorous data governance model is to partner with the bank to help in defining the best architecture and strategy to make organisations data accessible, usable and secure with a good level of collaboration between the different stakeholders.</p><p>And this requires something new, as depicted in the diagram below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/0*Rm2EHYc99mJv02Nk" /></figure><p>In this model, the aforementioned Data Mesh paradigm is relied upon to empower each business team with ownership of their respective data domains, thus meaning that they are responsible for:</p><ul><li>Defining the business KPIs that are most useful for them</li><li>Ensuring proper data quality for the data assets they rely on to define their KPIs</li><li>Clearly defining the scope and data access policies for the business objects created within their perimeter</li></ul><p>The central organisations Google Cloud team, technology platform teams and data platform teams in this model are responsible for core tasks such as:</p><ul><li>Building and maintaining the data pipelines</li><li>Implementing the data access policies</li><li>FinOps</li><li>Defining the best DevOps and DataOps practices</li></ul><p>In Google Cloud Dataplex that new data governance model can be implemented like so:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*v4tGA4h1YBLM7iEq" /></figure><h3>Common Frameworks — Data quality</h3><p>Dataplex comes with an out of the box data quality framework integrated with BigQuery and BigLake for executing data quality rules and with BigQuery to store the outcome of the data quality checks together with the SQL to identify non-compliant data.</p><p>Storing the outcome in BigQuery allows the integration with Looker and other reporting tools to build dashboards for data quality monitoring, like so:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3BO1ygcrJS9BEk6-" /></figure><h3>Common Frameworks — Ingestion Patterns</h3><p>Through building ingestion frameworks and patterns the data platform team can introduce standardisation across different data product teams on how to ingest data depending on type and speed of data ingested.</p><p>In addition, company wide standards for logging and monitoring, collection of telemetry metrics will be part of the framework / patterns which will bring consistency across all data products.</p><p>Google Cloud Platform has key services and integrations which can be used in ingestion frameworks and patterns to accelerate data load in BigQuery for both file ingestion and streaming.</p><p>There are 2 storage services which are a key differentiator for Google in this area:</p><ul><li><a href="https://cloud.google.com/biglake">BigLake</a> — storage engine that unifies data warehouses and lakes, by providing uniform fine-grained access control, performance acceleration across multi-cloud storage and open formats. BigLake is integrated with Dataplex and data in buckets can be automatically registered in BigQuery for analytics without additional load routine. An example of an ingestion pattern in BigQuery of batch data is below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vbJNq8ReMwMbHC8l" /></figure><ul><li><a href="https://cloud.google.com/bigquery/docs/write-api#write-api-overviewreaming">BigQuery Storage API</a> — The BigQuery Storage Write API is a unified data-ingestion API for BigQuery. It combines streaming ingestion and batch loading into a single high-performance API. The Storage Write API can be used to stream records into BigQuery in real time or to batch process an arbitrarily large number of records and commit them in a single atomic operation. The write API is integrated with Datastream for applying changes to BigQuery tables in an automated, CICD way.</li></ul><p>There are many options for designing ingestion patterns, including Datastream, Pub/Sub and more, and the right approach can be chosen only by looking at concrete use cases and the non-functional requirements that organisations has.</p><h3>A Data Product Approach within a Data Mesh</h3><p>So far this document has described how to build a Data Mesh on Google Cloud, and now the concept of a Data Product itself can be explored.</p><p>The architecture of a data product can be conceptualised as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7tcwNgYPfH-sRaPi" /></figure><p><strong>Analytics Hub</strong></p><p><a href="https://cloud.google.com/analytics-hub">Analytics Hub</a> is Google’s secure data exchange that can allow organisations to curate a library of internal and external data assets backed by the power of BigQuery.</p><p>With Analytics Hub, organisations can act as both a data subscriber and publisher, for internal and external data.</p><p>Analytics Hub breaks down data into three classifications; Public, Private, and Organisational.</p><p>As a subscriber to external data, either public or private, organisations are able to consume external datasets such as ESG data from Morningstar. Google is expanding the number of curated datasets and is keen to partner with organisations in this area to ensure the right data providers are available.</p><p>Organisational data allows for organisations to share large datasets internally without moving the data. For example, organisations could combine datasets into a single dataset and publish it within the organisation to be consumed internally.</p><p>As a publisher, either internally or externally, organisations can list the data product manager’s contact details, the category of data, and other documentation to aid in consumption of the dataset — information which is then visible to potential consumers. Dataset egress can also be restricted to prevent copying and export of shared data and / or query results.</p><p>Sharing data often makes people consider data sovereignty and legislation like GDPR. To meet such requirements an Analytics Hub data exchange can be created for a specific region and can contain BigQuery objects available only in that region.</p><p>The datasets being shared are BigQuery datasets, and listings can be any of the following types of objects:</p><p>Authorised Views, Authorised Datasets, BigQuery ML models, External Tables, Materialised Views, Tables, Table Snapshots and Views.</p><p>Note that shared datasets maintain support for row-level and column-level security.</p><h3>Dataset Discoverability</h3><p>As with API products, datasets in Analytics Hub first need to be discovered before they can be consumed. This is possible using the Google Cloud Console and API. When searching for datasets, filters can be applied to find the relevant category, data classification or provider.</p><p>Potential consumers can peruse the dataset’s documentation, such as licensing, that the publisher has provided and decide if the dataset is suitable for their needs.</p><p>In the future, in-built access and approval workflows should provide a more controlled way for subscribers to discover and register for dataset usage.</p><p>The exchange metadata can be integrated in Dataplex as product tag and the information regarding the exchange and the link to the exchange can be available as part of the product information.</p><h3>Dataset Reporting</h3><p>Usage metrics in Analytics Hub allow publishers the ability to visualise aggregated usage, jobs that are running against a shared dataset and other consumption details. Sending these data points to Looker could make it simpler to visualise all data related to the data product in a single pane of glass. The idea of bundling analytics is currently on the roadmap, which is always subject to change, but when delivered should provide a more out of the box integration with Looker.</p><h3>Looker Semantic Layer</h3><p><a href="https://cloud.google.com/looker">Looker</a> is Google Cloud’s enterprise platform for business intelligence, data applications, and embedded analytics. Looker’s semantic modelling layer enables enterprises to standardise business logic across the organisation.</p><p>In the context of data mesh and data product organisation, Looker ML plays an important role as it abstracts the physical location of the data assets in multiple data products and presents a complete view of data managed in different data products for BI. The semantic layer built in LookerML can be accessed for building visualisations from Looker, Tableau, Power BI and other BI tools.</p><p>An illustration of a data mesh architecture on GCP with Looker Semantic Layer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PLxocJ3VQmJwQJx6" /></figure><p>Using Looker ML, data analysts can create in a governed and secured way additional metrics and KPIs based on the raw data shared through the data products and further use them in creating visualisations in a BI tool: Looker, Tableau or Power BI through the available connectors.</p><p>In terms of consumers of the looker semantic layer, Looker Semantic Layer supports 1st party tools like Looker and Google Sheets and 3rd party tools like Tableau and Power BI along with other exciting features on the roadmap regarding JDBC connection, Open API framework and Open SQL Interface.</p><p>In conclusion, Google’s Lakehouse architecture empowers organizations to embrace a “data as a product” philosophy. By combining the strengths of data lakes and warehouses, it provides a unified platform for storing, processing, and accessing all data, regardless of its structure or format. This enables efficient data sharing, self-service analytics, and collaboration across different business domains. For more detailed info on Google Data Lakehouse, please refer Google <a href="https://cloud.google.com/discover/what-is-a-data-lakehouse?e=48754805&amp;hl=en">Lakehouse</a> resources.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df5c5f4cc07e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/implementing-data-products-on-google-data-lakehouse-architecture-df5c5f4cc07e">Implementing Data Products on Google Data Lakehouse Architecture</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Generative AI: Revolutionising Infrastructure As Conversation]]></title>
            <link>https://medium.com/google-cloud/generative-ai-revolutionising-self-service-data-platforms-c78f2aa88d64?source=rss-a00fbad32834------2</link>
            <guid isPermaLink="false">https://medium.com/p/c78f2aa88d64</guid>
            <category><![CDATA[generative-ai]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Jitendra Yadav]]></dc:creator>
            <pubDate>Mon, 02 Oct 2023 12:42:36 GMT</pubDate>
            <atom:updated>2023-10-05T21:23:27.631Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://cloud.google.com/ai/generative-ai">GenerativeAI</a> is here to accelerate the transformation of self-service platforms. Organizations in any industry can use GenerativeAI to create self-service platforms for example data, computing, networking and storage infrastructure, with organizational context and guardrails. These self-service platforms will improve typical IT accessibility issues and allow users with different personas to interact with the platform without much complexity or a steep learning curve.</p><p>Google Cloud offers a number of services that can be used to build and deploy generative AI models. With <a href="https://cloud.google.com/vertex-ai">Vertex AI</a>, interact with, customize, and embed foundation models into your applications — no ML expertise required. Access foundation models on <a href="https://cloud.google.com/model-garden">Model Garden</a>, tune models via a simple UI on <a href="https://cloud.google.com/generative-ai-studio?hl=en">Generative AI Studio</a>, or use models in a data science notebook. <a href="https://cloud.google.com/generative-ai-app-builder">Vertex AI</a> Search and Conversation offers developers the fastest way to build generative AI powered <a href="https://cloud.google.com/enterprise-search/index.html">search engines</a> and chatbots. And, <a href="https://cloud.google.com/blog/products/application-modernization/introducing-duet-ai-for-google-cloud">Duet AI</a> serves as an always-on AI collaborator that helps users of all skill levels where they need it.</p><p>One of the major changes we will see is how organizations design, deploy, and manage their IT infrastructure. It is becoming increasingly evident that generative AI will revolutionize the Infrastructure as Code (IaC) paradigm. What we currently think of as IaC will change to IaC (Infrastructure as Conversation/Chat). This paradigm will not only exponentially increase the productivity of IT professionals, but it will also decrease the time it takes to bring products and services to market when organizations are developing new offerings for their target audiences. By simplifying and abstracting away the complex IT tasks from the end user, it will greatly improve the organization’s efficiency and allow them to focus more on generating business value.</p><p>Infrastructure as Code(IaC):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6l-oFrBnZ2ohz2Hq2mtKEw.png" /></figure><p>Infrastructure as Conversation(IaC):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5YQQ1YSp2LuLFU8-" /></figure><p><strong>Data platform scalability challenges with people &amp; process:</strong></p><p>A data cloud platform that combines data and AI capabilities is critical for providing transformative experiences for customers through modern applications, unlocking timely insights from a variety of data sources, and enabling businesses to take action on data-driven decisions. Organizations need specialized professionals to collect, process, and analyze vast amounts of data on a daily basis. Building and maintaining these data pipelines takes a significant amount of time and effort. To get the most out of a data platform, the data producer must transform and enrich the data with enough business context so that business users or other consumers can use that data to create further data-driven applications or make informed decisions.</p><p>The end-to-end data to value process is a complex task due to the technologies involved and the way organizations operate internally. The ratio of data platform team members to platform end users is significantly high, making it impossible to serve each and every request quickly. An intelligent self-service platform can solve these people and process scalability issues by allowing end users to interact with the data platform without complexity or accessibility delays. This will reduce the time it takes to realise value from data and allow users to focus on business priorities.</p><p><strong>How to solve people and process scalability issues:</strong></p><p>Now let’s take a look at the “Art of Possible” for building an GenerativeAI Self Service Data Platform architecture. In this architecture I’m using a simple UI interface for user interaction. Underneath, <a href="https://www.langchain.com/">Langchain</a> framework is used to create an application on top of Google <a href="https://cloud.google.com/vertex-ai">Vertex AI</a> LLM model and Matching Engine to store the vector embeddings. One of the important components of this achitecture is the <a href="https://python.langchain.com/docs/modules/agents/tools/">Langchain Tools</a> feature. Tools are collections of functions which agents can use to fulfill certain tasks instructed by the LLM, you could simply create a custom tool or autonomous agent to call an API or use out of the box available <a href="https://js.langchain.com/docs/api/tools/">tools</a> or <a href="https://js.langchain.com/docs/api/agents/">agents</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GBa2GyCx0vEXSaRc" /></figure><p>Here is an example of a self-service data platform using Generative AI. As you can see, it is getting simpler for end users to engage with the data platform and carry out their tasks using plain language.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FRHlgR_DM26M%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DRHlgR_DM26M&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8d996a05d983b46facdca6ba0ecdc8c5/href">https://medium.com/media/8d996a05d983b46facdca6ba0ecdc8c5/href</a></iframe><p>In conclusion, <a href="https://cloud.google.com/ai/generative-ai">Generative AI</a> is poised to become the foundation of smart self-service platforms. As large language models and their AI application framework mature more in the future, it will become a key component of any platform modernisation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c78f2aa88d64" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/generative-ai-revolutionising-self-service-data-platforms-c78f2aa88d64">Generative AI: Revolutionising Infrastructure As Conversation</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>