<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Biju Krishnan @ DataSiens on Medium]]></title>
        <description><![CDATA[Stories by Biju Krishnan @ DataSiens on Medium]]></description>
        <link>https://medium.com/@biju.krishnan_73542?source=rss-6b4c2196235------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*2N827SqB78aGHroCPOJj_w.jpeg</url>
            <title>Stories by Biju Krishnan @ DataSiens on Medium</title>
            <link>https://medium.com/@biju.krishnan_73542?source=rss-6b4c2196235------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 18:23:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@biju.krishnan_73542/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Personalised Tutor for Data Architecture]]></title>
            <link>https://medium.com/@biju.krishnan_73542/personalised-tutor-for-data-architecture-1091dd5e7b3c?source=rss-6b4c2196235------2</link>
            <guid isPermaLink="false">https://medium.com/p/1091dd5e7b3c</guid>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[data-mesh]]></category>
            <category><![CDATA[feature-store]]></category>
            <dc:creator><![CDATA[Biju Krishnan @ DataSiens]]></dc:creator>
            <pubDate>Mon, 14 Oct 2024 08:22:05 GMT</pubDate>
            <atom:updated>2024-10-14T08:22:05.436Z</atom:updated>
            <content:encoded><![CDATA[<p>Wanted to announce a new chatbot feature on my website, <a href="http://learndataarchitecture.com/">learndataarchitecture.com</a>. It can be accessed via the “Chat about DA” button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*KFcUQtA2kSR-NYlOwiXF2g.png" /><figcaption>Chat about DA feature</figcaption></figure><p>Now you can get personalised answers on the topic of data architecture via this feature.</p><p>Lets look at how the bot responds to questions asked on the forum on Udemy.</p><pre>When we talk about feature store, it is like a cube, used in other programs? <br>Is this kind os the same thing?</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*nBzrMvsUQf4bRN81UFKn0Q.png" /><figcaption>Response from chatbot about feature stores</figcaption></figure><pre>is it best practice to keep structure data to a data lake? <br>please justify the answer if yes or no</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*Wf-dHS89lFtbDQxmAnOMaQ.png" /><figcaption>Chatbot response to if structure data can be stored in data lake</figcaption></figure><p>I hope you enjoy this new feature and am looking forward to any feedback you may have about the chatbot.</p><p>If you find any issues with the chatbot, do feel free to let me know in the comments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1091dd5e7b3c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Fabric — Realize the data mesh]]></title>
            <link>https://medium.com/@biju.krishnan_73542/data-fabric-realize-the-data-mesh-ae43fa337c6b?source=rss-6b4c2196235------2</link>
            <guid isPermaLink="false">https://medium.com/p/ae43fa337c6b</guid>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[data-mesh]]></category>
            <category><![CDATA[data-fabric]]></category>
            <dc:creator><![CDATA[Biju Krishnan @ DataSiens]]></dc:creator>
            <pubDate>Tue, 21 May 2024 08:13:36 GMT</pubDate>
            <atom:updated>2024-05-21T08:13:36.183Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Fabric — Realize the data mesh</h3><p>In the evolving landscape of data management, businesses are continually seeking ways to harness the full potential of their data. The concept of data mesh has emerged as a significant paradigm, advocating for decentralized data ownership and democratized access to data and analytics. However, implementing data mesh principles can be challenging, especially in diverse technological environments. Enter data fabric, an architectural framework designed to address these challenges and enhance the implementation of data mesh. In this blog post, we will explore what data fabric is, the problems it solves, and its benefits in modern data architecture.</p><h3>Understanding Data Fabric</h3><p>Data fabric is an architectural framework that supports the implementation of data mesh principles. It aims to provide a unified and consistent way to manage, access, and govern data across various environments, whether on-premises, in the cloud, or in hybrid settings. The primary goal of data fabric is to democratize data and analytics, making it easier for business units to extract value from data while allowing them to choose the technology stacks they are most comfortable with.</p><h3>The Role of the Data Platform Team</h3><p>The data platform team plays a crucial role in implementing the frameworks, philosophy, and concepts of data mesh. They are responsible for ensuring that the organization’s data infrastructure supports the decentralized and democratized approach promoted by data mesh. If you’re unfamiliar with the role of the data platform team, I highly recommend revisiting the introduction to data mesh lecture in our course.</p><h3>Architectural Framework: Data Fabric</h3><p>One of the key architectural frameworks that has emerged from the data mesh discussion is data fabric. This framework is designed to address several challenges that organizations face when implementing a data mesh.</p><h4>Challenges in Implementing Data Mesh</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/1*SuHJn5qd7SCT4imNMIUyAw.png" /><figcaption>Challenges in implementing a data mesh</figcaption></figure><ol><li>Data Silos</li></ol><p>In many organizations, different departments use their own infrastructure to manage data, leading to the creation of data silos. While this approach works well within individual departments, it hinders collaboration and the creation of cross-departmental data products. Data fabric aims to break down these silos by providing a unified layer that allows seamless data access across departments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-uqoRFpIY2WpDqT6gVhRCg.png" /><figcaption>Data Silos resulting from independent choices made by business</figcaption></figure><p>2. Data Duplication</p><p>Creating data products often requires copying data from one infrastructure to another, resulting in multiple copies of the same data. This duplication not only increases storage costs but also complicates data governance. Data fabric addresses this issue by enabling data virtualization, which allows users to access data without having to move or duplicate it.</p><p>3. Diverse Tech Stacks</p><p>Organizations often have diverse technology stacks, with different departments using different tools and platforms. For example, marketing analytics might be hosted on AWS, while sales could use Azure, and finance might rely on on-premises solutions due to regulatory requirements. This diversity poses several challenges, including data integration, security, and governance. Data fabric provides a standardized approach that overlays existing environments, allowing organizations to manage data more effectively without requiring a complete overhaul of their technology stacks.</p><p>4. Data Format Variability</p><p>Different departments may store data in various formats, requiring additional effort to unify these formats for analysis. Data fabric helps manage and organize data through data cataloging and profiling tools, which standardize data formats and make them easier to analyze.</p><p>5. User Experience</p><p>The use of diverse technologies means that users often have to learn new tools and languages to access different datasets, slowing down analysis and reducing efficiency. Data fabric simplifies user experience by providing a unified semantic query layer, which allows users to query data using familiar interfaces.</p><p>6. Security and Governance</p><p>Different technologies have unique security and governance implementations, making it challenging to unify these controls across the organization. Data fabric addresses this by providing a consistent layer for security and governance, ensuring that data remains secure and compliant across all environments.</p><p>7. Monitoring and Audit Logs</p><p>Managing audit logs and monitoring data usage can be complicated due to the need to gather logs from diverse technologies. Data fabric simplifies this process by providing centralized monitoring and logging capabilities.</p><p>8. Billing and Licensing</p><p>A diverse data landscape makes it difficult to visualize and control costs. Data fabric helps organizations manage billing and licensing more effectively by providing a unified view of all data-related expenses.</p><h3>The Benefits of Data Fabric</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/867/1*xaShOEViCupswM8V1G1LVw.png" /><figcaption>Data Fabric Logical Architecture Diagram</figcaption></figure><p>Reference: Data Fabric Primer by Rahul Barik, accessible <a href="https://www.globallogic.com/in/wp-content/uploads/sites/21/2023/12/Paper-Data-Fabric-primer.pdf">here</a>.</p><h4>Automated Metadata Collection</h4><p>The first step in implementing a data fabric architecture is the automated collection of metadata from all data sources. Metadata provides critical information about data, such as its source, format, and usage. By collecting and organizing this metadata, data fabric creates a foundation for effective data management.</p><h4>Building a Semantic Layer</h4><p>A semantic layer built on top of the metadata allows the creation of business-specific data models. This layer translates complex data structures into understandable business terms, enabling teams to create different models using the same dataset. For instance, sales and marketing teams can develop distinct models tailored to their specific needs.</p><h4>Knowledge Graphs</h4><p>Data fabric uses knowledge graphs to connect data across datasets, creating a graph of entities and their relationships. Knowledge graphs enhance data discovery and provide a comprehensive view of data relationships, making it easier to analyze and derive insights.</p><h4>Data Catalog and Profiling Tools</h4><p>Data catalog and profiling tools help manage and organize data within the data fabric framework. These tools provide a centralized repository for data assets, enabling users to easily discover, understand, and access data.</p><h4>Semantic Inference Layer</h4><p>Powered by AI and machine learning, the semantic inference layer allows users to query data using natural language. This feature simplifies data interaction, making it accessible to users who may not have technical expertise.</p><h4>Data Virtualization</h4><p>Data virtualization unifies access to data across different technologies, enabling users to run queries without having to learn new languages or move data between environments. This approach reduces the complexity of data integration and improves efficiency.</p><h4>Simplified Data Queries</h4><p>The semantic query layer in data fabric provides interfaces similar to ChatGPT, simplifying data queries and making it easier for users to interact with data. This layer abstracts the complexity of underlying data structures, allowing users to focus on deriving insights.</p><h4>Realizing the Potential of Data Fabric</h4><p>The benefits of data fabric are significant. By providing a unified layer for data governance, access, and integration, data fabric enhances the ability to manage and utilize data across the organization. Business users can more easily discover and access data, improving business processes and decision-making. Data scientists can leverage data across silos to create recommendations and machine learning applications, driving innovation and value.</p><h4>Improved Data Governance</h4><p>Effective data governance is crucial for ensuring data quality, security, and compliance. Data fabric enables consistent governance processes across diverse environments, reducing the effort required to manage data and ensuring that data remains reliable and secure.</p><h4>Enhanced Data Discovery</h4><p>Data fabric improves data discovery by providing centralized tools for managing and cataloging data. This makes it easier for users to find the data they need and understand its context, leading to more informed decision-making.</p><h4>Streamlined Data Integration</h4><p>By enabling data virtualization, data fabric simplifies data integration and reduces the need for data duplication. This not only lowers storage costs but also minimizes the complexity of data management, allowing organizations to focus on deriving value from their data.</p><h4>Empowering Business Users</h4><p>With data fabric, business users can access and interact with data more easily, improving their ability to drive business outcomes. The use of natural language queries and intuitive interfaces makes data analysis accessible to a broader audience, fostering a data-driven culture.</p><h4>Supporting Data Scientists</h4><p>Data scientists benefit from the ability to access data across silos, enabling them to create more comprehensive models and analyses. This enhances their ability to develop recommendations and machine learning applications, driving innovation and competitive advantage.</p><h3>Conclusion</h3><p>Data fabric represents a transformative approach to data management, addressing the challenges of implementing data mesh and providing a unified framework for data governance, access, and integration. By leveraging automated metadata collection, semantic layers, knowledge graphs, and data virtualization, data fabric enables organizations to harness the full potential of their data. Whether you are a business user, data scientist, or IT professional, understanding and implementing data fabric can significantly enhance your ability to manage and utilize data, driving better business outcomes and fostering a data-driven culture.</p><p>Learn more about Data Fabrics and other data architectural frameworks in the <a href="https://www.udemy.com/course/data-architecture-for-data-scientists/?referralCode=7D6C8D77978E7B1D2A55">best selling course</a> on Udemy on this topic.</p><figure><a href="https://www.udemy.com/course/data-architecture-for-data-scientists/?referralCode=7D6C8D77978E7B1D2A55"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HN-gs5vGgDdJGfzmKPE7gQ.png" /></a><figcaption>Learn more about Data Fabric in my course Data Architecture for Data Scientists</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae43fa337c6b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Contracts — Don’t Leave Quality to Chance]]></title>
            <link>https://medium.com/@biju.krishnan_73542/data-contracts-dont-leave-quality-to-chance-8f0d1029b3b3?source=rss-6b4c2196235------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f0d1029b3b3</guid>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[data-contract]]></category>
            <category><![CDATA[data-mesh]]></category>
            <dc:creator><![CDATA[Biju Krishnan @ DataSiens]]></dc:creator>
            <pubDate>Tue, 27 Feb 2024 12:27:32 GMT</pubDate>
            <atom:updated>2024-02-27T12:27:32.311Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Contracts — Don’t Leave Quality to Chance</h3><p>In the realm of data management, the concept of a data mesh has emerged as a pivotal framework for understanding how data flows and is utilized across different domains within an organization. At the heart of this framework lies the principle of data contracts, a mechanism designed to ensure seamless and reliable data exchange between the source domains (creators of data) and consumer domains (users of data). For data scientists navigating the complexities of modern data ecosystems, grasping these concepts is essential.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pbpUoN6dOPmWyjgK7lnZQw.png" /><figcaption>Data Contracts — Ensuring order in a decentralised framework like Data Mesh</figcaption></figure><p>Data contracts serve as the foundation for creating data products that are both independent and autonomous, facilitating a provider-consumer relationship between entities within an organization.</p><p>This structure, however, brings forth the challenge of ensuring that changes in one domain do not adversely affect another. For instance, a modification like the deletion of a column in a dataset by the source domain could cause applications or data pipelines in the consumer domain to fail. Data contracts emerge as a solution to mitigate such risks while preserving the autonomy of each domain.</p><p>Drawing parallels to real-life scenarios where contracts govern transactions like buying an apartment or leasing a car, data contracts establish a set of rules designed to maintain harmony between domains. These contracts can be verbal, written, or automated, each with its strengths and limitations.</p><h3>Verbal, Written, and Automated Data Contracts</h3><ul><li><em>Verbal contracts</em> involve informal agreements on data creation and maintenance to prevent pipeline disruptions. However, their reliability is often questionable in today’s fast-paced and complex data environments.</li><li><em>Written contracts</em> provide a more tangible solution by documenting the agreed-upon schema and data types. These documents, maintained on platforms like wiki pages or Confluence, offer clarity but lack enforceability.</li><li><em>Automated contracts</em> represent the optimal approach, acting as checks and balances that verify alignment with agreed-upon formats as data is created, written, and read. This automation prevents unplanned outages and ensures the integrity of data products.</li></ul><h3>An Example Data Contract in YAML</h3><p>To better understand the practical application of data contracts, let’s examine a sample data contract articulated in a YAML-like declarative syntax. This example focuses on validations for a dataset, specifically a timeseries dataset, ensuring its schema integrity and the reliability of its data.</p><pre>checks for timeseries:<br>  - schema:<br>  - invalid_count(email) = 0:<br>      valid format: email<br>  - valid_count(email) &gt; 0:<br>      valid format: email<br>  - failed rows:<br>      samples limit: 50<br>      fail condition: x &gt;= 3<br> - correlation:<br>  - avg_x_minus_y between -1 and 1:<br>      avg_x_minus_y expression: AVG(x - y)</pre><p>This YAML snippet outlines a series of validations:</p><ol><li>Email Validation Checks: These checks ensure that all email addresses in the dataset conform to a standard email format. The invalid_count(email) = 0 validation ensures there are no email addresses that deviate from the valid format, while valid_count(email) &gt; 0 confirms the presence of at least one legitimate email address in the dataset.</li><li>Failed Rows Check: This validation is designed to identify rows that meet or exceed a specific condition, in this case, x &gt;= 3, with a sample limit of 50 rows. This check helps in spotlighting potential outliers or anomalies within the dataset.</li><li>Correlation Check: Focusing on the relationship between two variables, this validation ensures that the average difference between x and y (avg_x_minus_y) remains within a range of -1 to 1. This is crucial for analyses that rely on the stability or consistency of these relationships over time.</li></ol><p>By incorporating such validations into data contracts, organizations can safeguard against data quality issues and foster a more reliable data ecosystem.</p><h3>The Future of Data Contracts</h3><p>Looking ahead, advancements in technology might enable large language models to autonomously generate data contracts based on dataset observations. Enforcing these automated data contracts becomes a critical step in ensuring that data exchanged between domains complies with established formats and agreements.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7I7vD8PpgWpRL0-V3wjnMA.png" /><figcaption>Depiction of data contracts in a data pipeline</figcaption></figure><p>When a source domain contributes data to a data lake or lakehouse, it must pass through these automated validations to ensure compliance. Similarly, when the target domain retrieves data, it undergoes verification to guarantee alignment with pre-established agreements.</p><h3>Conclusion</h3><p>For data scientists, understanding and implementing data contracts within a data mesh framework is crucial for navigating the complexities of data management and ensuring the integrity and reliability of data products. As the field continues to evolve, embracing automated data contracts will be key to fostering seamless collaboration between domains and enhancing the overall quality of data ecosystems.</p><figure><a href="https://www.udemy.com/course/data-architecture-for-data-scientists/?referralCode=7D6C8D77978E7B1D2A55"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uWApc6VxilotmxjkLJOKDQ.png" /></a><figcaption>Data Architecture for Data Scientists on Udemy</figcaption></figure><p>If you would like to have a quick overview of topics such as data lake, datawarehouse, data mesh and many more, do consider enrolling in my course on Udemy titled “Data Architecture for Data Scientists”.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f0d1029b3b3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Governance Maturity Assessment]]></title>
            <link>https://medium.com/@biju.krishnan_73542/ai-governance-maturity-assessment-35d7e1586611?source=rss-6b4c2196235------2</link>
            <guid isPermaLink="false">https://medium.com/p/35d7e1586611</guid>
            <category><![CDATA[ai-governance]]></category>
            <category><![CDATA[eu-ai-act]]></category>
            <category><![CDATA[ai-ethics]]></category>
            <dc:creator><![CDATA[Biju Krishnan @ DataSiens]]></dc:creator>
            <pubDate>Sun, 17 Sep 2023 11:27:38 GMT</pubDate>
            <atom:updated>2023-09-17T11:27:38.921Z</atom:updated>
            <content:encoded><![CDATA[<p>The blog first appeared on <a href="http://aiethicsassessor.com/blog/ai-governance-maturity-assessment">aiethicsassessor.com</a></p><h3>AI Governance Maturity Assessment Tool</h3><p>Artificial intelligence (AI) is being rapidly adopted by enterprises across industries. However, implementing AI comes with significant risks and challenges around ethics, bias, transparency, and accountability. That is why organisations need a thoughtful approach to AI governance — the processes, policies, and structures implemented to ensure AI is developed and used responsibly.</p><p>For quick assessment of your enterprises AI Governance maturity, try our <a href="https://aiethicsassessor.com/aigovernanceassessment">FREE AI Governance Maturity Assessment tool</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*DskMNzT80JyLrFnZE6NMwA.png" /><figcaption>Three levels of AI Governance maturity assessed by aiethicsassessor.com</figcaption></figure><p>AI governance maturity can be mapped into three levels: <em>Seeker, Steward, and Scaler</em>. Understanding where your organisation falls on this spectrum can help create a roadmap to advance your AI governance capabilities.</p><h3>The Seeker</h3><p>Organisations at the Seeker level have limited or no formal AI governance in place. AI projects are ad hoc and experimental in nature rather than aligned to strategic goals. There are no guidelines on how to develop, test or deploy AI responsibly.</p><p>Seekers launch AI pilots to explore potential use cases but they happen in silos without coordination. Data scientists have free rein to experiment with minimal oversight. AI ethics is an afterthought and bias mitigation is not baked into the process.</p><p>With AI still in proof-of-concept mode, Seekers can get away with a lax governance approach in the short-term. But the lack of guardrails means they are likely to encounter issues as projects scale. Hurriedly retrofitting governance at that point causes delays and cost overruns.</p><h3>The Steward</h3><p>Steward organisations understand the importance of AI governance and are working to build their capabilities. They have basic policies, processes and structures in place to align AI projects with business goals and ensure they create value.</p><p>Stewards establish cross-functional AI oversight teams comprising stakeholders from legal, compliance, IT, security, risk management, and the business. AI projects are reviewed to assess benefits and risks before being approved. Data collection, model development and deployment undergo audits for quality, bias, and ethical risks.</p><p>While not yet robust, Stewards’ governance practices promote responsibility across the AI lifecycle. Project teams adhere to model cards, ethics checklists and documentation standards that increase transparency. Key metrics are defined to track AI risks and returns.</p><p>By instilling accountability early, Stewards lay the foundation for AI projects that are not only performant but aligned with organisational values. But reliance on manual governance processes limits their ability to scale AI across the enterprise.</p><h3>The Scaler</h3><p>Scaler organisations have mature AI governance hardwired into their structure and culture. They take a holistic approach spanning policy, process, technology and people. AI strategy is integral to corporate strategy.</p><p>Scalers establish centers of excellence to provide AI development teams with tools, best practices, training and oversight. Automated pipelines, model inventories and testing frameworks provide technological guardrails without impeding innovation.</p><p>Comprehensive policies cover data management, model risk, target setting, human oversight, ethics reviews and mitigation of bias and other harms. Risk assessment is standardised using frameworks like Model Cards for Model Reporting. Portfolio management provides visibility into all AI projects in the pipeline.</p><p>With AI governance woven into the fabric of the organisation, Scalers have the foundations to deploy AI responsibly at scale. Anticipating risks keeps them from “governance debt” which could necessitate costly fixes later. Independent audits provide assurance to both internal and external stakeholders.</p><h3>Advancing Your AI Governance</h3><p>Assessing where your organisation falls on the AI governance maturity spectrum can spotlight gaps as well as strengths to build upon. Seekers have the opportunity to lay the right foundations instead of having to retrofit governance. Stewards can level up their policies, processes and tools to scale responsibly.</p><p>And even Scalers have room for improvement — governance must evolve continually as technology advances. Prioritising the well-being of people and society should be the true mark of AI maturity.</p><p>Here are some specific steps that organisations can take to improve their AI maturity:</p><ul><li><strong>Develop a clear AI strategy.</strong> This strategy should define the organization’s goals for AI, the resources that will be dedicated to AI initiatives, and the governance framework that will be used to manage AI projects.</li><li><strong>Establish a mature AI governance framework.</strong> This framework should address issues such as the ethical use of AI, the security of AI systems, and the compliance of AI systems with regulations.</li><li><strong>Scale AI initiatives across the organization.</strong> This means embedding AI into the organization’s core processes and decision-making systems.</li><li><strong>Measure the impact of AI initiatives.</strong> This will help organisations to understand the value that AI is delivering and to identify areas for improvement.</li></ul><h3>Tools For Advancing AI Governance</h3><p>Here are some tools that we at aiethicsassessor.com have gathered for you to advance your AI Governance maturity.</p><ol><li>We have created a simple maturity calculator to aid in understanding your current AI Governance maturity.</li><li>If you wish to learn about software tools available to streamline your AI Governance maturity, check out our blog post on the list of AI Governance software.</li><li>Try out our AI Verify playground to get hands on experience on tooling for AI Governance.</li></ol><p>And finally if you wish assistance from experts in improving your AI Governance Maturity, feel free to contact us using our contact form on the about page.</p><p><a href="https://aiethicsassessor.com/about">AI Ethics Assessor - About Us</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=35d7e1586611" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gamify Ethical AI Development]]></title>
            <link>https://medium.com/@biju.krishnan_73542/gamify-ethical-ai-development-20dfe382f306?source=rss-6b4c2196235------2</link>
            <guid isPermaLink="false">https://medium.com/p/20dfe382f306</guid>
            <category><![CDATA[eu-ai-act]]></category>
            <category><![CDATA[ethical-ai-development]]></category>
            <category><![CDATA[ai-ethics]]></category>
            <category><![CDATA[ieee-ai-ethics-assessment]]></category>
            <dc:creator><![CDATA[Biju Krishnan @ DataSiens]]></dc:creator>
            <pubDate>Sat, 16 Sep 2023 12:18:43 GMT</pubDate>
            <atom:updated>2023-09-16T12:20:10.050Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*elq0ig8IRu0XLKC9" /><figcaption>Photo by <a href="https://unsplash.com/@introspectivedsgn?utm_source=medium&amp;utm_medium=referral">Erik Mclean</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This blog post first appeared on <a href="https://x.com/Kwizera_sa/status/1669806176584753159?s=20">aiethicsassessor.com</a></p><p>Ethics in AI can often come across as a dull sermon to your development team, leaving many uninspired.</p><p>Here are some opinions found on social media suggesting the trend.</p><iframe src="https://cdn.embedly.com/widgets/media.html?type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=twitter&amp;url=https%3A//twitter.com/james_bohrman/status/1546063247278841856%3Fs%3D20&amp;image=https%3A//i.embed.ly/1/image%3Furl%3Dhttps%253A%252F%252Fabs.twimg.com%252Ferrors%252Flogo46x38.png%26key%3Da19fcc184b9711e1b4764040d3dc5c07" width="500" height="281" frameborder="0" scrolling="no"><a href="https://medium.com/media/c536f2c658923a612d374382bfe504cf/href">https://medium.com/media/c536f2c658923a612d374382bfe504cf/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=twitter&amp;url=https%3A//twitter.com/Qivshi1/status/1637513137170546694%3Fs%3D20&amp;image=https%3A//i.embed.ly/1/image%3Furl%3Dhttps%253A%252F%252Fabs.twimg.com%252Ferrors%252Flogo46x38.png%26key%3Da19fcc184b9711e1b4764040d3dc5c07" width="500" height="281" frameborder="0" scrolling="no"><a href="https://medium.com/media/5b68c24d6ced6b7ff6aa13f7d64d5032/href">https://medium.com/media/5b68c24d6ced6b7ff6aa13f7d64d5032/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?type=text%2Fhtml&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;schema=twitter&amp;url=https%3A//twitter.com/Kwizera_sa/status/1669806176584753159%3Fs%3D20&amp;image=https%3A//i.embed.ly/1/image%3Furl%3Dhttps%253A%252F%252Fabs.twimg.com%252Ferrors%252Flogo46x38.png%26key%3Da19fcc184b9711e1b4764040d3dc5c07" width="500" height="281" frameborder="0" scrolling="no"><a href="https://medium.com/media/3351a6e783bce7cbd8ae2b54593d4538/href">https://medium.com/media/3351a6e783bce7cbd8ae2b54593d4538/href</a></iframe><p>In this blog post, we’ll explore innovative ways to infuse the spirit of gaming into the process, making ethical considerations an integral and enjoyable part of AI product and service development. So, if you’re ready to transform ethics discussions into thrilling endeavours, read on to discover the power of gamification in shaping the future of AI.</p><h3>Microsoft Judgement Call</h3><p>Imagine a game that not only entertains but also serves a higher purpose. One that takes Microsoft’s core AI principles of fairness, privacy, security, reliability, transparency, inclusion, and accountability and transforms them into an engaging team-based activity. Welcome to “Judgment Call” — a unique experience designed to put these principles into action and foster empathy among stakeholders.</p><p>In my opinion this is the most relevant and complete toolkit out there for a good all round discussion on Ethical AI.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FLXqlAXEMGI0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DLXqlAXEMGI0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FLXqlAXEMGI0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a7f1d26830d3413ee04081622acc423c/href">https://medium.com/media/a7f1d26830d3413ee04081622acc423c/href</a></iframe><h3>Moral Machine by MIT</h3><p>If your aim is to foster sensitivity and ethical awareness within your team, look no further than MIT’s <a href="https://www.moralmachine.net/">Moral Machine</a>. This remarkable tool not only boasts a comprehensive catalog of pre-existing scenarios but also offers a versatile scenario designer.</p><p>The scenario designer empowers your colleagues to craft scenarios that your future products might encounter in real-world situations. These scenarios can then be utilized to evaluate the decision-making process of your peers when faced with similar ethical challenges. This invaluable feature facilitates the collection of diverse opinions, ultimately assisting you in refining the ethical responses of your AI products.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FXCO8ET66xE4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DXCO8ET66xE4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FXCO8ET66xE4%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8468e4a1cba816e6c939efd111469c9b/href">https://medium.com/media/8468e4a1cba816e6c939efd111469c9b/href</a></iframe><h3>EUR Dilemma Game</h3><p>Although originally intended to discuss ethical dilemmas in general, this mobile game can help product managers and ethical AI champions design scenarios that an AI product might face in the future and seek opinions and reactions from peers to find ways to eliminate potential harms.</p><p>The EUR dilemma game has been created by the Erasmus University of Roterdam and is available both on Android and Apple Store. It has a group mode, where a moderator can post scenarios and the participants are encouraged to provide decision input to complicated ethical scenarios which are later used in analysis.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FSKhT7qHh9T8%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DSKhT7qHh9T8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FSKhT7qHh9T8%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ceaf4fa83d7b81a51d5c8301e78d61a1/href">https://medium.com/media/ceaf4fa83d7b81a51d5c8301e78d61a1/href</a></iframe><h3>Conclusion</h3><p>In a world where the discussion of “Ethics in AI” can sometimes feel like a monotonous lecture, it’s essential to inject creativity and engagement into the conversation. As we’ve explored in this blog post, there are fascinating tools available that can transform ethical AI discussions into immersive experiences.</p><p>These tools not only make ethics discussions more enjoyable but also pave the way for well-rounded ethical AI solutions. So, embrace these innovative tools, and let the power of gamification reshape your approach to ethical AI in product and service development. It’s a journey that’s both insightful and transformative, enriching not only your team’s understanding but also the impact of your AI solutions on the world.</p><h3>Contact Us</h3><p>If you wish for some assistance with making Ethical AI development more fun and engaging, contact us using the online form on our website.</p><p><a href="https://aiethicsassessor.com/about">AI Ethics Assessor - About Us</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=20dfe382f306" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI ethics and governance assessment tools]]></title>
            <link>https://medium.com/@biju.krishnan_73542/ai-ethics-and-governance-assessment-tools-c97dd3130976?source=rss-6b4c2196235------2</link>
            <guid isPermaLink="false">https://medium.com/p/c97dd3130976</guid>
            <category><![CDATA[ai-ethics]]></category>
            <category><![CDATA[eu-ai-act]]></category>
            <category><![CDATA[ieee-ai-ethics-assessment]]></category>
            <category><![CDATA[ai-governance]]></category>
            <dc:creator><![CDATA[Biju Krishnan @ DataSiens]]></dc:creator>
            <pubDate>Sat, 16 Sep 2023 12:09:46 GMT</pubDate>
            <atom:updated>2023-09-16T12:11:13.531Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6KhqcghjvnFP_2b1" /><figcaption>Photo by <a href="https://unsplash.com/@tobiastu?utm_source=medium&amp;utm_medium=referral">Tobias Tullius</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This blog was first featured on <a href="http://aiethicsassessor.com/blog/ai-ethics-assessment-tools">aiethicsassessor.com</a>.</p><p>The upcoming EU AI Act will make it mandatory for organisations to audit their high risk AI systems primarily for ethical compliance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IB_NAY0AN5JUud-sqKyhmg.png" /><figcaption>EU AI Act (AIA) — Classification of AI Systems</figcaption></figure><p>Do you wish to find out how ready your organisation is to facilitate such audits?</p><p>Here are some FREE online tools to help you assess the maturity of your AI Governance and AI Ethics assessments.</p><h3>AIETHICSASSESSOR.COM Assessor</h3><p>Here at aiethicsassessor.com we have created a quick assessment tool to help you judge the maturity of your AI Governance.</p><p>The tool makes you respond to 15–20 questions and provides you an assessment of your maturity level based on the AI maturity pyramid we have created.</p><p>Access the assessor by clicking on the image below.</p><figure><a href="https://aiethicsassessor.com/aigovernanceassessment"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2v1eRH4sRelNncxjbZW_tA.png" /></a><figcaption>aiethicsassessor.com AI Maturity assessment</figcaption></figure><h3>GSMA AI Ethics assessment</h3><p>The GSMA is a global organisation unifying the mobile ecosystem to discover, develop and deliver innovation that helps business and society thrive. Although developed by Telco’s, the GSMA AI Ethics assessment could apply for any industry with some modification.</p><p>Click the image below to access the assessment tool which is also available open source.</p><figure><a href="https://gsma.github.io/bespoke_aiethics"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-kJTFelR-z_uZJZZrmLwSg.png" /></a><figcaption>GSMA AI Ethics Assessment</figcaption></figure><p>The <a href="https://www.gsma.com/betterfuture/wp-content/uploads/2022/01/The-Mobile-Industry-Ethics-Playbook_Feb-2022.pdf">GSMA AI Ethics playbook</a>, is also worth a read.</p><h3>Credo AI Governance Readiness survey</h3><p>Credo AI is an AI governance platform that helps companies adopt, scale, and govern AI safely and effectively. Credo offers a free online AI Governance assessment tool for prospects to assess their AI Governance maturity level.</p><figure><a href="https://www.credo.ai/ai-governance-readiness-survey"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CaPVlMHpx1scvUuBNrpZdg.png" /></a><figcaption>CredoAI AI Governance Readiness survey</figcaption></figure><h3>ALTAI (Assessment List for Trustworthy Artificial Intelligence)</h3><p>In 2018, the European Commission opened a process to select a group of experts in Artificial Intelligence (AI) coming from civil society, academia and the industry. As a result, the High-Level Expert Group on Artificial Intelligence (AI HLEG) was created in June 2018 with a total of 52 people from different countries of the European Union (EU). The main objective of this independent group is to provide support in the creation of the European Strategy for Artificial Intelligence with a vision on “ethical, secure and cutting- edge AI”.</p><p>One of the deliverables of this expert group was an assessment tool to assess the trust worthiness of AI systems. Accessibility to this system is unfortunately not straightforward.</p><p>Click on the image below to access the login page of the ALTAI system. One needs to create an account to access the assessment tool.</p><figure><a href="https://altai.insight-centre.org/Identity/Account/Login"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vstr0PiLfwgYKeF7hT058g.png" /></a><figcaption>Assessment List for Trustworthy Artificial Intelligence</figcaption></figure><h3>Conclusion</h3><p>In conclusion, as the impending EU AI Act draws closer, organisations are facing the necessity of auditing their high-risk AI systems for ethical compliance.</p><p>To help assess your organisations readiness for such audits, there are several FREE online tools available. Whether it’s the quick assessment provided by AIETHICSASSESSOR.COM, the adaptable GSMA AI Ethics assessment, Credo AI’s Governance Readiness survey, or the ALTAI system designed to assess the trustworthiness of AI, these resources offer valuable insights into your AI Governance and AI Ethics maturity.</p><p>Embracing these tools can pave the way for a more responsible and ethically sound AI landscape. So, are you ready to take the first step toward ethical AI practices? Explore these tools and assess your organization’s readiness today.</p><p>If you need assistance in establishing AI Governance and Ethics assessment or help in understanding the impact of the EU AI Act, contact me by filling the online form.</p><p><a href="https://aiethicsassessor.com/about">AI Ethics Assessor - About Us</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c97dd3130976" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The AWS CoPilot NowTo Guide — Gathering requirements]]></title>
            <link>https://medium.com/@biju.krishnan_73542/the-aws-copilot-nowto-guide-gathering-requirements-9e6b8f8c1f5f?source=rss-6b4c2196235------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e6b8f8c1f5f</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[aws-copilot]]></category>
            <category><![CDATA[ecs-fargate]]></category>
            <category><![CDATA[docker-compose]]></category>
            <category><![CDATA[iaac]]></category>
            <dc:creator><![CDATA[Biju Krishnan @ DataSiens]]></dc:creator>
            <pubDate>Mon, 28 Aug 2023 19:08:57 GMT</pubDate>
            <atom:updated>2023-08-28T19:08:57.169Z</atom:updated>
            <content:encoded><![CDATA[<h3>The AWS CoPilot NowTo Guide — Gathering requirements</h3><p>My main occupation is not DevOps or application development, rather I’m focussed on AI Solutions and AI Governance. I have experience as a SRE manager and solution architect and hence do have technical knowledge of how networks, containers, dns etc work. I also have deployed complex applications entirely using AWS Cloudformation.</p><p>I think knowing my background will help you appreciate how much AWS Copilot was able to help me in this task.</p><h3>Application overview</h3><p><a href="https://aiverifyfoundation.sg/what-is-ai-verify/">AI Verify</a> is an open source application developed by the Infocomm Media Development Authority (IMDA) of Singapore in line with their vision for AI governance.</p><p>The application is designed to be hosted on the practitioners desktop and be useful in checking machine learning models for bias &amp; fairness among other things.</p><p>Lets review the docker-compose.yaml file included in the apps repo and learn how the application can be deployed using AWS Copilot.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/13138729efbacc4fe77ac9cfbc325992/href">https://medium.com/media/13138729efbacc4fe77ac9cfbc325992/href</a></iframe><p>From the docker-compose.yaml we can observe that this application stack consists of FOUR microservices, portal, db, redis and test-engine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yz5I8aGKUcnO-QdrG_b7dw.png" /><figcaption>AI verify microservice architecture</figcaption></figure><p>While portal serves the frontend, the other three serve as the backend.</p><h3>To sidecar or not</h3><p>The copilot manifest is intended to deploy a single micro-service and not the whole application stack. Multiple containers can be included in the same manifest, in case one container is intending to serve as a <a href="https://aws.github.io/copilot-cli/docs/developing/sidecars/">sidecar</a>.</p><blockquote>Sidecars are additional containers that run along side the main container. They are usually used to perform peripheral tasks such as logging, configuration, or proxying requests.</blockquote><p>Now here is my 2 cents on when to and when not to use a sidecar.</p><p>I would recommend using a sidecar if you wish to isolate a helper process and yet share the filesystem from the main container with this helper process. I would NOT recommend using a sidecar if the containers are large as they will share the same 20GB ephermal volume that Fargate provides by default (Larger ephermal volumes can be provisioned using copilot).</p><p>In my case I did toy around making test-engine the sidecar of the portal container but since the containers are large, Fargate default ephermal volume ran out of storage and the processes crashed. Instead of provisioning a larger epehermal volume, I decided to use shared EFS filesystem which I consider to be a better architectural decision.</p><p>I’ll explain how I went about making storage persistent and share the filesystem between two containers in a another post.</p><p>So in the end I had FOUR seperate containers serving the four services.</p><h3>Things the manifest cannot do</h3><p>To understand what the manifest cannot do and that the docker-compose can. Lets look at the snippet of docker-compose.yaml from earlier and focus just on the section which provisions the portal container.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/28ca9e14f7f508e7af9f14ec71fde837/href">https://medium.com/media/28ca9e14f7f508e7af9f14ec71fde837/href</a></iframe><ol><li>Host port mapping: You can expose ports but as you are deploying the container on Fargate, its not necessary and also not possible to map the host port to the container port.</li><li>Container dependencies: The container dependencies are only possible between sidecar containers. If you are creating copilot pipelines for CI/CD, then I assume you’ll need to incorporate the logic of factoring dependencies there.</li><li>Working directory: It was not possible to set the working directory to run the command and hence I had to move the command step into the Dockerfile which builds the image.</li><li>Volume mappings : Since we are deploying the containers on a container service like fargate, there is no mapping of host volumes to container volumes. In this scenario I mounted an EFS volume on the /app directory for both portal and test-engine to share.</li><li>Networks: Networking works very different from docker on a host and I hope to write a post on this topic alone.</li></ol><p>In the end I ended up splitting the docker-compose.yaml into eight files and the file structure appears as shown in the table below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9dc7af42b15b45a2d3afcb50bf084f6c/href">https://medium.com/media/9dc7af42b15b45a2d3afcb50bf084f6c/href</a></iframe><p>I have also uploaded the files in my <a href="https://github.com/bijucyborg/aiverify/tree/main/setup-aiverify/aiverify-aws-ecs">fork</a> of the applications github repo. Feel free to poke around and let me know in the comments if you would do somethings better.</p><p>With the planning and preparation out of the way, lets now focus on deploying the stack on AWS Fargate. Since this post is already quite long this will be reserved for my next post.</p><p>Don’t forget to follow me so that you get updates as and when I continue this series. Wish you all a nice day ahead.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e6b8f8c1f5f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The AWS CoPilot NowTo Guide — Introduction]]></title>
            <link>https://medium.com/@biju.krishnan_73542/the-aws-copilot-nowto-guide-introduction-793e3f1bd132?source=rss-6b4c2196235------2</link>
            <guid isPermaLink="false">https://medium.com/p/793e3f1bd132</guid>
            <category><![CDATA[ecs-fargate]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[aws-copilot]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[iaac]]></category>
            <dc:creator><![CDATA[Biju Krishnan @ DataSiens]]></dc:creator>
            <pubDate>Mon, 28 Aug 2023 15:59:53 GMT</pubDate>
            <atom:updated>2023-08-28T19:14:23.476Z</atom:updated>
            <content:encoded><![CDATA[<h3>The AWS CoPilot NowTo Guide — Introduction</h3><p>In this blog series I will share my experiences of deploying a two tier application (frontend &amp; backend) with automation aided by AWS CoPilot, on AWS Fargate Spot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bwVHwQBn-PkdsbCkBukvVw.png" /><figcaption>The ship’s bridge</figcaption></figure><p>Some ground rules for application deployment that need no debates anymore.</p><ol><li>All applications should be deployed using automation. Infra as a Code (IaaC) is not a nice to have anymore, its a <strong>must</strong> in most enterprises.</li><li>Use of containers is the defacto standard for hosting applications.</li><li>Use of container services (ECS, Azure Container Services) where possible over Kubernetes to minimize complexity.</li><li>CI/CD ensures faster and yet reliable deployments, making platform releases a non event. (remember go-live days?).</li></ol><p>There do exist challenges in adoption of these principles though.</p><ol><li>Writing IaaC can be a time consuming affair and needs a lot of expertise. Traditionally there are dedicated DevOps engineers solely focussed on writing IaaC for deploying application and infrastructure in an automated fashion.</li><li>Interpreting IaaC code can be as difficult as interpreting application code. Every engineer seems to have their preferences.</li><li>Enforcing best architectural practices can be a challenge.</li><li>Even though developers can easily deploy containers on their own, building the entire application stack still requires architects and devops engineers. The hand-off from developers to devops can add additional time in the application going live.</li><li>CI/CD pipeline creation can be a cumbersome process and again subject to preferred styles by different developers.</li></ol><h3>AWS Copilot</h3><p>This is where AWS CoPilot comes in. Its targeted at aiding developers to deploy the entire application stack, including DNS, Load Balancers, Storage and CI/CD without having to rely on DevOps teams or write IaaC code.</p><p>Here are some excerpts from the landing page of <a href="https://aws.github.io/copilot-cli/">Copilot CLI.</a></p><blockquote>Instead of modeling individual resources, Copilot provides common cloud architectures: <a href="https://aws.github.io/copilot-cli/docs/concepts/services/#request-driven-web-service">Request-Driven Web Service</a>, <a href="https://aws.github.io/copilot-cli/docs/concepts/services/#load-balanced-web-service">Load Balanced Web Service</a>, <a href="https://aws.github.io/copilot-cli/docs/concepts/services/#backend-service">Backend Service</a>, <a href="https://aws.github.io/copilot-cli/docs/concepts/services/#worker-service">Worker Service</a>, <a href="https://aws.github.io/copilot-cli/docs/concepts/jobs/">Scheduled Job</a>. The necessary infrastructure is generated from the chosen pattern. Focus your time on writing business logic instead of connecting AWS resources.</blockquote><blockquote>No need to worry about gluing Copilot commands in a script to create an automated release process. Copilot provides commands to create multiple deployment <a href="https://aws.github.io/copilot-cli/docs/concepts/environments/">environments</a> in separate AWS accounts and regions, as well as creating an AWS CodePipeline <a href="https://aws.github.io/copilot-cli/docs/concepts/pipelines/">pipeline</a> to build your container images, deploy your services, and run automated tests.</blockquote><blockquote>Modeling, provisioning, and deploying services are only part of the application lifecycle for the developer. Copilot also supports workflows around troubleshooting and debugging to help when things go wrong. <a href="https://aws.github.io/copilot-cli/docs/commands/svc-logs/">Tail your logs</a>, <a href="https://aws.github.io/copilot-cli/docs/commands/svc-exec">get a shell</a> to a running container, <a href="https://aws.github.io/copilot-cli/docs/commands/svc-status/">view the health</a> of your services from the comfort of your terminal.</blockquote><p>There are however still some limitations which I was able to overcome thanks to to the helpful members of the Copilot <a href="https://matrix.to/#/#aws_copilot-cli:gitter.im">community</a>, and the challenges are infact my inspiration to write this blog series.</p><ol><li>The biggest hurdle I faced was that I had a docker-compose.yaml file already which worked fine when running on docker locally, but Copilot does not have any means of either interpreting it directly or a guide to help convert it. So I spent over a week in learning the ropes of implementing the same logic embedded in the docker-compose.yaml inside the Copilot manifests. (There is an experimental <a href="https://github.com/aws/copilot-cli/issues/1612">feature</a> in the making for a long time).</li><li>Creation and mounting of EFS has been made quite easy, however for me personally it was still an uphill battle to host containers on persistent EFS storage. I had to deploy the containers with local storage, then manually copy the contents to an EFS volumes and then deploy the container again with the EFS storage as the application volume.</li></ol><p>The next post in this blog series will focus on my requirements for hosting an open source application named “AI Verify”, whose source code can be found <a href="https://github.com/IMDA-BTG/aiverify">here</a>. This application is designed to be hosted on local docker and hence it was quite a challenge to host it on ECS and that too with automation. There was learning curve when it came to AWS Copilot, however I wouldn’t have achieved my objectives in such a short time if not for AWS Copilot.</p><p>Thanks to AWS Copilot, my application runs on AWS Fargate spot and has been deployed almost entirely with automation. There are some parts which I did not consider worthy of automation (lazy me), and I will make sure I highlight these in the blog series as well.</p><p>So follow along on my journey to automate the deployment of AI Verify on AWS Fargate with AWS Copilot CLI.</p><p>Click below for the next chapter on Gathering requirements.</p><p><a href="https://medium.com/@biju.krishnan_73542/the-aws-copilot-nowto-guide-gathering-requirements-9e6b8f8c1f5f">The AWS CoPilot NowTo Guide — Gathering requirements</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=793e3f1bd132" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hosting your own LLM’s? 10 things to consider.]]></title>
            <link>https://medium.com/@biju.krishnan_73542/hosting-your-own-llms-10-things-to-consider-69215e1e4aa8?source=rss-6b4c2196235------2</link>
            <guid isPermaLink="false">https://medium.com/p/69215e1e4aa8</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[llmops]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Biju Krishnan @ DataSiens]]></dc:creator>
            <pubDate>Tue, 25 Jul 2023 18:10:01 GMT</pubDate>
            <atom:updated>2023-07-25T18:10:01.549Z</atom:updated>
            <content:encoded><![CDATA[<p>In this blog series I plan to cover all that is needed to host your own LLMs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*sxhF0nH36to_J3tN" /><figcaption>LLM’s in an enterprise</figcaption></figure><h3><strong># 10 Things to Consider When Hosting Your Own LLM</strong></h3><p>Large language models (LLMs) like GPT-3 provide powerful AI capabilities, but deploying your own custom LLM service requires thorough planning. Here are 10 key aspects to think through:</p><p><strong>1. Fine Tuning</strong></p><p>A large language model (LLM) will need to be fine tuned with your own dataset to adapt it to your own use case. Fine tuning will most likely need specialised compute infra based on GPU, TPU’s etc.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ae0302bfc4cdac5c72fabff70ac4e116/href">https://medium.com/media/ae0302bfc4cdac5c72fabff70ac4e116/href</a></iframe><p><strong>2. Data Engineering</strong></p><p>Data needs to be in a format that the fine tuning tools require, for example causal language modelling requires two columns, prompt and answer or question and answer. The raw data is likely to be in the JSONL format and some of the tools mentioned above can ingest it directly in this format, but some need it to be converted to CSV.</p><p>If the conversation is in the form of a thread, then each line of conversation will also need a parent conversation id linked to it.</p><p>Here are some nice reads to help you understand the nitty gritties of data engineering for LLM’s.</p><ul><li>Folks are truefoundry have nicely outlined how to <a href="https://blog.truefoundry.com/training-fine-tuning-of-llms-with-your-own-data/">preprocessed</a> their confluence data for fine tuning their chat bot.</li><li><a href="https://www.flowrite.com/blog/dataset-engineering-llm-finetuning">Here</a> are some details from the clever people at Flowrite, specifically on dataset engineering for LLMs.</li><li><a href="https://github.com/h2oai/h2o-wizardlm">H2O Wizard LM</a> is an open source implementation of Wizard LM which helps convert documents into Q&amp;A pairs for LLM fine tuning.</li></ul><p><strong>3. Experiment Tracking</strong></p><p>Use ML tools like Weights &amp; Biases, Neptune, or WhyLabs to meticulously track fine tuning experiments, evaluate variations, and run fine tuning experiments. Continual experimentation leads to iterative improvements.</p><p><strong>4. Model Deployment</strong></p><p>Once trained, serve your model for predictions using Docker, Kubernetes, AWS SageMaker, Hugging Face Inference or other platforms tailored for scale and low latency.</p><p><strong>5. Orchestration</strong></p><p>Connect your LLM to internal systems like databases using orchestrators like LangChain, Microsoft’s Semantic Kernel, or Langflow. Proper orchestration weaves your model into business workflows.</p><p><strong>6. Monitoring &amp; Guardrails</strong></p><p>Carefully monitor all model outputs and have rigorous guardrails in place to catch inappropriate or dangerous responses. Use tools like NVIDIA NeMo, Microsoft’s Responsible AI Guidance, and Hugging Face’s Inference API.<br>Watch <a href="https://build.microsoft.com/en-US/sessions/70c6d334-0e4a-4235-ad57-92004b06d7e7?source=sessions">this talk</a> during Microsoft Build to understand how Microsoft is supporting its customer in serving generative AI responsibly.</p><p><strong>7. Vector Databases</strong></p><p>Store key data in vector databases like Weaviate, Pinecone, or Redis to quickly find relevant information to combine with model predictions.</p><p>You might not need fine tuning of models frequently if you could create embeddings of your knowledgebase and use the search results as input to the LLM model.</p><p><strong>8. Security</strong></p><p>Conduct extensive security reviews covering data protection, access controls, infrastructure vulnerabilities, and attack surface reduction. Follow <a href="https://owasp.org/www-project-top-10-for-large-language-model-applications/">guidance</a> from OWASP.</p><p><strong>9. Compliance</strong></p><p>Build thorough compliance checks and controls covering fairness, transparency, and responsible AI principles. Review FTC’s <a href="https://www.washingtonpost.com/documents/67a7081c-c770-4f05-a39e-9d02117e50e8.pdf">questionnaire</a> to OpenAI as a starting point.</p><p><strong>10. FinOps</strong></p><p>Closely manage costs in the cloud with FinOps tracking, monitoring, and optimization. LLM infrastructure can get expensive without careful cost governance.</p><p>Right now integrating all the tools and implementing guardrails is quite a task, however hyperscalers are accelerating their efforts to provide an integrated suite of services for you to launch an LLM service with your own data. It might be worth it to create an experimentation stack the hard way, so as to understand the nitty gritty of hosting LLMs within your own VPC or datacenter.</p><p>Follow me for a detailed write up on each of these aspects to be released every week.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=69215e1e4aa8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Free course — Data Architecture for Data Scientists]]></title>
            <link>https://medium.com/@biju.krishnan_73542/free-course-data-architecture-for-data-scientists-8ba02a868a93?source=rss-6b4c2196235------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ba02a868a93</guid>
            <category><![CDATA[free]]></category>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[course]]></category>
            <category><![CDATA[data-mesh]]></category>
            <category><![CDATA[data-scientist]]></category>
            <dc:creator><![CDATA[Biju Krishnan @ DataSiens]]></dc:creator>
            <pubDate>Mon, 17 Jul 2023 11:16:20 GMT</pubDate>
            <atom:updated>2025-08-10T18:08:17.776Z</atom:updated>
            <content:encoded><![CDATA[<h3>Free course — Data Architecture for Data Scientists</h3><p>I have converted the transcripts of my course “Data Architecture for Data Scientists” into a series of blog posts on medium. I offer this series for FREE to help data scientists expand their knowledge on this topic.</p><p>Data architecture is considered as an ancillary skill for data scientists, and hence I have purposely made this course concise so that you could complete the reading in approximately 1 hour or so. Although its an ancillary skill, its quite important to upskill on to put modely into production.</p><p>I hope you enjoy this course and also share it with your colleagues and friends. Bookmark this page so that you could follow the topics and follow me on medium to get updates when I add new topics to the course.</p><p>Course contents</p><ol><li><a href="https://medium.com/@biju.krishnan_73542/data-architecture-for-data-scientists-introduction-5a5932ad0d43">Introduction</a> to data architecture for data scientists</li><li><a href="https://medium.com/@biju.krishnan_73542/data-architecture-for-data-scientists-data-types-structured-data-22a69b52b097">Data Types — Structured data</a></li><li><a href="https://medium.com/@biju.krishnan_73542/unstructured-data-the-inevitable-d4978be02588">Data Types — Unstructured data</a></li><li><a href="https://medium.com/@biju.krishnan_73542/semi-structured-data-things-you-love-and-hate-at-the-same-time-20e171b1e5b7">Data Types — Semi-structured data</a></li><li><a href="https://medium.com/@biju.krishnan_73542/datawarehousing-old-but-still-relevant-552b543cec20">Datawarehousing for Data Scientists</a></li><li><a href="https://medium.com/@biju.krishnan_73542/data-lakes-an-easy-introduction-for-data-scientists-4eb38e22e3c0">Data Lakes for Data Scientists</a></li><li><a href="https://medium.com/@biju.krishnan_73542/data-lakehouse-made-easy-for-data-scientists-c291c04bdbf0">Data Lakehouse for Data Scientists</a></li><li><a href="https://medium.com/@biju.krishnan_73542/data-mesh-a-not-so-complicated-explanation-a191616e69fa">Data Mesh made simple for Data Scientists</a></li><li><a href="https://medium.com/@biju.krishnan_73542/streaming-data-in-data-science-29eaf37484b3">Streaming data for Data Scientists</a></li><li><a href="https://medium.com/@biju.krishnan_73542/do-you-really-need-feature-stores-d23639a78d35">Feature stores — a gentle introduction</a></li><li><a href="https://medium.com/@biju.krishnan_73542/vector-databases-a-gentle-introduction-9948ba535c14">Vector databases — an introduction</a></li></ol><p>If you’re interested in learning more about data architecture, I encourage you to enroll in my <a href="https://www.udemy.com/course/data-architecture-for-data-scientists/?couponCode=BESTPRICEAUG2025">course</a> on Udemy. The course covers all aspects of data architecture described above with a more engaging delivery.</p><figure><a href="https://www.udemy.com/course/data-architecture-for-data-scientists/?couponCode=BESTPRICEAUG2025"><img alt="" src="https://cdn-images-1.medium.com/max/700/0*Ce3Ov3EWQM3h2nc2.png" /></a><figcaption>Data Architecture for Data Scientists on Udemy</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ba02a868a93" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>