<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[fme DevOps Stories - Medium]]></title>
        <description><![CDATA[As developers and cloud engineers we share our insights, experiences and stories. - Medium]]></description>
        <link>https://medium.com/fme-developer-stories?source=rss----710d04106367---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>fme DevOps Stories - Medium</title>
            <link>https://medium.com/fme-developer-stories?source=rss----710d04106367---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 07:17:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/fme-developer-stories" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Managing AWS Lambda Versions with AWS Step Functions: A comprehensive guide]]></title>
            <link>https://medium.com/fme-developer-stories/managing-aws-lambda-versions-with-aws-step-functions-a-comprehensive-guide-7a0793869118?source=rss----710d04106367---4</link>
            <guid isPermaLink="false">https://medium.com/p/7a0793869118</guid>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[aws-step-functions]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Maureen Plank]]></dc:creator>
            <pubDate>Wed, 24 Apr 2024 06:43:59 GMT</pubDate>
            <atom:updated>2024-04-24T06:43:58.932Z</atom:updated>
            <content:encoded><![CDATA[<p>Each AWS account has a regional default storage limit for Lambda and Lambda Layers (Zip archives) of 75 GB. This sounds a lot, and it can even be increased by demand, but many functions combined with frequent deployments might lead to storage pollution quite quickly. Therefore, it makes sense to clean up old deployment packages regularly. This blogpost introduces an AWS Step Functions state machine that is designed to automate the identification and deletion of older Lambda function versions.</p><h3>Starting point</h3><p>AWS creates a new version for each lambda function deployment and keeps them all available — just in case an older one is needed in future. Every deployment package is stored and counts to the overall size limit of 75 GB. Even though this amount of storage is quite large it is advisable ( <a href="https://serverlessland.com/content/service/lambda/guides/aws-lambda-operator-guide/code-storage">and recommended by AWS</a>) to clean up older versions which are no longer needed on a regular basis.</p><p>As of now, AWS does not provide a direct solution to manage these versions, prompting developers to create custom solutions. One example of a custom solution is the use of a Lambda function like the <a href="https://serverlessrepo.aws.amazon.com/applications/us-east-1/374852340823/lambda-janitor">Lambda Janitor</a> by Yan Cui. Our idea for an implementation was a state machine — an automated and visual solution for the Lambda version management.</p><p>This state machine handles all Lambda functions (optionally targeting specific ones based on predefined criteria) and deletes / removes old versions based on threshold which determines how many versions should be kept.<br> A visual representation of the final workflow as well as a brief explanation of the most important steps is given below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/0*-EfRm_hGg0xvzvum.png" /></figure><h3>Retrieving and filtering the function names</h3><p>The workflow begins with calling and listing the names of all existing Lambda functions within the AWS region. The “lambda:listFunctions” API call returns up to 50 items per call and a marker token in case more results are available. To take this into account the <a href="https://serverlessland.com/workflows/pagination-loop">paginator pattern</a> after the “Iterate Functions” map state is being used.</p><p>For the first step we transform the result with the ResultSelector to extract only the function names. By doing that we get an output like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/0*dVITbwgWj4jtmFBE.png" /></figure><p>To iterate through the array of lambda function names in the list, we use a Map state, which allows us to set a concurrency limit of 5. This limit helps to control the number of simultaneous requests and prevents throttle calls.</p><p>In our case we just want to clean up versions of Lambda functions which belong to our project. Others which are managed by the platform team need to stay untouched.</p><p>The Lambda functions from our project have a specific prefix in the name, that makes it easy for us to filter through all functions in the region. To do that, we added a “Choice” step to determine whether the state machine should continue the version cleanup or ignore the current Lambda function. This prefix can also be retrieved from the state machine invocation event.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/473/0*Y-mnkd1zGG2ihqiD.png" /></figure><h3>Getting all the available versions</h3><p>In the following step, all available versions are retrieved using the “lambda:listVersionsByFunction” API call. The outcome of this step is shown below — a list of all versions and the name of the function currently processed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/0*8lFosP70yX2jPfoK.png" /></figure><p>Keeping the function name in this step and adding it to the ResultPath is mandatory because we will need it again in a few steps when deleting the versions.</p><p>The “lambda:listVersionsByFunction” API call returns up to 50 items and a marker token to retrieve more results. To take care of the pagination — the paginator pattern would have been required again. However, we have decided not to implement it to keep the workflow simpler.</p><p>There are not that many deployments in our environment that we would reach more than 50 versions per Lambda function before the workflow runs the next time. In addition, the workflow which is triggered by an AWS EventBridge scheduler could run more often to avoid this situation — for instance, every day instead of every week — just as an example.</p><h3>Removing the $LATEST version and get version count</h3><p>The Lambda API returns the versions in a way that the latest (called $LATEST<strong> 😊</strong>) one comes first followed by all other versions in ascending order — from oldest to newest. We decided not to rely on this implicit order but to exclude the $LATEST version explicitly from all further processing steps as well as to sort the remaining version numbers — just in case AWS is going to change the response format.</p><p>Unfortunately, AWS Step Functions does only provide a limited set of array and JSON processing functions — so called intrinsic functions. A Lambda function is required to perform the necessary work. Hopefully, AWS adds some more power to the intrinsic function’s palette so that these kind of simple helper functions will no longer be necessary in future.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/541/0*GqFLiVIK2P0m5fcI.png" /></figure><p>The outcome of this step consists of the lambda function name, the array of versions to be processed and their count.</p><h3>Check number of available versions and remove outdated ones</h3><p>In the next step, the state machine compares the number of available versions against a threshold. This is done with a “Choice” State. In our case we wanted to always keep the three most recent versions + the $LATEST one, so the “version_count” is compared with 3:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/0*RYZ_C9-fQW4ImCyG.png" /></figure><p>If the number of available versions exceeds the threshold we set, the oldest version (first position in the array) is deleted using the “lambda: deleteFunction” API call in the step “Delete oldest version”.</p><p>After that, a modification of the payload is required as the deleted version number and the version count itself needs to be adapted. A “Pass” state is used to remove the oldest version (the first position in the array with the versions) and to decrease the version counter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/0*iBnmsQ4_TvTFzKi9.png" /></figure><p>The execution keeps checking and deleting old versions until their number has reached the threshold value (three in this example).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/0*PsHiGHwcsghYqs3c.png" /></figure><p>One remaining thing to consider related to Lambda Aliases</p><p>A Lambda alias reference one or two function versions which cannot be deleted if the alias exists. It is possible to retrieve all aliases of a function via the “lambda:listAliases” API call. One way to make sure to take aliases into account would be to get all involved versions and to remove them from the versions array before the version deletion action. This requires some custom code in a Lambda function.</p><p>Another option — which we have implemented — consists in defining an error handler for the “Delete old version” step. This one captures the “Lambda.ResourceConflictException” which is thrown when trying to delete a version which cannot be deleted for instance because of an alias reference. This error handling makes sure that the state machine does not fail in case of alias references or other unforeseen problems when trying to perform the deletion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/0*svdyeRmqXBmVHITj.png" /></figure><p>The state machine finishes after all Lambda functions which have been supposed to be processed do not possess more than the most recent versions — all others have been deleted. This mechanism helps to prevent deployment package pollution when used regularly.</p><h3>Conclusion</h3><p>In this guide on managing AWS Lambda versions with AWS Step Functions, I’ve shown how developers can leverage automation to streamline the management of Lambda function versions. By automating these processes, teams can focus more on development rather than maintenance, maintaining operational efficiency and resource optimization. Overall, this solution provides a practical, visual tool for managing Lambda functions effectively, helping users navigate the complexities of cloud operations with greater ease.</p><p><em>Originally published at </em><a href="https://en.fme.de/blog/managing-aws-lambda-versions-with-aws-step-functions-a-comprehensive-guide/"><em>https://en.fme.de</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a0793869118" width="1" height="1" alt=""><hr><p><a href="https://medium.com/fme-developer-stories/managing-aws-lambda-versions-with-aws-step-functions-a-comprehensive-guide-7a0793869118">Managing AWS Lambda Versions with AWS Step Functions: A comprehensive guide</a> was originally published in <a href="https://medium.com/fme-developer-stories">fme DevOps Stories</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Custom Software: The key to success in the digital business world]]></title>
            <link>https://medium.com/fme-developer-stories/custom-software-the-key-to-success-in-the-digital-business-world-3d06f1c19c9f?source=rss----710d04106367---4</link>
            <guid isPermaLink="false">https://medium.com/p/3d06f1c19c9f</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[softeare-engineering]]></category>
            <category><![CDATA[custom-software]]></category>
            <dc:creator><![CDATA[Florian Maul]]></dc:creator>
            <pubDate>Fri, 29 Sep 2023 13:11:13 GMT</pubDate>
            <atom:updated>2023-09-29T13:11:13.269Z</atom:updated>
            <content:encoded><![CDATA[<p>In today’s fast-paced and highly competitive business landscape, organizations are constantly seeking ways to optimize their processes, enhance productivity, and deliver superior user experiences. One key consideration in this pursuit is the selection of software solutions. While existing software, Software-as-a-Service (SaaS) offerings, and enterprise software have their merits, custom software development stands out as a powerful approach to cater to the unique needs of businesses. In this blog post, we will explore the advantages of custom software and how it can empower companies to unlock their full potential.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*O6UjDyGAFJTjA5Ac.jpeg" /></figure><h3><strong>Tailored to Your Needs</strong></h3><p>One of the most significant advantages of custom software is its ability to be precisely tailored to meet your business requirements. Off-the-shelf software solutions, SaaS offerings, and enterprise software often come with a wide range of features that may not align perfectly with your organization’s needs. With custom software, you have the opportunity to work closely with developers to define and build a solution that addresses your specific pain points, streamlines your workflows, and enhances your overall efficiency.</p><blockquote>Consider a <strong>financial services firm</strong> requiring a customized risk management system to handle complex investment portfolios and regulatory compliance. In collaboration with experienced developers a solution can be created that is tailored to the firm’s specific requirements. The software can incorporate <strong>sophisticated algorithms and models</strong>, specific to the firm’s investment approach, enabling accurate risk assessments and scenario analysis. Integration with existing systems, such as trading platforms and data providers, consolidates real-time market data for a holistic risk management view. The custom software’s user interface can be designed to present relevant risk metrics and reports in a <strong>visually appealing and user-friendly </strong>manner, empowering efficient decision-making and providing a <strong>competitive edge</strong> in the financial industry.</blockquote><h3><strong>Scalability and Flexibility</strong></h3><p>As your business grows and evolves, so do your software requirements. Custom software can be designed to scale seamlessly with your organization, accommodating increased data volumes, user loads, and complex functionalities. Unlike pre-packaged solutions that may have limitations, custom software allows for ongoing customization and adaptability, ensuring that your technology keeps pace with your changing business needs.</p><blockquote>Let’s consider a rapidly growing<strong> e-commerce startup</strong> that anticipates a surge in user traffic and transaction volumes. A custom solution offers the advantage of scalability and flexibility to accommodate the organization’s evolving needs. The software can be designed to handle increasing user loads, ensuring seamless performance and minimal downtime during peak periods. Additionally, the solution can adapt to <strong>new features, integrations,</strong> and business processes as the e-commerce startup expands its offerings and enters new markets. This scalability and flexibility provide the <strong>agility</strong> needed to respond to market demands quickly and maintain a competitive edge,<strong> without being limited</strong> by the constraints of pre-packaged software or SaaS offerings.</blockquote><h3><strong>Enhanced User Experience</strong></h3><p>User experience is a critical factor in the success of any software solution. Custom software development places the user at the center of the design process, allowing for a highly personalized and intuitive interface. By understanding your users’ preferences, workflows, and pain points, custom software empowers you to create an engaging and efficient user experience that drives productivity and user satisfaction.</p><blockquote>Custom software solutions enable a <strong>healthcare </strong>organization for example to create a patient management system with an enhanced user experience. By working with UX/UI designers and developers, the software can be tailored to align with the <strong>unique needs </strong>and workflows of the organization and its patients. The solution can offer intuitive navigation, personalized dashboards, and seamless integration with existing systems, such as EHRs (Electronic Health Records) and appointment scheduling tools. This level of customization fosters <strong>trust</strong>, improves <strong>productivity</strong>, and enhances <strong>patient satisfaction</strong>, setting the organization apart from generic off-the-shelf software or enterprise solutions.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*psCcry5poFNnP77IUO8tUQ.jpeg" /></figure><h3><strong>Integration and Interoperability:</strong></h3><p>Today’s businesses often rely on a diverse set of software solutions to manage various aspects of their operations. Custom software offers the advantage of seamless integration with your existing systems and software infrastructure. By integrating disparate systems and applications, you can achieve a unified and holistic view of your business processes, eliminate duplicate data entry, and automate workflows, resulting in improved efficiency and reduced operational costs.</p><blockquote>Let’s consider a <strong>logistics company </strong>that needs to streamline its operations by integrating various systems, such as transportation management, warehouse management, and order fulfillment. Custom software development allows the company to build a <strong>unified</strong> <strong>platform</strong> that seamlessly integrates with these disparate systems. Through <strong>custom APIs </strong>and data connectors, the software can facilitate real-time data exchange, automate data synchronization, and eliminate the need for manual data entry or duplicate processes. This integration and interoperability enhance efficiency, reduce errors, and provide a holistic view of operations, enabling better decision-making and improved customer service. By leveraging custom software, a logistics company can optimize its business processes and unlock the <strong>full potential of its interconnected software ecosystem</strong>.</blockquote><h3><strong>Cost Efficiency</strong></h3><p>While it might seem counterintuitive, custom software development can offer long-term cost efficiency compared to off-the-shelf or SaaS solutions. While the upfront investment may be higher, custom software eliminates recurring licensing fees or subscription costs associated with pre-packaged software. Additionally, custom software provides the opportunity to build modular, future-proof solutions, reducing the need for frequent software replacements or extensive upgrades.</p><blockquote>Consider a <strong>medium-sized manufacturing company</strong> that relies on an off-the-shelf enterprise software for its inventory management needs. Over time, the company realizes that the software lacks certain critical features and requires expensive customization to meet their specific requirements. As a result, they face ongoing licensing fees and maintenance costs. By opting for custom software development, the company can make an upfront investment to build a tailored inventory management system that meets their exact needs. This <strong>eliminates the need for costly customizations</strong> and recurring <strong>licensing fees</strong> associated with off-the-shelf software. In the long run, the custom software proves to be a more cost-efficient solution as it eliminates unnecessary expenses and provides a <strong>future-proof platform </strong>that can adapt and grow with the company’s evolving needs without requiring frequent replacements or extensive upgrades.</blockquote><h3><strong>Competitive Advantage</strong></h3><p>In a crowded market, gaining a competitive edge is essential for success. Custom software can provide unique functionalities, tailored features, and optimized workflows that differentiate your business from competitors. By aligning technology with your specific business processes, you can streamline operations, reduce errors, and deliver a superior customer experience, giving your organization a significant advantage in the marketplace.</p><blockquote>Imagine a <strong>retail company </strong>operating in a highly competitive market. By investing in custom software development, the company can <strong>differentiate itself from competitors</strong> by offering unique functionalities and optimized workflows. For instance, the custom software can provide <strong>personalized</strong> customer recommendations based on browsing and purchase history, enhancing the shopping experience and driving customer loyalty. It can also incorporate advanced analytics to forecast demand and optimize inventory management, enabling the company to stay ahead of market trends and minimize stockouts or excess inventory. By aligning technology with specific business processes and customer needs, custom software empowers the retail company to <strong>deliver a superior customer experience</strong>, gain a competitive edge, and outperform rivals in the dynamic retail landscape.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JCofUXacDp_nzWBL0oP0TA.jpeg" /></figure><h3>Conclusion</h3><p><strong>Custom software</strong> development offers numerous <strong>advantages</strong> over existing solutions, SaaS offerings, and enterprise software. By leveraging custom software, businesses can <strong>unlock their full potential</strong>, optimizing processes, <strong>enhancing user experiences</strong>, and gaining a <strong>competitive edge</strong>. Through tailored solutions that align with your unique requirements, custom software empowers organizations to maximize efficiency, boost productivity, and achieve their business objectives in an ever-evolving digital landscape.</p><p><strong>Contact us today</strong> to discuss your unique needs and embark on a transformative journey towards tailored solutions and enhanced user experiences. Together, we can create a customized software solution that empowers your organization to thrive in the digital era.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d06f1c19c9f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/fme-developer-stories/custom-software-the-key-to-success-in-the-digital-business-world-3d06f1c19c9f">Custom Software: The key to success in the digital business world</a> was originally published in <a href="https://medium.com/fme-developer-stories">fme DevOps Stories</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Standing on the Shoulders of Giants with ElasticSearch (or OpenSearch)]]></title>
            <link>https://medium.com/fme-developer-stories/standing-on-the-shoulders-of-giants-with-elasticsearch-or-opensearch-d4cca06cbac6?source=rss----710d04106367---4</link>
            <guid isPermaLink="false">https://medium.com/p/d4cca06cbac6</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[search]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Florian Maul]]></dc:creator>
            <pubDate>Fri, 08 Sep 2023 13:33:28 GMT</pubDate>
            <atom:updated>2023-09-08T13:32:16.547Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NQpkYjkrNh7k74lr3Jjzkw.jpeg" /></figure><p>As a software engineer, I have had the opportunity to work with a diverse range of technologies, each with their unique strengths and weaknesses. However, ElasticSearch stands out as an exceptional tool that goes beyond the typical capabilities of traditional databases, offering a comprehensive API for data search and analysis. This technology, in my opinion, is akin to standing on the shoulders of giants, enabling us to see further and reach new heights in our development journey.</p><h3>A Modern Solution for Data Search and Analysis</h3><p>ElasticSearch as well as OpenSearch are a powerful real-time distributed search and analytics engines that allows for rapid exploration of your data. They are used for full-text search, structured search, analytics, and combinations of all three, making it a versatile tool for any project.</p><h4>The Power of Faceted Search</h4><p>One significant advantage of ElasticSearch is its support for faceted search. This feature categorizes search results into different subsets, or ‘facets’, greatly enhancing user experience by making data search faster and more efficient. In contrast, implementing faceted search in traditional databases can be challenging and time-consuming, underscoring the advantage of ElasticSearch.</p><h4>Scalability and Speed</h4><p>ElasticSearch is more than just a database. It’s designed as a distributed system, allowing you to scale your operations as your project grows. Its use of an inverted index allows it to retrieve data at impressive speeds, providing a level of search functionality that outpaces traditional databases.</p><h4>Comprehensive and Flexible API</h4><p>One of the primary strengths of ElasticSearch is its comprehensive and flexible API. With ElasticSearch, there’s no need to build your own API for data search and analysis. The API it provides supports a variety of operations such as bulk data upload, document updates, and complex queries, to name a few. This capability eliminates the need for significant development time and resources often associated with traditional databases.</p><h4>Real-Time Data Analysis</h4><p>Unlike traditional databases, ElasticSearch shines when it comes to real-time data analytics. In the modern world where data is being generated at a remarkable rate, the ability to analyze data in real time is invaluable. ElasticSearch can index data as it comes in and makes it searchable almost instantly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j5ZmlkLwGNed10JTmqlsXA.jpeg" /></figure><h3>In Conclusion</h3><p>ElasticSearch, with its speed, scalability, and comprehensive API, is truly like standing on the shoulders of giants. It enables us to build far better data driven applications in our projects by offering superior capabilities such as faceted search and real-time data analytics. As a software engineer, I believe that it is an invaluable asset and recommend it for any developer seeking to enhance their data search and analysis capabilities.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4cca06cbac6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/fme-developer-stories/standing-on-the-shoulders-of-giants-with-elasticsearch-or-opensearch-d4cca06cbac6">Standing on the Shoulders of Giants with ElasticSearch (or OpenSearch)</a> was originally published in <a href="https://medium.com/fme-developer-stories">fme DevOps Stories</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Azure CosmosDB — why technology choices matter]]></title>
            <link>https://medium.com/fme-developer-stories/azure-cosmosdb-why-technology-choices-matter-fb1eab3206c2?source=rss----710d04106367---4</link>
            <guid isPermaLink="false">https://medium.com/p/fb1eab3206c2</guid>
            <category><![CDATA[azure-cloud]]></category>
            <category><![CDATA[cosmosdb]]></category>
            <category><![CDATA[azure-cosmos-db]]></category>
            <dc:creator><![CDATA[Jens Goldhammer]]></dc:creator>
            <pubDate>Tue, 08 Aug 2023 08:08:55 GMT</pubDate>
            <atom:updated>2023-08-09T05:50:05.378Z</atom:updated>
            <content:encoded><![CDATA[<h3>Azure CosmosDB — why technology choices matter</h3><p>Some months ago, my colleague Florian and me joined a development team of one of our clients. We are involved as architects and engineers of the application used in their retail stores.</p><p>The client currently migrates the core software from runni ng decentralized in each retail store (with its own databases) to a central solution. They heavily invested in Microsoft Azure as a Cloud provider and are moving more and more workloads to the Azure Cloud.</p><p>The client currently uses MSSQL databases in combination with the open-source Firebird database and has started to migrate data into the cloud. They have decided to use Cosmos DB as standard database for all new services in the cloud some years ago as it was the cheapest choice from their point of view.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/423/0*GS1iIa_YmZd7wKUc.png" /></figure><p>Azure Cosmos DB is the solution of Microsoft for fast NoSQL databases. For those who live in the AWS (Amazon Web Services) world, Cosmos DB can be compared to the service DynamoDB. You can learn more about Cosmos DB here: <a href="https://azure.microsoft.com/en-us/products/cosmos-db">https://azure.microsoft.com/en-us/products/cosmos-db</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t-wA8kTzwsS11Yh6.jpeg" /></figure><p>Source: <a href="https://azure.microsoft.com/en-us/products/cosmos-db">https://azure.microsoft.com/en-us/products/cosmos-db</a></p><p>When working with Cosmos DB, you have to forget all the things you learned in the relational database world. To design a good data model, you need to learn how to design a data model depending on your future access patterns to your data, because the performance of Cosmos DB depends on the partitions. Therefore you must put more effort into the data modelling upfront. You can find more about here: <a href="https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/modeling-data">https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/modeling-data</a></p><p>Cosmos DB instances can be created on demand and can be used in many programming languages.</p><p>The unique point of Cosmos DB — in comparison to traditional relational databases — is the distribution of the stored data around the world, the on-demand scalability and the effortless way to get data out of it. Due to the ensured low response times Cosmos DB allows multiple use cases in the web, mobile, gaming and IoT applications to handle many reads and writes.</p><p>Further use cases can be found here: <a href="https://learn.microsoft.com/en/azure/cosmos-db/use-cases">https://learn.microsoft.com/en/azure/cosmos-db/use-cases</a></p><p>Having joined the project as an architect and engineer, I was critical of using Azure Cosmos DB from the start, as I am a big fan of relational databases, especially for transactional data. My Cosmos DB journey began with writing a centralized device service to store clients’ purchased devices. We have used Azure Functions to implement the business logic on top of Azure Cosmos DB to retrieve and store the data.</p><p>Microsoft allows their customers to create several Cosmos DB instances in one Azure tenant. You can compare it to a database which holds several database tables. These instances can be used to separate workloads for different teams, stages or use cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0pikatc5Ofcnq0Ks.png" /></figure><p>Of course, in the document database world you can model your data without a specific schema. You can create your own JSON structures which makes it flexible as well. Often the idea is to combine different data into one item to allow fast reads. To reference data in other domains, you can use unique identifiers like the property id in the customer object.</p><p>You can find more about the structure of Azure Cosmos DB here: <a href="https://learn.microsoft.com/en-us/azure/cosmos-db/account-databases-containers-items">https://learn.microsoft.com/en-us/azure/cosmos-db/account-databases-containers-items</a></p><p>Azure provides multiple ways to query data from Azure Cosmos DB.</p><p>The following interfaces are possible:</p><ul><li>NoSQL API</li><li>MongoDB API</li><li>Cassandra API</li><li>Gremlin API</li><li>Table API</li></ul><p>An overview can be found here: <a href="https://learn.microsoft.com/en-us/azure/cosmos-db/choose-api">https://learn.microsoft.com/en-us/azure/cosmos-db/choose-api</a></p><p>We started to use the NoSQL API in our project with the SQL like interface. It was an easier migration path coming from SQL based relational databases in comparison to the other interfaces. To access the data, you can also use the Azure Portal with the Data Explorer — the data explorer allows you to access your collections, query and manipulate data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mG3xCRSSgQvq6oZj.png" /></figure><p>Cosmos DB allows to import data flexibly with different APIs.</p><p>There are two options at the moment:</p><ul><li>Import via Azure Datafactory service which can be used out of the box</li><li>Import via custom CLI Tool which uses the cosmosDB API -&gt; this tool needs to be developed</li></ul><p>One way to import mass data is via Azure Data Factory which allows to pipeline data and map data from various sources and import into a Cosmos DB collection. We have used this mechanism a lot to transfer data from on-premises relational databases into the cloud and migrate data via pipelines into Cosmos DB collections.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/0*C0XueCA1QW_VDCiI.png" /></figure><p>Source: Azure Portal example pipeline</p><p>Azure Data Factory works quite well, is fast and very flexible, but has its own drawbacks and challenges. Unfortunately, this topic is a subject in itself, so we cannot go into more detail here.</p><p>In the past we have also written our own CLI tools; they are more flexible for the datamapping and can be reviewed easier by other team members. By using bulk import with parallel threads with the cosmos API you will be as fast as the importing data via Azure Datafactory.</p><p>You can find a list of available SDKs here: <a href="https://developer.azurecosmosdb.com/community/sdk">https://developer.azurecosmosdb.com/community/sdk</a></p><p>Azure Cosmos DB provides several APIs to retrieve the data out of the containers. We have decided to use the SQL interface to get data out in the used Azure functions.</p><p>You can for example take this SQL like query to select all devices of a customer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*m4XTq6KUE5WD9Zjm.png" /></figure><p>This looks familiar, right?</p><p>After some time, you notice that the SQL capabilities are limited as Azure Cosmos DB implements only a limited set of SQL specifications:</p><ul><li>Cosmos DB does not allow to join items from different collections — it only allows to join the item with itself which means that you need to read data separately from different collections to join your data. The documentation says that you have to change your data model if you have needs for joining.</li><li>Cosmos DB provides functions as well, but you may know only a few of them and sometimes in a completely different way as you may know from SQL. You have to learn the cosmos specific syntax as there is no standard for querying data in NoSQL databases.</li><li>Cosmos DB has limited capabilities for the group-by with having clause. Sometimes there are workarounds, sometimes not.</li><li>Cosmos DB supports Limit and Offset, but this is very slow (as it is implemented) and you should use continuation tokens instead. Why? If you are interested to understand this, read here: <a href="https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/query/offset-limit">https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/query/offset-limit</a> and <a href="https://stackoverflow.com/questions/58771772/cosmos-db-paging-performance-with-offset-and-limit">https://stackoverflow.com/questions/58771772/cosmos-db-paging-performance-with-offset-and-limit</a></li></ul><p>My experience was that you often found good workarounds or a complete cosmos specific way, but sometimes you didn’t find a solution which was a little bit frustrating.</p><p>Nevertheless, the most painful issue was that CosmosDB only report errors with following message: “One of the input values is invalid.” in your query without a useful hint.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/0*egxpqI11N_u_Pgqs.png" /></figure><p>In this case, I made the mistake of putting a semicolon at the end of the query.</p><p>Updating one or multiple rows in a relational database with one SQL statement is a common request for processing data.</p><p>Azure Cosmos DB exactly allows to update one item within a container by using multiple requests.</p><p>The procedure looks like this:</p><ul><li>Retrieve the whole document you want to update</li><li>Update the fields you want to update in your application code</li><li>Write back the whole document to Cosmos DB</li></ul><p>Microsoft provides a new SQL update API to update one item without reading it before. The syntax for updating data is driven by the JSON PATCH standard (https://jsonpatch.com). This feature was a long time in preview and now is generally available in Azure Cosmos DB.</p><p><a href="https://learn.microsoft.com/en-us/azure/cosmos-db/partial-document-update-getting-started?tabs=dotnet">https://learn.microsoft.com/en-us/azure/cosmos-db/partial-document-update-getting-started?tabs=dotnet</a></p><p>Mass updates to a larger set of documents cannot be done out of the box with one SQL statement. You must update each document separately. This limitation is a little bit surprising when you want to evolve your document schema.</p><p>Yes, you can write a tool based on the bulk API. But you probably know that updating a lot of data is slow and involves much more effort instead of writing a single update query like in the relational data world.</p><p>Deleting data in Azure Cosmos DB is generally possible by removing one item at a time via API. But there is unfortunately no support for SQL Delete-Statements!</p><p>In general all limitations for mass operations in Cosmos DB may have reasons — for instance, the guaranteed response times for any action in the Cosmos DB. Operations on a bigger set of data might lead to higher execution times.</p><p>But indeed this is an annoying point while writing and testing your software. Sometimes you need to remove specific data very quickly. One workaround is to drop the whole container and create your test data again, but often you have the case that you want to keep specific data in it.</p><p>For example, we had to remove two million entries from a collection to repeat a migration, but wanted to keep other data in the collection. This action took half an hour by using a developed tool.</p><p>Azure Cosmos DB provides a simple transactional concept. It allows to group a set of operations in a batch operation. Unfortunately, the batch concept is not integrated nicely into the API as it does not allow to wrap your code into a transactional block like interfaces to relational databases allow.</p><p>Additionally, transactions do not allow to update documents from different partitions which is understandable from a technical point of view, but this limits a lot. In our service we had the use case to update several documents at once from several partitions and we had to live without transactions in the end.</p><p>I was very surprised starting with Cosmos DB to find such limited resources, articles and tools around the platform. But I understood this situation quickly because Cosmos DB is an exclusive, commercial service of Microsoft and is not as popular as Amazon DynomoDB for example.</p><p>Microsoft itself provides limited tooling only:</p><ul><li>the azure portal with the data explorer (<a href="https://cosmos.azure.com/">https://cosmos.azure.com</a>)</li><li>a visual studio code extension <a href="https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-cosmosdb">https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-cosmosdb</a></li><li>a CosmosDB emulator which runs natively under Windows <a href="https://learn.microsoft.com/de-de/azure/cosmos-db/data-explorer">https://learn.microsoft.com/de-de/azure/cosmos-db/data-explorer</a></li></ul><p>The community has written some tooling:</p><ul><li>The Cosmos DB Explorer for Windows <a href="https://github.com/sachabruttin/Cosmos%20DBExplorer">https://github.com/sachabruttin/Cosmos DBExplorer</a></li><li>CosmicClone to clone data from one container to another <a href="https://github.com/microsoft/CosmicClone">https://github.com/microsoft/CosmicClone</a></li></ul><p>Unfortunately there is no big community around Cosmos DB like for example for PostgreSQL or MySQL/MariaDB. Anyways, most of the good, known database tools out there which are supporting several vendors do not support Azure Cosmos DB. Mostly because it works completely different than relational databases.</p><p>Additionally Azure Cosmos DB allows to use stored procedures. Wait — Are stored procedures not a thing from the last century? Why should we use it? Probably you will notice that you need to use stored procedures for some scenarios like a mass deletion of entries in your collection as this is not supported out of the box.</p><p>Writing stored procedures is supported in Cosmos DB with JavaScript. As you may know, testing becomes challenging for this kind of code and besides that, most of the backend developers in our team are not familiar with JavaScript. Due to these challenges, we decided not to use them within the application apart from administrative purposes!</p><p>There are many advanced topics for Cosmos DB like scaling, partition keys etc. — these topics need their own blog post. You can read more about that in the official documentation: <a href="https://learn.microsoft.com/en-us/azure/cosmos-db/">https://learn.microsoft.com/en-us/azure/cosmos-db/</a></p><p>Using a document-based database is not a no-brainer. Document-based databases like Azure Cosmos DB are not a replacement for relational databases and it was never the intention.</p><p>Yes, Azure Cosmos DB has its use cases:</p><ul><li>If you have the use case “write once — read many” (for example just store data with a stable structure), you can use it.</li><li>If you need global distribution of your data, you probably need it.</li></ul><p>The problem is that you do not have these requirements in general for business applications very oftem. In my opinion most of the applications do not need to scale this way (besides, you are Amazon, Microsoft, Netflix or another global player).</p><p>On the other side Azure Cosmos DB has some heavy limitations when working with the data, especially if you want to evolve your schema. If you want to store relational data within Cosmos DB and have a lot of changes in the data over time, Cosmos DB makes it very complicated and is currently not a good choice from my point of view.</p><p>Besides these considerations one task has become very, very important from the beginning: design how to model and to partition your data. But this is a story on its own.</p><ul><li><a href="https://medium.com/@yurexus/features-and-pitfalls-of-azure-cosmos-db-3b18c7831255">https://medium.com/@yurexus/features-and-pitfalls-of-azure-cosmos-db-3b18c7831255</a></li><li><a href="https://mattruma.com/adventures-with-azure-cosmos-db-limit-query-rus/">https://mattruma.com/adventures-with-azure-cosmos-db-limit-query-rus/</a></li><li><a href="https://blog.scooletz.com/2019/06/04/Cosmos%20DB-and-its-limitations">https://blog.scooletz.com/2019/06/04/Cosmos DB-and-its-limitations</a></li></ul><p><em>Originally published at </em><a href="https://content.fme.de/en/blog/technology-azure-cosmosdb"><em>https://content.fme.de</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb1eab3206c2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/fme-developer-stories/azure-cosmosdb-why-technology-choices-matter-fb1eab3206c2">Azure CosmosDB — why technology choices matter</a> was originally published in <a href="https://medium.com/fme-developer-stories">fme DevOps Stories</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Insights on API Design]]></title>
            <link>https://medium.com/fme-developer-stories/insights-on-api-design-cb0e942c0f0b?source=rss----710d04106367---4</link>
            <guid isPermaLink="false">https://medium.com/p/cb0e942c0f0b</guid>
            <category><![CDATA[api]]></category>
            <category><![CDATA[design-patterns]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Florian Maul]]></dc:creator>
            <pubDate>Tue, 08 Aug 2023 08:08:35 GMT</pubDate>
            <atom:updated>2023-08-08T08:08:35.677Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AnVauYIVYm9uTsQIRfdVBQ.jpeg" /></figure><p>As a seasoned full-stack developer, I’ve often found myself immersed in the challenging world of API design. During the creation and implementation of several projects, it quickly became evident how integral API design is to the backbone of any web application. API design isn’t just a necessary part of the implementation process, it’s a vital aspect that can make or break the longevity of your applications. Today, I like to share some fundamental concerns and best practices that must be considered for a future-proof API, particularly for web-based REST APIs.</p><ol><li><strong>Consistency</strong>: Consistency is the key to a comprehensible API. This applies not only to endpoints but also to request and response structures, naming conventions, and error handling. Any deviation can lead to confusion of fellow developers.</li><li><strong>Versioning</strong>: An API, much like any software, evolves over time. A prudent strategy for versioning is crucial. The objective? Ensuring backward compatibility so that any enhancements don’t unexpectedly break applications using older versions of the API.</li><li><strong>Authentication and Authorization</strong>: Ensuring that users or systems are authenticated correctly and have proper access rights is fundamental in API design. This way, you are guaranteeing they’re only tapping into the data and functionality that’s intended for them. At the very least APIs should always require API Keys to manage and control the consumers of the API.</li><li><strong>Error Handling:</strong> The utility of clear, descriptive error messages can’t be overstated. They can expedite the integration process and debugging, making your API a joy to work with.</li><li><strong>Rate Limiting:</strong> To avert potential abuse or overuse of your resources, rate limiting is an essential feature. This also acts as a shield, protecting your service from DDoS attacks.</li><li><strong>Documentation</strong>: For any API, a clear, comprehensive, and current documentation is a must. Ideally the documentation is created in the form of machine-readable OpenAPI specifications. OpenAPI provides a standard, language-agnostic interface to RESTful APIs, and can be used to automatically generate clients and server code. This saves a significant amount of time during development and eliminates potential manual errors. Moreover, it provides consistent, reliable, and up-to-date documentation that evolves together with the API.</li><li><strong>Security</strong>: The importance of security can never be overstated. APIs should be designed with stringent security protocols to prevent data breaches and attacks. This includes protective measures like HTTPS and careful validation and sanitation of inputs.</li><li><strong>Performance</strong>: As developers and users, we all prefer things that are fast and reliable. API design should prioritize a speedy response time to ensure user satisfaction.</li><li><strong>Scalability</strong>: The ability to scale up and handle increasing loads gracefully should be built into the API design from the outset. After all, we all aspire for our applications to grow.</li><li><strong>Usability</strong>: A user-friendly API encourages developers to use it. A simple and intuitive API makes it easier for developers to get up and running with it.</li><li><strong>Data Format:</strong> Whether you’re dealing with JSON or XML for data interchange, a decision should be made regarding which format to support or whether to support both.</li></ol><p>Addressing these fundamental points during the design phase ensures a robust, easy-to-use, and secure API that will stand the test of time. As we continue to engineer the web of the future, let’s place API design at the forefront, recognizing its role as a crucial building block of our digital architecture.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb0e942c0f0b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/fme-developer-stories/insights-on-api-design-cb0e942c0f0b">Insights on API Design</a> was originally published in <a href="https://medium.com/fme-developer-stories">fme DevOps Stories</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Microsoft Graph API- a practical example in python]]></title>
            <link>https://medium.com/fme-developer-stories/microsoft-graph-api-a-practical-example-in-python-28cbe5cc8ff6?source=rss----710d04106367---4</link>
            <guid isPermaLink="false">https://medium.com/p/28cbe5cc8ff6</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[microsoft-graph]]></category>
            <category><![CDATA[microsoft-azure]]></category>
            <dc:creator><![CDATA[Torben Bruns]]></dc:creator>
            <pubDate>Thu, 23 Mar 2023 13:07:02 GMT</pubDate>
            <atom:updated>2023-03-28T06:31:15.441Z</atom:updated>
            <content:encoded><![CDATA[<p>“Nothing is as constant as change.” Following this theme, Microsoft is planning to discontinue Azure AD Graph in 2023 and introduce something new: Microsoft Graph. It will not only replace the former API but also enhance it with new capabilities. Apart from interacting with Azure AD Graph, the new API can also communicate with Microsoft 365 products. If you want a successful pipeline run to post a message in a Microsoft Teams channel, Microsoft Graph can do it. And if an application needs to send emails to users, Microsoft Graph can also handle that.</p><p>To put it simply, Microsoft Graph is a REST-API and acts as gateway to numerous services Microsoft365 offers [1].</p><h3>Using Microsoft Graph in your environment</h3><p>To begin with, you need an active subscription for Microsoft 365. The actual plan does not matter, as even the Basic tier is sufficient. If you want to get a first look at the API’s capabilities, check out Microsoft Graph Explorer. (<a href="https://developer.microsoft.com/en-us/graph/graph-explorer">https://developer.microsoft.com/en-us/graph/graph-explorer</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*sdYz7F54fuehGgw8.png" /></figure><h3>Writing our own application</h3><p>If you want to create your own application, let’s get started. Let’s consider an application that monitors inventory stock. As soon as the stock falls below a certain number, an email should be sent to the orders team.</p><p>We will focus on the following things:</p><p>• Registering an application in Azure AD</p><p>• Setting up a Graph Client in Python</p><p>• Sending an email</p><p>The image below visualizes what we want to achieve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/0*c7JSLjrlptUXYQIA.png" /></figure><p>Monitoring the stock is not covered within this article.</p><h3>AzureAD Registration</h3><p>There are two types of permissions in AzureAD:</p><ul><li>Delegated permissons</li><li>Application permissions</li></ul><p>With delegated permissions the application acts as a logged in user like the Graph Explorer does. Application permissions on the other hand allow the app to act as own entity rather than on behalf of a user. Downside is that for this type of permission you need administrative rights.</p><p>After this short explanation on types of permissions in Azure let us begin with registering an application in AzureAD.</p><ol><li>Go to portal.azure.com and login with your credentials</li><li>Click on Azure Active Directory</li><li>From the left side select App Registrations</li><li>Click on New Registration and copy the configuration from below image</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/0*dXGmgFxhjK5761SO.png" /></figure><p>The supported account types can be adjusted to your needs.</p><ul><li>Click on the newly created app registration</li><li>Select Authentication from the menu on the right</li><li>Add a new Authentication of type Mobile and desktop application</li></ul><p>For our example to work enter below configuration:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/0*wcgrJYgcM-3PenpN.png" /></figure><ul><li>Switch the slider for Allow Public Client Flows to the “on” position and save</li><li>From the menu select Certificates &amp; Secrets</li><li>Add a new client secret and remember to save it as it is only shown once</li><li>Go to API permissions and select permissions like shown below</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/0*NVpfo-_TLWI1P-bn.png" /></figure><p>That is all, the configuration of the application in the Azure portal is done.</p><p>Save the following values for later:</p><ul><li>Client id</li><li>Client secret</li><li>Tenant id</li></ul><h3>Implementation</h3><p>For the purpose of simplicity, I used Python. Microsoft offers SDK’s for different languages like C#, Java, Go and PHP. Still, all that is necessary is implementing HTTP-Calls. If there is no SDK for your specific language, you are only losing some comfort.</p><p>Let us have a look at the source code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/0*wl0dDErgPMlOjk3Q.png" /></figure><p>The packages “msgraph” and “azure” make it relatively simple to implement a Microsoft Graph API client. First, a GraphClient is created, which then queries the API for a list of users. Then, we call the “send_mail” function, which takes a GraphClient and userlist as inputs. It sends an email with some example text on behalf of the first user found in the list using their Outlook account to the recipients listed under the keyword “toRecipients”. If you want to know the exact mechanism, please refer to Microsoft’s documentation [4].</p><p>A mail is not limited to plain text, it is also possible to send attachments through a call to the url</p><p><em>/users/{id | userPrincipalName}/mailFolders/{id}/messages/{id}/attachments</em></p><p>The result of the above call to the API looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*_ZOS0gclrrl-UYh5.png" /></figure><h3>Conclusion</h3><p>The Graph API is a powerful gateway to the services offered by Microsoft. There are numerous applications imaginable, such as status updates on pipeline runs through Teams, email notifications like in the example, or user management within Azure AD.</p><h3>Sources</h3><p>1. <a href="https://learn.microsoft.com/de-de/graph/overview">https://learn.microsoft.com/de-de/graph/overview</a></p><p>2. <a href="https://learn.microsoft.com/de-de/graph/outlook-things-to-know-about-send-mail">https://learn.microsoft.com/de-de/graph/outlook-things-to-know-about-send-mail</a></p><p>3. <a href="https://learn.microsoft.com/en-us/graph/api/resources/mail-api-overview?view=graph-rest-1.0">https://learn.microsoft.com/en-us/graph/api/resources/mail-api-overview?view=graph-rest-1.0</a></p><p>4. <a href="https://github.com/mattfeltonma/query-microsoft-graph">https://github.com/mattfeltonma/query-microsoft-graph</a></p><p>5. <a href="https://learn.microsoft.com/en-us/azure/active-directory/develop/permissions-consent-overview">https://learn.microsoft.com/en-us/azure/active-directory/develop/permissions-consent-overview</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28cbe5cc8ff6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/fme-developer-stories/microsoft-graph-api-a-practical-example-in-python-28cbe5cc8ff6">Microsoft Graph API- a practical example in python</a> was originally published in <a href="https://medium.com/fme-developer-stories">fme DevOps Stories</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recommender systems based on AWS Personalize]]></title>
            <link>https://medium.com/fme-developer-stories/recommender-systems-based-on-aws-personalize-6a11deb1f0ed?source=rss----710d04106367---4</link>
            <guid isPermaLink="false">https://medium.com/p/6a11deb1f0ed</guid>
            <category><![CDATA[aws-personalize]]></category>
            <category><![CDATA[recommender-systems]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[Jens Goldhammer]]></dc:creator>
            <pubDate>Tue, 27 Sep 2022 08:39:41 GMT</pubDate>
            <atom:updated>2022-09-27T08:49:20.804Z</atom:updated>
            <content:encoded><![CDATA[<p>With its Personalize service, AWS offers a complete solution for building and using recommendation systems in its own solutions. The service, which is now also offered in the Europe/Frankfurt region, has been available since 2019 and is constantly being improved. Only last year, major improvements in the area of filters were added to the product.</p><p>AWS Personalize allows customers to create recommendations based on an ML model for platform or product users. The following activities are abstracted and made particularly easy by AWS Personalize:</p><ul><li>Import business data into AWS Personalize</li><li>Continuous training of models with current data</li><li>Read recommendations with filtering capabilities from AWS Personalize</li></ul><p>You might think that recommendation systems based on machine learning are old news and that you can do it all yourself anyway. Machine learning has been around for a few years now, and with it the possibility of developing such recommendation systems yourself.</p><p>But the difference is: AWS Personalize takes the complete management of Machine Learning environments off the users’ hands and allows to take first steps here very quickly. And we don’t need the best ML experts on the team, because AWS Personalize takes a lot of more complex issues off our hands. Why it’s still good to understand Machine Learning is shown by the challenges.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/0*9GcPltNsqtcDTJVT.png" /><figcaption>Source: <a href="https://aws.amazon.com/personalize/">https://aws.amazon.com/personalize/</a></figcaption></figure><p>AWS Personalize makes it easier than ever for us to create recommendations. From a technical perspective, everything seems easy to master. We find challenges mainly in the clear delineation of the use case and the meaningfulness of the recommendations.</p><p>The recommendations should be as clearly delimited as possible for a use case. This influences both the selection of data and the structure of the data model and schema.</p><p>A recommendation system lives from the meaningfulness and topicality of the displayed recommendations. If, for example, I make recommendations to a user that he already knows, are 2 years old or are not relevant at all, I lose his interest and trust. Recommendations are therefore first viewed critically and must therefore be convincing from the outset, even if this is of course partly viewed subjectively.</p><p>Therefore, we need to ask the following questions from the beginning:</p><p>It is very important to constantly validate the recommendations created by AWS Personalize. At the start, it is important to validate the recommendations manually, i.e., to check randomly whether the recommendations appear meaningful to a user at all. Recommendation is therefore to start with a recommendation system whose validity can be easily checked. In order to give a user recommendations that he or she does not yet know, it is necessary to work a lot with recommendation filters, so that favourites of users or content that has already been seen do not appear again.</p><p>Now how do we make Personalize create recommendations for us? To do this, there are a few steps to complete.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/0*zPs9h7HscAn3BHa3.png" /><figcaption>Source: <a href="https://docs.aws.amazon.com/personalize/latest/dg/images/domain-how-it-works.png">https://docs.aws.amazon.com/personalize/latest/dg/images/domain-how-it-works.png</a></figcaption></figure><ul><li>First, you should select a domain that best matches the use case (1).</li><li>In case of a user-defined use case, data models are defined afterwards. Importing your own data into Personalize is done once or continuously based on the defined data models (2).</li><li>Amazon Personalize uses the imported data to train and provide recommendation models (3).</li><li>To query recommendations, both an HTTP-based real-time API for one user and batch jobs for multiple users can be integrated (4).</li></ul><p>Now let’s take a look at these data models.</p><h3>Data models</h3><p>In addition to the e-commerce and video use cases, AWS Personalize offers the option of mapping your own use cases (domain). The bottom line is that it is always about the following data sets:</p><ul><li>Users</li><li>Items</li><li>Interactions of the user with these items</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/0*WZSan4rxa0KddkPv.png" /><figcaption>Source: <a href="https://aws.amazon.com/blogs/architecture/automating-recommendation-engine-training-with-amazon-personalize-and-aws-glue/">https://aws.amazon.com/blogs/architecture/automating-recommendation-engine-training-with-amazon-personalize-and-aws-glue/</a></figcaption></figure><p>These datasets form a dataset group and are used as a whole in Personalize. Crucial here are the interactions that are necessary for most ML models and are used for training. A short example will illustrate this data model:</p><p>“A fme employee reads the blog post “AWS Partnership” on the social intranet and writes a comment below it.”</p><p><strong>Item</strong>: Blogpost “AWS Partnership</p><p><strong>Interactions</strong>: read | comment</p><p>For this data a developer can define his own schema — one schema each for Interactions, Users and Items.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/0*G_S5ByS2u6ik6CsF.png" /></figure><p>The following is an example schema for a user with 6 fields. These fields can be later used to get recommendations for content of specific users, e.g. users from a specific company or country.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/426/0*1nT3L-ThlPwLwGCu.png" /></figure><p>When importing data, this schema must then be followed. All three datasets have mandatory attributes (e.g. ID) as well as additional attributes that help to refine the ML model so that the recommendations can become even more precise. The additional attributes can be textual or categorical. They can also be used to filter recommendations.</p><p>However, there are a few restrictions in modeling that you need to be aware of, such as the restrictions on 1000 characters per metadata. This is especially important if you want to model lists of values.</p><p>Further info can be found <a href="https://docs.aws.amazon.com/personalize/latest/dg/custom-datasets-and-schemas.html">here</a>.</p><h3>Import data into Personalize</h3><p>The quality of recommendations are dependent on the data provided. But how does the data get into the system ?</p><p>The import of data always takes place in these data pots, so-called datasets (see above) — there is exactly 1 dataset each for Users, Items and Interactions. These datasets are combined in a dataset group.</p><p>To be able to train the ML model, the data sets have to be imported at the beginning (bulk import via S3). It is also possible to update the data continuously (via an API), which ensures that the model can always be improved.</p><p>When you start with AWS Personalize, you usually already have a lot of historical data in your own application. This is necessary because recommendations only “work” meaningfully once a certain amount of data is available ( as with any ML application).</p><p>Here it is recommended to use the bulk import APIs of AWS Personalize. For this, the data must first be stored in S3 in CSV format and according to the previously defined schema. Then you can start import jobs ( 1 per record) via AWS Console, AWS CLI, AWS API or AWS SDKs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/0*9rko19Km1BUHM2I5.png" /></figure><p>For continuous updating of the Users and Items datasets, AWS Personalize provides REST APIs that can be easily used with the AWS Client SDKs.</p><p>A so-called event tracker can be used for updating the interactions. This previously created tracker can be used for a large amount of events within a very short time to get data into the system via HTTP .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/0*cwo6gbp1m4eMV_ku.jpeg" /><figcaption>Source: <a href="https://aws.amazon.com/blogs/machine-learning/increasing-the-relevance-of-your-amazon-personalize-recommendations-by-leveraging-contextual-information/">https://aws.amazon.com/blogs/machine-learning/increasing-the-relevance-of-your-amazon-personalize-recommendations-by-leveraging-contextual-information/</a></figcaption></figure><h3>Train models</h3><p>Once the initial data is imported, AWS Personalize can now use this data in the form of the Dataset Group to train a model. To do this, you can first create a Solution, which is a “folder” for models. This sets the Recipe that Personalize should use.</p><p>The recipe represents the ML model, which is then later trained (as a Solution version) with user-defined data. There are different types of recipes that offer different types of recommendations. For example, <em>USER_PERSONALIZATION</em> provides personalized recommendations (from all items) and <em>PERSONALIZED_RANKING</em> can provide a list of items with rankings for a particular user. Some recipes use all three data sets and some use only parts of them (e.g. SIMS does not need user data).</p><p>After creating a solution, it can then be trained with the current state of the data sets, resulting in a solution version. Depending on the amount of data, this can take a little longer — our tests showed runtimes of around 45 minutes. A solution version is the fully trained model that can be used directly for batch inference jobs or as the basis for a campaign — a real-time API for recommendations.</p><h3>Use recommendations in your own application</h3><p>Now it’s time to integrate recommendations into our own application. AWS provides a REST interface that allows us to retrieve recommendations from AWS Personalize in real-time. This makes it easy for us to integrate with any system</p><p>Recommendations in AWS Personalize are always user-related. Recommendations can therefore look different for each user — but can also be the same for certain recipes, as in the case of “Popularity count”.</p><p>The response is a list of recommendations in the form of IDs of the recommended items, each with a score. The items are uniquely referenced via the ID.</p><p>These recommendations can now be evaluated in your own application, linked with the content from your own database and then displayed to the user in a user interface. The performance of the query (at least for smaller amounts of data) is so good that this query can be done live. However, one can also think about keeping the results of the query for a while per user, so as not to have to constantly request the service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/0*WOVHU2oifZHwYm90.png" /><figcaption>Example of a response for a recommendation request</figcaption></figure><p>If you need recommendations for a large number of users for mailings, batch jobs ( <strong>batch inference jobs </strong>) can efficiently create these recommendations in the background. These batch jobs can be “fed” with the UserIds — the result are recommendations for each user within one big JSON file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/0*GDUsIxVWkPwy1ZRa.png" /><figcaption>Example for a result of the batch inference jobs</figcaption></figure><h3>Is Personalize worth the effort?</h3><p>The pricing model of the service can be quite demanding, so it is advisable to define in advance a result that you want to achieve with appropriate recommendations and resulting follow-up activities or repeat business.</p><p>As a guide, to get individual recommendations for individual users in the Personalize Batch, we assume about 0.06 ct per recommendation for the user. That doesn’t sound like a lot, but with several hundred thousand users and individual recommendations, it’s part of the overall consideration. Depending on how often and to what extent batch runs for mailings etc. take place, it can get expensive. And the instances AWS uses for batch runs are very large and very fast. We created several batch jobs for mass exporting recommendations for 200k users for testing purposes. The batch jobs then ran overnight. We incurred costs of several hundred Euros — we had probably underestimated the numbers in the AWS Calculator a bit.</p><p>If referrals have a positive impact on the business and thus directly generate more sales for the customer, it can pay off very well. But what if my recommendations do not have a direct positive impact on my sales? One reason could be to bind customers more closely (subscription model) — in the long term, this will in turn lead to more sales, but perhaps not in the short term.</p><h3>Summary</h3><p>AWS Personalize is a service that makes it very easy to get started with recommendation systems. As a development team, all you have to do is deliver the data in the right format and pick up the recommendations. It doesn’t get much easier than that from a technical perspective.</p><p>AWS Personalize can therefore be used well to extend existing systems without having to make deep changes. With the ability to create custom data models and tune the different ML algorithms, you can apply AWS Personalize to a wide variety of scenarios.</p><p>The real work is in finding meaningful use cases, delineating them from one another, and providing the system with the right data.</p><p>As always, this comes at a price. Is it worth it for them? Let’s find out together.</p><h3>References and Links</h3><p>Below are a few more links to help dig deeper into the topic:</p><p>Official documentation: <a href="https://aws.amazon.com/de/personalize">https://aws.amazon.com/de/personalize</a></p><p>AWS Blogposts: <a href="https://aws.amazon.com/de/blogs/machine-learning/category/artificial-intelligence/amazon-personalize/">https://aws.amazon.com/de/blogs/machine-learning/category/artificial-intelligence/amazon-personalize/</a></p><p>AWS Personalize Best Practices: <a href="https://github.com/aws-samples/amazon-personalize-samples/blob/master/PersonalizeCheatSheet2.0.md">https://github.com/aws-samples/amazon-personalize-samples/blob/master/PersonalizeCheatSheet2.0.md</a></p><p>Efficiency of models: <a href="https://aws.amazon.com/de/blogs/machine-learning/using-a-b-testing-to-measure-the-efficacy-of-recommendations-generated-by-amazon-personalize/">https://aws.amazon.com/de/blogs/machine-learning/using-a-b-testing-to-measure-the-efficacy-of-recommendations-generated-by-amazon-personalize/</a></p><p>AWS Personalize Code Samples: <a href="https://github.com/aws-samples/personalization-apis">https://github.com/aws-samples/personalization-apis</a></p><p><em>Originally published at </em><a href="https://content.fme.de/en/blog/aws-personalize"><em>https://content.fme.de</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a11deb1f0ed" width="1" height="1" alt=""><hr><p><a href="https://medium.com/fme-developer-stories/recommender-systems-based-on-aws-personalize-6a11deb1f0ed">Recommender systems based on AWS Personalize</a> was originally published in <a href="https://medium.com/fme-developer-stories">fme DevOps Stories</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fortifying federated access to AWS via OIDC]]></title>
            <link>https://medium.com/fme-developer-stories/fortifying-federated-access-to-aws-via-oidc-89ba7d85dea?source=rss----710d04106367---4</link>
            <guid isPermaLink="false">https://medium.com/p/89ba7d85dea</guid>
            <dc:creator><![CDATA[Konstantin Troshin]]></dc:creator>
            <pubDate>Tue, 13 Sep 2022 14:45:43 GMT</pubDate>
            <atom:updated>2022-09-13T14:45:43.716Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/652/0*WPonDFTET8ln9oHw.jpg" /></figure><p>In order to avoid management of numerous long-term IAM users, AWS provides federated access options that include <a href="https://wiki.oasis-open.org/security/FrontPage">SAML2.0</a> and <a href="https://openid.net/connect/">OIDC</a> <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers.html"><strong>identity providers</strong> (IDP)</a><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers.html).">.</a> Whereas the SAML option is used by many of our customers and there are <a href="https://www.google.com/search?q=aws+saml+access+keycloak">numerous examples</a> of how to set it up, the e<a href="https://www.google.com/search?q=aws+oidc+access+keycloak">xamples of use of OIDC</a> are much scarcer. Thus, while selecting our own method of access federation, we decided to try OIDC out to get better understanding of its limits and advantages and be able to better advise our customers.</p><h3>Differences between SAML and OIDC identity federation</h3><p>To demonstrate the key differences between OIDC and SAML, I have created a small <a href="https://github.com/fmeAG/keycloak-aws">repo</a> that allows to deploy <a href="https://www.keycloak.org/">Keycloak</a> on an EC2 instance and then configure the SAML and OIDC clients to use with AWS. For those unfamiliar with Keycloak, it is an open source Identity and Access Management tool sponsored by RedHat and widely used by many of our customers and ourselves as an <strong>identity provider</strong>. Among other features, Keycloak supports SAML and OIDC protocols for identity management and provides user federation via LDAP that allows to use it with an existing user base from an Active Directory. After <a href="https://github.com/fmeAG/keycloak-aws#deployment">deployment</a> of Keycloak and configuring the <a href="https://github.com/fmeAG/keycloak-aws#access-to-aws-with-saml">SAML</a> and <a href="https://github.com/fmeAG/keycloak-aws#access-to-aws-via-oidc">OIDC</a> clients, we can use Keycloak to login into AWS.</p><p>The SAML login can be performed by going to <a href="https://auth.${tf_var_root_dn}/realms/awsfed/protocol/saml/clients/amazon-aws"><em>https://auth.${TF_VAR_root_dn}/realms/awsfed/protocol/saml/clients/amazon-aws</em></a> where <em>${TF_VAR_root_dn}</em> is the subdomain you need to create before the deployment. After entering the credentials for the user <strong>testuser</strong> that is created by the deployment scripts, we get redirected to the AWS console for the AWS account to which Keycloak has been deployed. If we would have assigned multiple roles to the same Keycloak group (or multiple groups to <strong>testuser</strong>), a page like the one below would appear (which would look familiar to everyone who already used SAML federation with AWS).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*60guHL7QEnB4ul4U.png" /></figure><p>If you like to experiment and have deployed everything from the <a href="https://github.com/fmeAG/keycloak-aws">repo</a>, you can go to the network tab of the development tools of the browser, find the <em>saml</em> document there and copy its contents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*anDBKoRc4vTe4Vpq.png" /></figure><p>Save the contents as <em>aws-saml/assertion</em> and run the <em>saml.sh</em> from the same folder. If you are fast enough (per default, the SAML assertion for AWS is valid only for 5 minutes), the assuming should work for the first role but fail for the second. If you look at the trust policies for the corresponding roles (whose names should end with <em>_Federated_Admin-SAML</em> and <em>_Federated_Admin-SAML2</em>, respectively), you will see that those are identical and allow the <em>AssumeRoleWithSAML</em> operation for the same SAML provider. So, why is access granted for the first and denied for the second role? This is because AWS actually checks the SAML assertion itself for the presence of the role that you try to assume. Looking at the <a href="https://github.com/fmeAG/keycloak-aws/blob/main/aws-saml/kccommands.sh">script</a> we ran to configure Keycloak, we can see these two lines:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mg70_XoSNBataAjbRNAr8w.png" /></figure><p>These lines create an entry for the first role (the one without 2) in Keycloak and map this role to a group <strong>aws_access</strong> that is later assigned to our testuser. Thus, this role shows up in the SAML assertion and can be assumed. Since the same thing does not happen for the second role, the access to it is denied to <strong>testuser </strong>(of course, this would change if you created the corresponding entry and mapping in Keycloak for this one too).</p><p>But what about OIDC? Running the <em>./oidc.sh</em> script from the aws-oidc folder, we can see that our <strong>testuser</strong> can assume the role for which our OIDC provider is listed in the trust policy. A closer look at <a href="https://github.com/fmeAG/keycloak-aws/blob/main/aws-oidc/iam.tf#L5">this policy</a> shows that it contains only two things: the ARN of the OIDC provider and the <strong>client ID</strong> as aud. This corresponds to what AWS Console is doing if a role is created there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ROjT9e5xNoqtx2cv.png" /></figure><p>Also note that (as opposed to the SAML case), there was no need to do anything in Keycloak after running terraform scripts in the <em>aws-oidc</em> folder. What does this mean? Well, in the case of OIDC, AWS does not check for any role or group assignments in the <strong>ID token</strong>. The only two things that matter with the default settings are the <strong>IDP</strong> itself (which is defined by the URL and the thumbprint as you can clearly see from the <a href="https://github.com/fmeAG/keycloak-aws/blob/main/aws-oidc/openid.tf">openid.tf </a>file) and the <strong>client ID</strong> (defined in the <em>aud </em>section of the trust policy).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wtVcbeKd3hTaq6bWy90dJw.png" /></figure><p>This all means that any user that has access to the corresponding Keycloak realm can assume any role that trusts the <strong>IDP</strong> which is not very granular or secure and way inferior to SAML, right? Well, that would be so if not for a very important thing — the way I used OIDC in this example is not how it is supposed to be used. Let’s look at the oidc.sh script more closely.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YmSSlKnMeC6UpxI8QTwDZA.png" /></figure><p>Here, I use <em>kcadm.sh</em> (which is containerized and kind of hidden behind <em>source ../kcadm.sh</em>) to get the <strong>client secret</strong> for the Keycloak OIDC client. This operation requires admin rights and would be equal to a Keycloak administrator giving a <strong>client secret</strong> to a user in a regular context. This secret is then used together with the username and password for <strong>testuser</strong> to directly get the <strong>ID token</strong> that is in turn submitted to AWS STS. Of course, as a Keycloak admin I would never do this in the non-test environment because the <strong>client secret</strong> (which is bound to the <strong>client ID</strong> that is, in turn, specified in the IAM trust policy) is not meant to be available for the users. But what is it for then? Looking at the <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html">AWS documentation on the OIDC topic</a> , we can see that it mentions an <strong>identity broker</strong>. And this <strong>identity broker</strong> (which is not provided by AWS as in the case of SAML) is actually what the <strong>client ID and secret</strong> are destined for.</p><p>So, what is an <strong>identity broker</strong> anyway? An <strong>identity broker (IB)</strong> is an application that should function as a link between AWS and Keycloak and take over the management of user rights (it should know which user should be able to assume what role). A proper OIDC login flow should be started by the <strong>IB </strong>that redirects the user to the <strong>IDP</strong> (Keycloak in our case) which, after verifying the user credentials, provides the <strong>ID Token</strong> for that user to the <strong>IB. </strong>The<strong> IB </strong>uses <strong>client ID and secret</strong> to authenticate itself against the <strong>IDP</strong>. As you also can see from the <em>oidc.sh</em> script, it would be a bad idea to provide the <strong>ID token</strong> to the user because a combination of the role ARN and the <strong>ID token</strong> is all you need to assume a role with OIDC. Instead, the <strong>IB</strong> should check if the user has access to a requested role and then use the <strong>ID token</strong> to get the temporary AWS credentials (by using the <em>AssumeRoleWithWebIdentity</em> operation) and then return these credentials to the user (or use them to get the login URL for the AWS console). In my demo above, I use cURL as an <strong>IB</strong> which is obviously a very poor choice for a production environment since it grants access to any role to any user.</p><h3>Hardening the OIDC-based roles</h3><p>Whereas use of a proper <strong>identity broker</strong> minimizes the risk of the OIDC access to AWS being misused, the experiments above brought me to the question whether it is possible to get AWS STS to look at the user attributes from the <strong>ID token</strong> and not only at the <strong>client ID</strong> (<em>aud</em>) and the <strong>IDP</strong> itself. Looking at the <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-idp_oidc.html">documentation for GitHub</a> (which also uses OIDC) as IDP, I saw that there is another attribute — <em>sub</em> — that is used in trust policies. For Keycloak, the default value of <em>sub </em>is the user ID, which is not very useful, but Keycloak has mappers that can be assigned to clients and can override the defaults. Experimenting with mappers, I discovered that it is indeed possible to get Keycloak to provide any LDAP user attribute (we use LDAP user federation in our environment) as <em>sub</em> to AWS. The only caveat here is that this attribute needs to be a string, so that it is not directly possible to use group memberships (which would be arrays) to additionally secure the trust policies. It is, however, possible to use the <em>StringLike</em> operator to match substrings. Using this operator, it is possible to check for LDAP groups with AWS STS as long as those are stringified. For instance, the following trust policy checks for a certain group (provided by terraform as ${var.group}) in a group string looking like this: <em>-group1-group2-group3-</em>…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8deJFTAjV01mDRZoY-5WLw.png" /></figure><p>So, what could this group string come from? One option would be to write a custom plugin for Keycloak and another would be to let the <strong>IB </strong>(which is a custom app) handle this. My <a href="https://github.com/fmeAG/keycloak-aws">repo</a> actually contains such a custom mapper (next section also discusses that a bit more in detail) that should be active in your Keycloak if you deployed it as described above. To see the mapper in action, we can run the <em>./oidc_protected.sh</em> script from the <em>aws-oidc</em> folder. As you will see, you would be able to assume the first role but not the second one.</p><p>Why? Let’s take a look at the <a href="https://github.com/fmeAG/keycloak-aws/blob/main/aws-oidc/openid.tf">trust policies</a><a href="https://github.com/fmeAG/keycloak-aws/blob/main/aws-oidc/openid.tf):">:</a> the one for the first role contains the aws_access group which as we know is assigned to our <strong>testuser</strong>, the one for the second role refers to the <strong>aws_access_exclusive</strong> group which does not even exist in Keycloak yet. So, even though our user had a valid <strong>ID Token</strong>, it was not possible to assume a protected role because this token did not contain the correct group. If you want to verify that the access will be granted once you create the corresponding group and assign it to the testuser and also to look at the new Keycloak UI (which is at preview for Keycloak 18.0.2), you can do so at <a href="https://auth.${tf_var_root_dn}./"><em>https://auth.${TF_VAR_root_dn}</em>.</a> In this case, you would need to use the admin credentials (<strong>admin</strong> and <strong>TF_VAR_keycloak_password</strong> defined in export.sh). Once the group is created and assigned, the access works as expected. Sweet!</p><h3>Developing custom mappers for Keycloak</h3><p>Keycloak documentation <a href="https://www.keycloak.org/docs/latest/server_development/#_script_providers">mentions</a> “JavaScript” providers which can be used to create custom mappers. As I read JavaScript, I was expecting to write something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-93Eyfy4-fqHYICaBZIBRA.png" /></figure><p>and then place this into a .jar file as described in the documentation. It turned out that it does not work like that at all. Firstly, the custom scripts are disabled by default as I found out looking at the Keycloak logs. To fix this, one needs either to activate the preview functions or enable the <em>scripts</em> option alone as described <a href="https://github.com/keycloak/keycloak-documentation/blob/main/server_installation/topics/profiles.adoc">here</a><a href="https://github.com/keycloak/keycloak-documentation/blob/main/server_installation/topics/profiles.adoc).">.</a> Secondly, “JavaScript” turned out to be very Java-based and needs to call functions from the corresponding Java classes of Keycloak:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1QOoxtGCeyWY-obPQNjwgw.png" /></figure><p>The <a href="https://github.com/fmeAG/keycloak-aws">repo</a> shows how it all comes together.</p><h3>Conclusions</h3><p>In conclusion, both SAML2 and OIDC are great options of access federation for AWS and have their advantages and drawbacks. If you decide to use OIDC like us, you need an <strong>identity broker (IB)</strong> that provides a link between an <strong>IDP</strong> (such as Keycloak) and AWS. It would be unwise and potentially dangerous to provide <strong>ID tokens</strong> directly to the federated users, because a combination of such a token with a role ARN is usually enough to be able to assume that role. Of course, it would be even more unwise to provide an AWS-trusted <strong>client ID</strong> to the users. A combination of the StringLike operator and the Keycloak mappers can be used to increase the security of OIDC-Federated AWS accounts by restricting the access to the roles to certain user attributes such as group membership similarly to how the SAML2 federation works.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=89ba7d85dea" width="1" height="1" alt=""><hr><p><a href="https://medium.com/fme-developer-stories/fortifying-federated-access-to-aws-via-oidc-89ba7d85dea">Fortifying federated access to AWS via OIDC</a> was originally published in <a href="https://medium.com/fme-developer-stories">fme DevOps Stories</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Certbot as an init container for AWS ECS]]></title>
            <link>https://medium.com/fme-developer-stories/certbot-as-an-init-container-for-aws-ecs-b100f2f0b493?source=rss----710d04106367---4</link>
            <guid isPermaLink="false">https://medium.com/p/b100f2f0b493</guid>
            <category><![CDATA[certbot]]></category>
            <category><![CDATA[aws-ecs]]></category>
            <category><![CDATA[tls-certificate]]></category>
            <category><![CDATA[tls]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Konstantin Troshin]]></dc:creator>
            <pubDate>Sun, 31 Jul 2022 17:08:22 GMT</pubDate>
            <atom:updated>2022-07-31T17:08:22.468Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OhIU8POYsBwiEcyfs8DVyA.png" /></figure><p>Encryption in transit has become a security standard for most network-based applications and is requested by the majority of our customers for all applications we help them to build or manage. Most of the modern applications support TLS out of the box but require the certificate and the corresponding private key to be provided externally. In some cases (for example, for intranet apps), self-signed certificates (or certificates signed by an internal CA) are sufficient, but if the application is internet-facing and needs to be used without additional steps on the client side, a certificate signed by a commonly trusted certificate authority (CA) is required. For AWS-based applications (as you may have guessed from the title, AWS are a main focus of this post), AWS Certificate Manager (ACM) can be used in combination with a load balancer to provide an amazon-signed certificate. This simple and efficient method is not applicable, however, if the certificate and the corresponding private key need to be provided to the application directly instead of an AWS-managed load balancer. This can be the case if the application is using TLS in combination with its own protocol which would make TLS termination on the load balancer impossible. <a href="https://letsencrypt.org/">Let’s Encrypt</a> is an open CA that provides trusted certificates which can be acquired by using a tool that supports the ACME protocol. In this case, the certificate and private key can then be provided to the application directly and used also for custom TLS-based protocols. <a href="https://certbot.eff.org/">Certbot</a> is one of such tools and can be used to obtain the TLS credentials.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*OpRieNEztQRGaAqB4s7kKQ.png" /></figure><h3>The use case</h3><p>Recently, I have been asked to provide a publically accessible <a href="https://neo4j.com/product/neo4j-graph-database">Neo4j</a> database to use for development purposes. Since a Neo4j installation is available as a docker container, I chose to use AWS ECS to run it (a Kubernetes-based solution such as EKS would be quite an overkill for such a simple use case). To start things up, I deployed a <a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html">Network Load Balancer</a> (NLB) with three listeners and the corresponding target groups for ports 7474 (HTTP), 7473 (HTTPS), and 7687 (bolt). To improve security of the database, I decided to activate TLS for the bolt and HTTPS endpoints. Neo4j provides <a href="https://neo4j.com/developer/kb/setting-up-ssl-with-docker/">support for both out of the box</a><a href="https://neo4j.com/developer/kb/setting-up-ssl-with-docker/),">,</a> but requires the certificates to be provided externally. My initial approach was to use TLS listeners in combination with an Amazon-signed ACM certificate and TLS target groups to talk to the Neo4j container. I used openssl to create a self-signed certificate and provided it via an ECS mount point to Neo4j. This worked just fine for the HTTPS endpoint but did not for bolt which is, however, crucial for the Neo4j clients. It has become clear that TLS termination would not work for this use case and that I needed to use TCP listeners and target groups and to provide the publically facing certificate directly to Neo4j. Since the request of the customer included a wish that the database can be easily accessed by the clients without much configuration on their side, I also wanted this certificate to be publically trusted. In many of our k8s-based projects, we use cert-manager which can directly obtain Let’s Encrypt (LE) certificates, which brought me to the idea of using LE for my current task. Thinking about k8s and init containers, I also remembered reading some stuff about container dependencies in <a href="https://aws.amazon.com/about-aws/whats-new/2019/03/amazon-ecs-introduces-enhanced-container-dependency-management/?nc1=h_ls">ECS</a>, so I came to an idea of using a certbot docker container as such an “init container” for my Neo4j database. The schematic architecture is depicted below and includes an EC2 ECS host on which three containers should run: first the <strong>certbot</strong> container is started that can request the certificate for the corresponding domain. Once the certificate and the private key are there, the certbot container exits successfully upon which the second container (copier) is started. This container just needs a shell (I decided to use <strong>debian:latest</strong> for this purpose) and its purpose is to copy the certificate and the private key into the folders and under the file names Neo4j expects. Upon the successful exit of this container, the <strong>Neo4j</strong> container is finally started.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*suCho6Skn-M2unmITkXGxA.png" /></figure><p>To achieve the correct order of the containers, AWS ECS supports the <strong>dependsOn</strong> attribute — a list of <a href="https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ContainerDependency.html"><em>ContainerDependency </em>objects</a> that in turn consist of <strong>containerName</strong> and <strong>condition</strong>. The condition attribute allows to specify whether the previous container should have started (START), exited (COMPLETE), ran successfully (SUCCESS) or is passing docker health checks (HEALTHY). In the present use case, SUCCESS is the correct <strong>condition,</strong> since both certificate retrieval and copy are crucial for the Neo4j container to work properly (the copier container is called debian here):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/0*I0PRMpDd4bhw8-tq.png" /></figure><h3>Routing to certbot</h3><p>A small challenge for the architecture above is to ensure that certbot can solve the HTTP challenge of Let’s encrypt which is a part of the ACME protocol (this is needed to verify that the domain for which the certificate is requested is indeed controlled by us). The problem here is that if targets of type <em>instance</em> are used with the load balancer (which makes sense for an ECS EC2 host), health checks are mandatory. On the other hand, since certbot is running only for a short time, it itself cannot be used for health checks on port 80. Also, LE expects the domain to be already routable to certbot requesting the certificate which means that a typical registration delay that load balancers have is not acceptable in this case. As a result, the instance should be registered at the corresponding target group of the NLB and already healthy <em>before</em> the certbot container is even started. To address this issue, I decided to use a simple trick based on a small <a href="https://github.com/konstl000/handshaker">handshaker app</a>. This app provides a golang-based http server listening on a specified port that simply replies “OK” to any request and can be deployed as a scratch-based docker container (or a binary). Since the app cannot block the port 80 (which is required by certbot once it is ready to accept the HTTP challenge), I configured the corresponding target group (TG80) to forward to port 80 but health check on another port (6666) which I then assigned to handshaker. To ensure the correct timing, I included starting the app into the <em>user data </em>script of the ECS EC2 instance and made terraform (with which the whole infrastructure is built) to register the auto scaling group that deploys these instances at TG80.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/0*TJ5nsvcwcEKyCTQi.png" /></figure><p>As expected, shortly after terraform apply, the instance was registered at TG80 and became healthy. After this, I used aws cli to scale the ECS service to 1 task (I usually initially deploy the ECS services with the task count of 0, so that the whole infrastructure such as load balancers, instances, Route53 entries, etc. is available before the containers are even started).</p><p>To my delight, certbot successfully requested the certificate, passed the HTTP challenge and stored the results in a shared folder mounted via a mount point. After this, the following script ran in the copier container followed by the successful start of Neo4j.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/0*tHsoCPoOTSDaNQe7.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/0*AJWVgy7lBR2Bnfy6.png" /></figure><h3>Alternatives</h3><p>Of course, the described approach is not the only way to get a certificate from LE and provide it to a Neo4j container (or another application). Some of the simple alternatives I can immediately think of would be:</p><p>· Run certbot directly on the EC2 host instead of a container</p><p>· Use k8s/k3s/k0s in combination with cert-manager</p><p>· Build a custom container that has certbot inside of it</p><p>That being said, I think that the init container approach shows a way of using ECS similar to k8s pods and can be successfully applied to other ECS-based solutions. Also, it allows to use the upstream containers which makes upgrades seamless as opposed to the “one custom container” approach. In case you wonder, how the hell I could run a custom bash script within an upstream debian container — I just used a mount point to mount a folder from the host that has been created and filled by the <em>user data</em> script during the EC2 deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/0*4u4wtrKadmBumpJG.png" /></figure><h3>Scaling</h3><p>In the current example, I used an auto scaling group with just one instance in it, which allows all the mount points to be folders on this instance. Of course, the local folder solution would not scale well. In this case, however, EFS can be used instead, so that the certificate and the key would be requested just once by one of the certbots (certbot exits automatically if a valid certificate is already present), but can then be used by all of the corresponding Neo4j containers. All other services used in the infrastructure above (NLB, NAT Gateway, ECS) support horizontal scaling, so that a solution based on this approach can be scaled out with ease.</p><h3>Conclusions</h3><p>In conclusion, AWS ECS provides a nice option to include k8s-like “init containers” by using container dependencies and non-essential containers. Those can be employed for a variety of purposes including generation of TLS certificates with a certbot container. The TLS credentials can be then immediately provided to an application running in the essential container on the same host resulting in a publically trusted secured app.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b100f2f0b493" width="1" height="1" alt=""><hr><p><a href="https://medium.com/fme-developer-stories/certbot-as-an-init-container-for-aws-ecs-b100f2f0b493">Certbot as an init container for AWS ECS</a> was originally published in <a href="https://medium.com/fme-developer-stories">fme DevOps Stories</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A custom SharePoint newsletter service]]></title>
            <link>https://medium.com/fme-developer-stories/a-custom-sharepoint-newsletter-service-6b82d7b71e98?source=rss----710d04106367---4</link>
            <guid isPermaLink="false">https://medium.com/p/6b82d7b71e98</guid>
            <category><![CDATA[rest]]></category>
            <category><![CDATA[newsletter]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[sharepoint]]></category>
            <category><![CDATA[canvascontent1]]></category>
            <dc:creator><![CDATA[Daniel Belz]]></dc:creator>
            <pubDate>Fri, 13 May 2022 12:14:13 GMT</pubDate>
            <atom:updated>2022-05-13T12:14:13.824Z</atom:updated>
            <content:encoded><![CDATA[<p>In a customer project, we had the task to implement a custom newsletter service for SharePoint. You may wonder: Why are you not using the <a href="https://support.microsoft.com/en-us/office/create-and-send-a-news-digest-42efc3c6-605f-4a9a-85d5-1f9ff46019bf">news digest</a> feature? <br>Well, the answer to that was that it didn’t match the requirements of the customer.</p><h3>Requirements</h3><ul><li>The application should be able to send emails that contain the content of a news post (except images).</li><li>The application should transform relative links into absolute links.</li><li>The application should remove unnecessary HTML fragments like webpart elements, no value attributes, custom SharePoint attributes, class names, empty &lt;div&gt; elements, empty &lt;span&gt; elements.</li><li>The application should support a basic template mechanism to define the layout for the mail.</li><li>The application should run on a single server that is not exposed to the outside.</li><li>The application should be run regularly with the Windows TaskScheduler to fetch news.</li><li>The application should only process news once.</li><li>The application should contain all dependencies without the need to install additional software.</li></ul><h3>Technologies</h3><p>For the implementation of our Azure AD application, we decided to use .NET Core (C#). It is like SharePoint a fundamental part of the Microsoft ecosystem and therefore, there is a large and active developer community that provides many good libraries (like <a href="https://github.com/pnp/pnpcore">PnPCore</a>) around it.</p><p>Our application uses the following 3rd party packages:</p><ul><li>AngleSharp (for working with the HTML DOM)</li><li>PnP.Core (for accessing the SharePoint/Microsoft Graph API)</li><li>Serilog (for logging)</li></ul><h3>Getting busy</h3><p>The first step was to figure out a way to retrieve the newsletter list. With PnP.Core this was pretty straightforward (simplified code):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/53f4e962d2f1d96c9b8af92ba6b5d40e/href">https://medium.com/media/53f4e962d2f1d96c9b8af92ba6b5d40e/href</a></iframe><p>With this small piece of code, we were able to get the correct SharePoint list instance which contained all of our newsletter items. But as always problems arose. We questioned ourselves:</p><p><em>What should we do when the list grows bigger over time? Fetching all the items again and again? Even if we already processed them?</em></p><p>Sounds like a really <strong>bad</strong> plan. Also because SharePoint (Online) will use throttling mechanisms when you are sending too many requests. <br>In the end, we decided to use change queries because SharePoint keeps track of changes made to sites, lists, list items, and more. To get an idea of how that works you can head over <a href="https://pnp.github.io/pnpcore/using-the-sdk/changes-sharepoint.html">here</a>. <br>You will get a change token when sending a request to the SharePoint REST API. When you store that token you can use it for the next request to get only the changed items. We stored it in a simple text file because it does not contain any sensitive information.<br>After that our application was able to receive only changed items. Great! The next requirement was to only process a newsletter once. For that purpose, we saved the unique list item id (after processing it) in a text file and called it a day!</p><h3>Processing a single list item</h3><p>To be able to send a newsletter mail we first needed to process the received list item and request some more information. We needed the following data:</p><ul><li>UniqueId</li><li>Title</li><li>Author</li><li>Recipients (custom field)</li><li>AsTestMail (custom field)</li><li>PublishDate</li><li>Topic (custom field, taxonomy)</li><li>Content (CanvasContent1)</li><li>Url</li><li>AttachmentUrl (custom field)</li></ul><p>The properties Author, UniqueId, Title, AsTestMail, PublishDate, Url, and AttachmentUrl were already in the right format and did not require any custom processing. The CanvasContent1 property will get its own section because it was more complex to deal with it.</p><h4>Recipients</h4><p>The initial value we received was a IEnumerable&lt;&gt; of a type implementing the interface <a href="https://pnp.github.io/pnpcore/api/PnP.Core.Model.SharePoint.IFieldUserValue.html">IFieldUserValue</a>. Because the <a href="https://pnp.github.io/pnpcore/api/PnP.Core.Model.SharePoint.IFieldUserValue.html">IFieldUserValue</a> type did not had all informations we needed we retrieved the <a href="https://pnp.github.io/pnpcore/api/PnP.Core.Model.Security.IGraphUser.html">IGraphUser</a> representation for it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f3a54180593e339e0296fce1a287735b/href">https://medium.com/media/f3a54180593e339e0296fce1a287735b/href</a></iframe><h4>Topic</h4><p>The initial value we received here was a (unique) TermId. Because we needed the string representation/real value we were forced to look it up from the term store:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d54e9619e1e2a460b3098493e2920d7f/href">https://medium.com/media/d54e9619e1e2a460b3098493e2920d7f/href</a></iframe><h3>Transforming ‘CanvasContent1’</h3><p>The CanvasContent1 property contains the HTML body of the list item. Unfortunately, it’s pretty cluttered with elements that we don’t want or can’t display in our mail body. So the next step was to unclutter it. Here we used <a href="https://github.com/AngleSharp/AngleSharp">AngleSharp</a> because it made working with the DOM a breeze. You should check it out!</p><p><strong>Notice:</strong> <em>Some of the snippets below could be optimized even more (for example by the usage of LINQ instead of a for-each loop), but they are currently running exactly like that “battle-proven” in productive.</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/24b9ae71554e4b3705003c4da27fc658/href">https://medium.com/media/24b9ae71554e4b3705003c4da27fc658/href</a></iframe><h4>Loading CanvasContent1</h4><p>To work with AngleSharp we need to parse the plain string into an IHtmlDocument:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f24fd0a6b27f1e7a4f86d53431a171c5/href">https://medium.com/media/f24fd0a6b27f1e7a4f86d53431a171c5/href</a></iframe><h4>Removing webparts</h4><p>Webpart elements are pretty easy to identify. They are always represented by a html &lt;div&gt; and have the custom html attribute named <em>data-sp-webpart. </em>You can find and replace them like that:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/005859e1baf06b3d24886026064e1906/href">https://medium.com/media/005859e1baf06b3d24886026064e1906/href</a></iframe><h4>Removing images</h4><p>This requirement is even easier to implement. We just need to find all html &lt;img&gt; elements and remove them:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aa767a5ac650d0badd4b232d94c5c766/href">https://medium.com/media/aa767a5ac650d0badd4b232d94c5c766/href</a></iframe><h4>Removing attributes without value</h4><p>For this we are getting all elements of the document and searching for attributes with empty value. With a little bit of our beloved LINQ this is also pretty easy to do:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/944d729f96382b21120ec908d6e58554/href">https://medium.com/media/944d729f96382b21120ec908d6e58554/href</a></iframe><h4>Removing custom SharePoint attributes</h4><p>All custom SharePoint attribute names start with <em>data-sp. </em>So we used that as an identifier.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1119cef638574abd568b434e29726307/href">https://medium.com/media/1119cef638574abd568b434e29726307/href</a></iframe><h4>Removing class attributes</h4><p>Due to the fact that we have access to all document elements we are simply searching for the html class attribute.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7c795db46d6ee78096d24a28c8c31b6f/href">https://medium.com/media/7c795db46d6ee78096d24a28c8c31b6f/href</a></iframe><h4>Removing empty div and span elements</h4><p>Because of the pre processing above we can simply replace &lt;span&gt; and &lt;div&gt; elements in the document body.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7a0ad77780afc2a307a4950307ec3a03/href">https://medium.com/media/7a0ad77780afc2a307a4950307ec3a03/href</a></iframe><h4>Transforming relative links to absolute links</h4><p>The last requirement for the newsletter body. Finally! And it’s a pretty common task. Just search all &lt;a&gt; tags, check their value, and if it’s a relative URL replace it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8965abedf0b3104971d7ee058d24f92d/href">https://medium.com/media/8965abedf0b3104971d7ee058d24f92d/href</a></iframe><h4>Bringing it all together + final result</h4><p>Now we should have a result which we can send without problems via mail. Let’s bring all parts together!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2de06c52304e630e3852fd3783adfeb1/href">https://medium.com/media/2de06c52304e630e3852fd3783adfeb1/href</a></iframe><p>And the final result is:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/22459fbbb78a44560001e1ec9070dd13/href">https://medium.com/media/22459fbbb78a44560001e1ec9070dd13/href</a></iframe><h3>The template “engine”</h3><p>After processing the CanvasContent1 we are now ready to craft our mail. To be flexible we decided to implement a simple template “engine”. The template is loaded from a local HTML file and contains different placeholders. These placeholders are then replaced with the actual values. The available placeholders are:</p><ul><li>%RECIPIENTS%</li><li>%PUBLISH_DATE%</li><li>%TOPIC%</li><li>%TITLE%</li><li>%CONTENT%</li><li>%AUTHOR%</li><li>%ATTACHMENT_URL%</li><li>%NEWSLETTER_URL%</li></ul><p>I think the placeholder names are speaking for themselves. The template we are currently using in productive is this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e0ba80442f20c63efca1dd96f0cb1ad0/href">https://medium.com/media/e0ba80442f20c63efca1dd96f0cb1ad0/href</a></iframe><h3>Final step — Sending the mail</h3><p>Unfortunately PnP.Core does not offer a (direct) way to send an email. But it does allows us to instantiate a GraphServiceClient which can do the job! We only need to use the PnP authentication provider like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a16432a6194fa3fd848ba06f32949db0/href">https://medium.com/media/a16432a6194fa3fd848ba06f32949db0/href</a></iframe><p>Afterwards we must create an instance of the <em>Message</em> class like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e014eab2d5b6ea79ccbb024cef6c5ba2/href">https://medium.com/media/e014eab2d5b6ea79ccbb024cef6c5ba2/href</a></iframe><p>Now we are authenticated, have our <em>Message</em> instance, and are ready to send! As the last step just retrieve the user who should send the mail and call the SendMail() method:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/42730b3e339418dafe993cecd7f7a29a/href">https://medium.com/media/42730b3e339418dafe993cecd7f7a29a/href</a></iframe><p>And this is the final result from the view of the recipient:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UyrGvgdiMeNBDphXlGLwrg.png" /><figcaption>Final email</figcaption></figure><p>I hope you liked my blog post about our adventure to build a custom SharePoint newsletter and could gather some insights from it 😊.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b82d7b71e98" width="1" height="1" alt=""><hr><p><a href="https://medium.com/fme-developer-stories/a-custom-sharepoint-newsletter-service-6b82d7b71e98">A custom SharePoint newsletter service</a> was originally published in <a href="https://medium.com/fme-developer-stories">fme DevOps Stories</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>