<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by ProjectAgoraEng on Medium]]></title>
        <description><![CDATA[Stories by ProjectAgoraEng on Medium]]></description>
        <link>https://medium.com/@project-agora?source=rss-93d6b25eed24------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*TPeogzRAFFd_06SduaQjmA.png</url>
            <title>Stories by ProjectAgoraEng on Medium</title>
            <link>https://medium.com/@project-agora?source=rss-93d6b25eed24------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 18:47:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@project-agora/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Automating Nightly Tests: A Comprehensive Guide using Robot Framework and Jenkins]]></title>
            <link>https://engineering.projectagora.com/automating-nightly-tests-a-comprehensive-guide-using-robot-framework-and-jenkins-932a317078b?source=rss-93d6b25eed24------2</link>
            <guid isPermaLink="false">https://medium.com/p/932a317078b</guid>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[robot-framework]]></category>
            <category><![CDATA[automated-testing]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[testing]]></category>
            <dc:creator><![CDATA[ProjectAgoraEng]]></dc:creator>
            <pubDate>Wed, 20 Sep 2023 11:34:10 GMT</pubDate>
            <atom:updated>2023-09-20T11:40:40.264Z</atom:updated>
            <content:encoded><![CDATA[<p><em>by </em><a href="https://www.linkedin.com/in/stavros-tsiolas/"><em>Stavros Tsiolas</em></a><em>, Quality Assurance Engineer</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*yDdcfeLk6Oe3nO_wuX0uLQ.png" /></figure><h4><strong>Introduction</strong></h4><p>Automated testing has become a crucial aspect of modern software development, enabling teams to ensure product quality, reduce manual effort, and detect issues early. In this engineering blog, we will explore how we leverage the power of Robot Framework and Jenkins to schedule the execution of automated testing scripts every night. By implementing this streamlined process, the testing team can improve efficiency and focus on more critical tasks while maintaining confidence in the software’s reliability.</p><h4>Efficient Test Automation: Harnessing the Power of Jenkins and Robot Framework Integration</h4><p>As we embrace automation as a cornerstone of our testing strategy, we will explore the seamless integration of Jenkins with Robot Framework, made possible through the Robot Framework Plugin. This powerful combination allows us to automate and schedule test runs effortlessly, enhancing the efficiency and reliability of our testing process. Jenkins takes center stage as the orchestrator, triggering test executions at specified intervals, enabling us to conduct scheduled nightly tests with ease. The Robot Framework Plugin for Jenkins streamlines the entire testing workflow, providing a user-friendly interface to configure test parameters, view real-time updates on test progress, and access detailed test reports. Integrating the Robot Framework Plugin with Jenkins is straightforward — simply navigate to the Jenkins management dashboard, access the “Manage Plugins” section, and search for “Robot Framework Plugin” to install it with a few clicks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*FS146tuRDgvoBMePINZIrA.png" /><figcaption>Image 1 — Plugins</figcaption></figure><p>The plugin’s configuration is straightforward, as shown in the screenshot below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*8jNhjfYLpog75VYXB6vhnQ.png" /><figcaption>Image 2 — Post-build Actions</figcaption></figure><p>With this integration, our testing efforts are optimized, promoting collaboration among team members and empowering us to make informed decisions based on reliable and actionable test results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*is2ZXSDBQ_65FySy3wPuAA.png" /><figcaption>Image 3 — Robot Framework Test Results</figcaption></figure><h4>Jenkins Configuration with Git and Email: Optimizing Automated Testing Workflow</h4><p>In our quest for automated testing efficiency, we delve into the powerful synergy of Jenkins configuration with Git and Email integration. This combination streamlines our automated testing workflow, ensuring a smooth and reliable process from start to finish. By configuring a dedicated Jenkins job for our automated tests, we establish an automation pipeline that defines the test execution process and incorporates the necessary configurations. Integrating our Git repository within the Jenkins job allows us to achieve version control for our Robot Framework test scripts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*7Bh9P1ZCkOVnacXBnEqUag.png" /><figcaption>Image 3 — Source Code Management</figcaption></figure><p>This integration ensures that our tests always utilize the latest code version, providing accurate and dependable results. Moreover, we leverage Jenkins’ built-in email notification feature to receive comprehensive test reports and notifications automatically. Configuring the email settings enables our team to stay promptly informed about test outcomes, including detailed pass/fail status, execution duration, and access to test logs and reports.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*etl72wS_0ZnHwY4619Kj4g.png" /><figcaption>Image 5 — Publisher Module NightRun Still Failing</figcaption></figure><p>With this easy-to-follow setup, our automated testing workflow becomes more transparent, collaborative, and efficient, empowering us to optimize the quality and reliability of our software products.</p><h3>Best Practices and Tips</h3><p>To ensure a smooth and successful automated testing process with Robot Framework and Jenkins, adopting best practices is essential. Firstly, establish a version control system to manage test scripts effectively and foster collaboration among team members. Regularly update and maintain the test scripts to reflect changes in the application under test. Implement a test data management strategy to handle data-dependent tests efficiently. Utilize tagging and categorization within Robot Framework to selectively execute test suites or scenarios based on specific criteria, optimizing test runs. Regularly review and analyze test results to identify patterns and improve test coverage. Additionally, invest in proper error handling mechanisms and ensure test environments are isolated and repeatable. Lastly, encourage continuous learning within the team to stay updated on the latest features and improvements in Robot Framework and Jenkins, empowering testers to leverage these tools to their full potential.</p><h3>Conclusion</h3><p>Automated testing has become an indispensable part of our software development process, enabling us to maintain high-quality standards and accelerate the delivery of our products. The integration of Robot Framework and Jenkins alongside the necessary plugins has empowered us to run tests efficiently every night, ensuring continuous feedback and quick identification of issues. By sharing our experiences with these tools, we hope to inspire fellow testers to enhance their test toolchain and streamline their testing workflows. Happy testing!</p><p><strong>Stay in the loop with the freshest news and insights from Project Agora! Don’t miss a beat — </strong><a href="https://engineering.projectagora.com/"><strong>subscribe to our blog for regular updates</strong></a><strong>. If you’re ready to become part of our dynamic team, explore the exciting job opportunities on </strong><a href="https://careers.projectagora.com/engineering/"><strong>our career page</strong></a><strong>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=932a317078b" width="1" height="1" alt=""><hr><p><a href="https://engineering.projectagora.com/automating-nightly-tests-a-comprehensive-guide-using-robot-framework-and-jenkins-932a317078b">Automating Nightly Tests: A Comprehensive Guide using Robot Framework and Jenkins</a> was originally published in <a href="https://engineering.projectagora.com/">Project Agora Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enhancing Engineering QA with ChatGPT: Revolutionizing the Testing Landscape]]></title>
            <link>https://engineering.projectagora.com/enhancing-engineering-qa-with-chatgpt-revolutionizing-the-testing-landscape-5e662f3f9d3d?source=rss-93d6b25eed24------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e662f3f9d3d</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[qa]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[qa-testing]]></category>
            <category><![CDATA[chatgpt]]></category>
            <dc:creator><![CDATA[ProjectAgoraEng]]></dc:creator>
            <pubDate>Wed, 19 Jul 2023 12:07:57 GMT</pubDate>
            <atom:updated>2023-07-19T12:07:57.028Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By </em><a href="https://www.linkedin.com/in/panagiotisdimitroulas/"><em>Panagiotis Dimitroulas</em></a><em>, Senior Engineering Manager</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9GSbSfKuy-IzC6wazPZJSQ.jpeg" /></figure><p>In the ever-evolving world of engineering, Quality Assurance (QA) plays a pivotal role in ensuring the reliability, functionality, and performance of software systems. Traditional QA methods often involve manual testing, which can be time-consuming, resource-intensive, and prone to human error. However, with the emergence of advanced artificial intelligence (AI) models like ChatGPT, there is significant potential to revolutionize engineering QA processes. In this article, I will explore how ChatGPT can improve engineering QA and pave the way for more efficient and effective testing methodologies, using the example of testing a page.</p><ol><li><strong>Automating Test Case Generation:</strong></li></ol><p>One of the most time-consuming tasks in engineering QA is the creation of test cases. Let’s consider a Behavior-Driven Development (BDD) approach for testing a page. Using ChatGPT, we can generate test cases that cover various scenarios, such as successful login, incorrect username or password, and account lockout after multiple failed attempts. For instance, a generated test case could be:</p><p><strong>Scenario: Successful Login<br>Given the login page is displayed<br>When I enter valid credentials<br>Then I should be logged in successfully</strong></p><p>By training ChatGPT on a dataset of historical page test cases, it can learn to generate new test cases that cover a range of scenarios. This automated test case generation significantly reduces manual effort and accelerates the QA process.</p><p><strong>2. Intelligent Bug Detection:</strong></p><p>Detecting and fixing bugs is a critical aspect of engineering QA. ChatGPT can assist in bug detection by analyzing codebases, logs, and system outputs specific to the page. By training ChatGPT on a dataset of page bugs and resolutions, the model can learn to identify common issues and potential problem areas. It can help flag potential bugs by analyzing factors like incorrect error messages, improper handling of failures, or security vulnerabilities. For example, ChatGPT can suggest bug detection rules like:</p><ul><li>Verify that an incorrect login attempt displays an appropriate error message.</li><li>Validate that the account is locked after a specified number of failed login attempts.</li></ul><p>This AI-powered bug detection approach improves the efficiency and accuracy of identifying and addressing bugs related to the page.</p><p><strong>3. Knowledge Base and Documentation Assistance:</strong></p><p>Maintaining an up-to-date knowledge base and documentation is crucial for engineering teams, particularly when testing a page. ChatGPT can assist in generating and updating documentation by providing accurate and contextually relevant information. QA engineers can query the model for explanations, best practices, and script examples specific to page testing. For instance, a QA engineer can ask ChatGPT for guidance on writing effective BDD scenarios for registration/login testing, and the model can respond with detailed explanations and examples, such as:</p><p><strong>Feature: User Registration</strong></p><p><strong>Scenario: Successful Registration<br>Given the user is on the registration page<br>When the user enters valid registration details<br>Then the user should be successfully registered</strong></p><p><strong>Scenario: Registration with Existing Email<br>Given the user is on the registration page<br>When the user enters an email that is already registered<br>Then an error message should be displayed</strong></p><p><strong>Feature: User Login</strong></p><p><strong>Scenario: Successful Login<br>Given the user is on the login page<br>When the user enters valid login credentials<br>Then the user should be logged in successfully</strong></p><p><strong>Scenario: Invalid Login Credentials<br>Given the user is on the login page<br>When the user enters invalid login credentials<br>Then an error message should be displayed</strong></p><p>This facilitates knowledge sharing and promotes consistent practices in page testing across the team, leading to improved QA efficiency and effectiveness.</p><p><strong>4. Script Execution and Automation:</strong></p><p>Scripting plays a crucial role in automating page testing. ChatGPT can assist in scripting test cases for page scenarios. QA engineers can leverage ChatGPT to generate and execute scripts that automate e.g. login page interactions.</p><p>By training ChatGPT on a dataset of login page testing scripts, it can learn to generate customized scripts for different scenarios, such as incorrect login attempts or account lockout testing. This automation simplifies the testing process and ensures consistent and reliable execution of login page test cases.</p><p>To connect ChatGPT with the device under test, an integration layer is required. This integration allows ChatGPT to interact directly with the system being tested, such as a login page. There are several approaches to establish this connection:</p><ol><li>API Integration: If the device under test exposes APIs, ChatGPT can communicate with the device through these APIs. The QA engineer can define API requests and responses within the test execution instructions given to ChatGPT. The model can make HTTP requests to the device’s APIs, simulating user interactions and validating system responses.</li><li>Automation Framework Integration: Many automation frameworks provide APIs or libraries that allow external systems to control and interact with the device under test. QA engineers can integrate ChatGPT with the automation framework being used for testing. This integration enables ChatGPT to execute test scripts and interact with the device through the framework’s APIs or libraries, facilitating seamless test execution.</li></ol><p><strong>Conclusion:</strong></p><p>ChatGPT presents significant opportunities to enhance engineering QA by automating test case generation, improving bug detection, providing a natural language interface for test execution, and assisting in knowledge base and documentation generation. By leveraging ChatGPT in the context of a page testing example, engineering teams can streamline the QA process, improve software quality, and foster efficient collaboration.</p><p><strong>Stay connected with all the latest news and insights from Project Agora! </strong><a href="https://engineering.projectagora.com/"><strong>Subscribe to our blog for regular updates</strong></a><strong>. Looking to join our team of talented individuals? Check out </strong><a href="https://careers.projectagora.com/engineering/"><strong>our career page</strong></a><strong> for exciting job opportunities.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e662f3f9d3d" width="1" height="1" alt=""><hr><p><a href="https://engineering.projectagora.com/enhancing-engineering-qa-with-chatgpt-revolutionizing-the-testing-landscape-5e662f3f9d3d">Enhancing Engineering QA with ChatGPT: Revolutionizing the Testing Landscape</a> was originally published in <a href="https://engineering.projectagora.com/">Project Agora Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[FastAPI: Accelerating web development with speed and reliability]]></title>
            <link>https://engineering.projectagora.com/fastapi-accelerating-web-development-with-speed-and-reliability-5ff1383555c9?source=rss-93d6b25eed24------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ff1383555c9</guid>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[data-validation]]></category>
            <dc:creator><![CDATA[ProjectAgoraEng]]></dc:creator>
            <pubDate>Fri, 16 Jun 2023 08:12:50 GMT</pubDate>
            <atom:updated>2023-06-16T08:12:50.431Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By </em><a href="https://www.linkedin.com/in/ynikoloudakis/"><em>Yannis Nikoloudakis</em></a><em>, Full Stack Software Engineer</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x0VR70f6vF7kTtTAPrkQLA.jpeg" /></figure><p>In today’s world of microservices and API-driven development, building and maintaining robust, secure, and well-documented endpoints can be challenging. As APIs grow in complexity, developers often face issues with data validation, manual documentation, code duplication, and performance bottlenecks. Fortunately, there is a solution that addresses these challenges head-on: FastAPI. This blazingly fast Python-based web framework offers an intuitive and efficient approach to building APIs, making development faster and more reliable.</p><p><strong>Data Validation Made Easy</strong></p><p>FastAPI leverages the power of Pydantic, a Python library for data validation and serialization. With Pydantic, you can create data models and define request/response bodies effortlessly. It performs native data validation, marshalling, and unmarshalling, reducing the need for custom code. Automatic documentation generation is seamlessly integrated with the data models, ensuring accurate and up-to-date documentation.</p><p>Example code</p><pre>from datetime import datetime<br>from typing import List, Optional<br>from pydantic import BaseModel<br><br><br>class User(BaseModel):<br>    id: int<br>    name = &#39;John Doe&#39;<br>    signup_ts: Optional[datetime] = None<br>    friends: List[int] = []<br><br><br>external_data = {<br>    &#39;id&#39;: &#39;123&#39;,<br>    &#39;signup_ts&#39;: &#39;2019-06-01 12:22&#39;,<br>    &#39;friends&#39;: [1, 2, &#39;3&#39;],<br>}<br>user = User(**external_data)<br>print(user.id)<br>#&gt; 123<br>print(repr(user.signup_ts))<br>#&gt; datetime.datetime(2019, 6, 1, 12, 22)<br>print(user.friends)<br>#&gt; [1, 2, 3]<br>print(user.dict())<br>&quot;&quot;&quot;<br>{<br>    &#39;id&#39;: 123,<br>    &#39;signup_ts&#39;: datetime.datetime(2019, 6, 1, 12, 22),<br>    &#39;friends&#39;: [1, 2, 3],<br>    &#39;name&#39;: &#39;John Doe&#39;,<br>}<br>&quot;&quot;&quot;</pre><p><strong>Extensible Data Models</strong></p><p>FastAPI’s data models are highly extensible, allowing you to implement custom data validators using the validator decorator. This empowers developers to enforce specific validation rules and ensure data integrity within the API.</p><p>Example Code</p><pre>from pydantic import BaseModel, ValidationError, validator<br><br><br>class UserModel(BaseModel):<br>    name: str<br>    username: str<br>    password1: str<br>    password2: str<br><br>    @validator(&#39;name&#39;)<br>    def name_must_contain_space(cls, v):<br>        if &#39; &#39; not in v:<br>            raise ValueError(&#39;must contain a space&#39;)<br>        return v.title()<br><br>    @validator(&#39;password2&#39;)<br>    def passwords_match(cls, v, values, **kwargs):<br>        if &#39;password1&#39; in values and v != values[&#39;password1&#39;]:<br>            raise ValueError(&#39;passwords do not match&#39;)<br>        return v<br><br>    @validator(&#39;username&#39;)<br>    def username_alphanumeric(cls, v):<br>        assert v.isalnum(), &#39;must be alphanumeric&#39;<br>        return v</pre><p><strong>The Power of Plugins</strong></p><p>FastAPI boasts a vibrant ecosystem of plugins that enhance and extend its capabilities. Plugins like Hypothesis for unit-testing and Httpx for HTTP client functionality, along with FastAPI-SQLAlchemy for database integration, make it easy to leverage additional features and simplify complex tasks in your API development workflow.</p><p><strong>Automatic Documentation Generation</strong></p><p>FastAPI is fully compatible with the OpenAPI standard (previously known as Swagger) and JSON Schema. After declaring your API endpoints and defining the necessary details such as query parameters, request/response schemas, and metadata, FastAPI automatically generates comprehensive documentation. The interactive API documentation can be accessed at http://&lt;server_address&gt;:&lt;server_port&gt;/docs, providing a seamless and up-to-date reference for your API.</p><figure><img alt="Automatic Documentation Generation (source: https://fastapi.tiangolo.com/features/#automatic-docs)" src="https://cdn-images-1.medium.com/max/960/1*UhjVFvXcw_rRGHkWBDq1Yg.png" /><figcaption>Automatic Documentation Generation (source: <a href="https://fastapi.tiangolo.com/features/#automatic-docs)">https://fastapi.tiangolo.com/features/#automatic-docs)</a></figcaption></figure><p><strong>Unleashing Unmatched Performance</strong></p><p>FastAPI, powered by Uvicorn and Pydantic, delivers exceptional performance comparable to frameworks like Node.js and Go. Its efficient and intuitive design allows developers to <strong>write code up to 200–300% faster</strong>. Additionally, FastAPI’s strict type checking helps reduce bugs by up to 40%, eliminating common human errors.</p><p><strong>Native WebSocket Support</strong></p><p>FastAPI offers native WebSocket support alongside REST, thanks to its asynchronous capabilities. This makes it easy to build real-time, bi-directional communication channels in your applications.</p><p>Example Code</p><p><em>Source: </em><a href="https://fastapi.tiangolo.com/lo/advanced/websockets/?h=websocket"><em>https://fastapi.tiangolo.com/lo/advanced/websockets/?h=websocket</em></a></p><pre>@app.get(&quot;/&quot;)<br>async def get():<br>    return HTMLResponse(html)<br><br><br>@app.websocket(&quot;/ws&quot;)<br>async def websocket_endpoint(websocket: WebSocket):<br>    await websocket.accept()<br>    while True:<br>        data = await websocket.receive_text()<br>        await websocket.send_text(f&quot;Message text was: {data}&quot;)</pre><p><strong>Conclusion</strong></p><p>FastAPI has gained significant popularity in the developer community, <strong>with over 58K stars and 4.9K forks on GitHub</strong>. Its active community support and continuous updates ensure a robust and reliable framework. By harnessing FastAPI’s intuitive syntax, automatic documentation generation, exceptional performance, and easy migration path from other frameworks, such as Flask or Sanic, developers can unlock new levels of productivity and efficiency in their API development workflows.</p><p>To learn more, explore the official FastAPI documentation at <a href="https://fastapi.tiangolo.com/">https://fastapi.tiangolo.com</a> and dive into the source code on GitHub.</p><p>Remember, with FastAPI, you can build powerful, performant, and well-documented APIs in Python, accelerating your development process and delivering exceptional experiences to your users.</p><p>Stay up-to-date with the latest news and insights from Project Agora! <a href="https://engineering.projectagora.com/">Subscribe to our blog now</a> and stay connected. Interested in becoming part of our talented team? <a href="https://careers.projectagora.com/engineering/">Visit our career page</a> to explore exciting job opportunities. Join us today!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ff1383555c9" width="1" height="1" alt=""><hr><p><a href="https://engineering.projectagora.com/fastapi-accelerating-web-development-with-speed-and-reliability-5ff1383555c9">FastAPI: Accelerating web development with speed and reliability</a> was originally published in <a href="https://engineering.projectagora.com/">Project Agora Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using MongoDB and Python friends to validate and store your data]]></title>
            <link>https://engineering.projectagora.com/using-mongodb-and-python-friends-to-validate-and-store-your-data-1328df2082ff?source=rss-93d6b25eed24------2</link>
            <guid isPermaLink="false">https://medium.com/p/1328df2082ff</guid>
            <category><![CDATA[mongoengine]]></category>
            <category><![CDATA[marshmallow]]></category>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[ProjectAgoraEng]]></dc:creator>
            <pubDate>Wed, 17 May 2023 14:14:42 GMT</pubDate>
            <atom:updated>2023-05-17T14:44:14.534Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By </em><a href="https://www.linkedin.com/in/dimosthenis-dimos-53b204185/"><em>Dimosthenis Dimos</em></a><em>, Software Engineer</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OxQGNofFy6eEPmtnv5RwCw.jpeg" /></figure><p>Building applications that handle complex data structures can be challenging. Fortunately, with the help of tools like Marshmallow schemas, MongoEngine, and MongoDB, you can make this task much easier.</p><p><a href="https://www.mongodb.com/docs/"><strong>MongoDB</strong> </a>is a popular NoSQL database that is known for its flexibility and scalability. It allows you to store and query data in a variety of ways, making it a great choice for many types of applications. However, working with complex data structures can be difficult when using MongoDB directly.</p><p>This is where Marshmallow schemas come in. <a href="https://marshmallow.readthedocs.io/en/stable/"><strong>Marshmallow</strong> </a>is a Python library that provides a simple and flexible way to validate and transform data structures. It allows you to define a schema that describes the structure of your data and the validation rules that apply to it.</p><p>For example, let’s say you are building an application that stores information about users. You could define a schema for the user data like this:</p><pre>from marshmallow import Schema, fields<br><br>class UserSchema(Schema): <br>  name = fields.String(required=True) <br>  email = fields.Email(required=True)<br>  age = fields.Integer()</pre><p>In this example, we define a <strong>UserSchema</strong> that includes fields for the user’s name, email, and age. The <strong>required=True</strong> parameter indicates that these fields are mandatory.</p><p>Once the schema is defined, you can use it to validate user data. Suppose you receive some user data from a form submission:</p><pre>data = { <br><br>  &#39;name&#39;: &#39;John Doe&#39;, <br>  &#39;email&#39;: &#39;johndoe@example.com&#39;, <br>  &#39;age&#39;: &#39;30&#39; <br>} <br><br>schema = UserSchema() <br>result = schema.load(data)</pre><p>In this case, we create a new <strong>UserSchema</strong> and use it to validate the <strong>data </strong>variable. The<strong> load()</strong> method returns a validated and deserialized object that matches the schema. If the data fails validation, a <strong>ValidationError </strong>will be raised.</p><p>Now that we have validated our user data using Marshmallow, we can store it in MongoDB using a tool like MongoEngine. <a href="https://docs.mongoengine.org/"><strong>MongoEngine</strong> </a>is an Object-Document Mapping (ODM) library that provides a simple and intuitive way to interact with MongoDB.</p><p>Here’s an example of how you could use MongoEngine to store our validated user data in MongoDB:</p><pre>from mongoengine import connect, Document, StringField, IntField <br> <br>connect(&#39;my_database&#39;) <br> <br>class User(Document): <br>    name = StringField(required=True) <br>    email = StringField(required=True) <br>    age = IntField() <br> <br>user_data = result <br>user = User(name=user_data[&#39;name&#39;], email=user_data[&#39;email&#39;], age=user_data[&#39;age&#39;]) <br>user.save() </pre><p>In this example, we define a <strong>User </strong>document using MongoEngine and a <strong>UserSchema</strong> using Marshmallow. We then receive some user data and validate it against the <strong>UserSchema</strong>. If the validation succeeds, we create a new <strong>User </strong>document using the validated data and save it to MongoDB.</p><p>Using Marshmallow schemas with MongoEngine and MongoDB allows you to easily handle complex data structures. Marshmallow schemas can be nested, allowing you to validate and transform nested objects, lists, and dictionaries. This is especially useful when working with APIs or databases that require complex data structures.</p><p>In conclusion, using Marshmallow schemas with MongoEngine and MongoDB provides a powerful way to validate and store complex data structures. By following best practices and using these tools together, you can build applications that are both flexible and reliable.</p><p>Stay in the loop with the latest news and insights from Project Agora! <a href="https://engineering.projectagora.com/">Subscribe to our blog</a> today and never miss an update. Looking to join our team of talented individuals? Head over to our <a href="https://careers.projectagora.com/engineering/">career page to explore our current job openings</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1328df2082ff" width="1" height="1" alt=""><hr><p><a href="https://engineering.projectagora.com/using-mongodb-and-python-friends-to-validate-and-store-your-data-1328df2082ff">Using MongoDB and Python friends to validate and store your data</a> was originally published in <a href="https://engineering.projectagora.com/">Project Agora Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tech for Good: Reducing Our Environmental Footprint]]></title>
            <link>https://engineering.projectagora.com/tech-for-good-reducing-our-environmental-footprint-a4cca870fa9?source=rss-93d6b25eed24------2</link>
            <guid isPermaLink="false">https://medium.com/p/a4cca870fa9</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[green-tech]]></category>
            <category><![CDATA[environment]]></category>
            <category><![CDATA[green-technology]]></category>
            <dc:creator><![CDATA[ProjectAgoraEng]]></dc:creator>
            <pubDate>Thu, 20 Apr 2023 15:37:41 GMT</pubDate>
            <atom:updated>2023-04-26T10:17:35.737Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By </em><a href="https://www.linkedin.com/in/vasilis-vasilatos-99957460/"><em>Vasilis Vasilatos</em></a><em>, Tech Lead and Engineering Manager of Data</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5K-fo9P9BvnCZ2gqRXQ6jQ.jpeg" /></figure><p><strong>Engineering a greener future: Why sustainability matters in Project Agora and IT</strong></p><p>As engineers, we have the power to make an impact in the world. At <a href="https://projectagora.com/"><strong>Project Agora</strong></a>, we are committed to making a positive change and leaving a lasting impression on the environment. We believe that every individual can make a difference, and we want to share some insights with you on how we can achieve this together.</p><p>On the first day of work at <strong>Project Agora</strong>, all new employees attend a call with our CEO. During this call, the values of <strong>Project Agora</strong> and <a href="https://thinkdigitalgroup.net/"><strong>ThinkDigital Group</strong></a> are presented, and it’s a moment that many of us remember. I can recall my first day and how my brain stopped at a specific slide that read, “We are on a mission. We have to keep the Open Web Open. We are the modern Robin Hoods.” Working for an organization that has a purpose beyond profit is inspiring, and it motivates us to do our best work every day.</p><p>Another great action of <strong>Project Agora</strong> was the invitation to <a href="https://www.linkedin.com/in/brianokelley/"><strong>Brian O’Kelley</strong></a>, CEO of <a href="https://www.scope3.com/"><strong>Scope3</strong></a>, as a special guest to our yearly main Strategy event. Brian is a person on a mission too. Scope3 is the source of truth for supply chain emissions data, helping companies factor carbon emissions into every business decision. And that is where we get the inspiration for this article.</p><p>Today, we’ll focus on the “green” area and how we, as the IT industry, can make an impact to improve sustainability. Let’s start by understanding the environmental impact that our industry has:</p><ul><li>In 2014, U.S. data centers consumed 70 billion kWh of electricity, which is 1.8% of total electricity consumption (Lawrence Berkeley National Laboratory).</li><li>The peak power associated with servers and data centers in 2007 was 7 GW. Existing technologies and efficient design strategies can reduce server energy use by 25% or more, while best management practices and consolidating servers can reduce energy use by 20%. U.S. Environmental Protection Agency (EPA).</li><li>The IT sector accounts for 4% of global greenhouse gas emissions, and this could double by 2025 (CIRAIG, 2022).</li></ul><p>That is a lot!!! I mean a LOT! And this comes from people like me and you.</p><p>These numbers are staggering, and it’s important that we take action to reduce our environmental impact.</p><p>As <strong>Project Agora</strong>, we’ve taken steps to reduce waste, especially when moving to the Cloud. It can make our lives easier when it comes to infrastructure handling, but it can also increase our waste. Here are some solutions that we use to reduce waste:</p><p><strong>Serverless Technologies</strong></p><p>We moved to serverless solutions where possible and always include them in our new designs when we have specific usage of servers. Please don’t go with a Kubernetes Cluster for example to deploy a couple of microservices that are serving 50 requests per day. Choose solutions that can scale to 0 and reduce the waste of always being up and running.</p><p><strong>Filter out waste as soon as possible</strong></p><p>This usually takes place in data handling. Many organizations process and store huge amounts of data for no reason, just to have the data. In our case, we are really careful with the data and always handle the lifecycle and processing during each step of the pipeline.</p><p><strong>Review the complexity. Do things simpler</strong></p><p>Spend some time optimizing your solution and your algorithms. It is really easy to deliver the solution to the functional requirements but here you can make a real impact since most of the engineers don’t pay attention or don’t have the time to optimize. We give that time, and we enforce through our processes the handling of the non-functional requirements and the optimization of our algorithms.</p><p><strong>Max utilization of resources by optimizing scalability</strong></p><p>Spend some time to see how your solution performs. Check the utilization of your resources. Optimize your scalability solution so that your system can scale when it reaches close to its full capacity. In PA the DevOps team works really closely with the Engineers so that meaningful dashboards from infra are in place so that actions can be taken to maximize the utilization of the resources. The Engineering team spends time in each sprint to get insights from these dashboards and investigate and design solutions to make our products faster and less power-consuming.</p><p><strong>Cache</strong></p><p>Cache mechanisms are used where possible. There is no reason to ask again and again the same thing. Don’t be like your younger little brother “Are we there yet?”. From the system level to the function level using a cache can really contribute to less useless data movement.</p><p><strong>Review DAG</strong></p><p>In distributed systems, you need to fully understand how each function executes. Just like we do with the dashboards described above we do the same with the DAGs for our pipelines. From there, important outcomes can be derived such as back-pressure which leads to part of your system working in reds and other part waiting because of the resisted flow of data.</p><p><strong>Review the configurations of your systems</strong></p><p>Another quick win! As an example, think about the case with the file and batch size that can reduce the I/O and network overhead contributing to less compute power when reading and writing data and of course less storage maintaining.</p><p>As the Tech Lead of the <strong>Project Agora</strong> Big Data section, I can assure you that meeting all of these points is challenging. Our pipelines are designed to process up to 1000k elements per second in batch jobs to meet timeliness and quality requirements, and up to 10k elements per second in stream. These numbers are likely to increase, requiring more effort. Is it worth it? Who will pay for it?</p><p>This article is not about money, but reducing costs and increasing performance can provide a huge boost to your company, especially as green initiatives become increasingly important to consumers. A commitment to sustainability can also provide a significant advantage over your competitors.</p><p>Consider the analogy of being in the forest with a friend and encountering a bear. You don’t need to be faster than the bear; you just need to be faster than your friend. It’s a race to be better.</p><p>Here are some facts to support this:</p><ul><li>88% of consumers are more loyal to companies that support social or environmental issues (Forbes).</li><li>83% of consumers always choose brands with better sustainability records (Wunderman Thompson).</li></ul><p>As engineers, billions of users rely on our products, just as they do with cars. Consider how the automotive industry is changing to meet customer demands for reduced energy costs and sustainability.</p><p>In conclusion, I hope I’ve given you an idea of the environmental impact our industry has and how we can work to reverse this negative footprint. Each one of us can be a man or woman on a mission to make an impact. It’s easier than you might think to contribute to the common good, and there’s plenty of room for us in the IT sector to make a difference, even if nobody has shown us the way.</p><p><strong>Don’t miss out on the latest news and insights from Project Agora! </strong><a href="https://engineering.projectagora.com/"><strong>Subscribe to our blog</strong></a><strong> and stay up-to-date on all the latest developments. If you’re interested in joining our team, be sure to check out our </strong><a href="https://careers.projectagora.com/engineering/"><strong>career page for current job openings</strong></a><strong>. Come join us!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a4cca870fa9" width="1" height="1" alt=""><hr><p><a href="https://engineering.projectagora.com/tech-for-good-reducing-our-environmental-footprint-a4cca870fa9">Tech for Good: Reducing Our Environmental Footprint</a> was originally published in <a href="https://engineering.projectagora.com/">Project Agora Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Showcasing the value of 3-Amigos Principle in test scripts using Robot Framework]]></title>
            <link>https://engineering.projectagora.com/showcasing-the-value-of-3-amigos-principle-in-test-scripts-using-robot-framework-75fbcfc79c3d?source=rss-93d6b25eed24------2</link>
            <guid isPermaLink="false">https://medium.com/p/75fbcfc79c3d</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[robot-framework-library]]></category>
            <category><![CDATA[quality-assurance]]></category>
            <category><![CDATA[qa]]></category>
            <category><![CDATA[robot-framework]]></category>
            <dc:creator><![CDATA[ProjectAgoraEng]]></dc:creator>
            <pubDate>Mon, 20 Mar 2023 16:06:43 GMT</pubDate>
            <atom:updated>2023-03-20T16:12:36.902Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By </em><a href="https://www.linkedin.com/in/nektarios-sourgias/"><em>Nektarios Sourgias</em></a><em>, Senior Quality Assurance Engineer</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B-LJNQ0jPGgvtrkLnmZjUQ.jpeg" /></figure><p>It is a common understanding that effective communication and teamwork are necessary soft skills for anyone working in the IT industry. Lacking the ability to establish proper teamwork can cause problems and the delivery of software. We should be able to learn and understand what we are trying to develop and finally deliver to a customer as software from the prior stages of software design.</p><p><strong>3-Amigos on the rescue</strong></p><p>The principle of 3-Amigos comes to save the day and strengthen the communication process of the knowledge of what will be delivered as software. This principle showcases that the person documenting the software requirements (BA or Product Owner), the developer and the QA Engineer are working together on a horizontal axis as a team. They assist in generating the use cases derived from the business and technical requirements and discussing any obstacle that occurs.</p><p><strong>3 players on the board</strong></p><p>The space where the use cases are generated is the board where each ‘actor’ is playing his/her role and contributes to it by adding or modifying the information contained like adding pieces to a puzzle where the final product is what comes to life from all these puzzle pieces. You may think of it like a game simulation where a person says ‘We want the user to do this, and then the system will do that’, then another person acts a question like ‘What if the user does that, what the action will be?’ and then the developer comes in the way and says ‘Yes, but technically this will be less feasible so when the user wants to do another action, let’s think of another technical solution’.</p><p><strong>QA in the front-line</strong></p><p>Normally, a QA engineer will be having his or her test cases documented after reading the use cases, without having much involvement during the use cases design, and will wait for the kick-off of the test execution. This was the scenario in the waterfall model mainly. Upon the arrival of the Agile model, the role of the QA engineer enhanced on a more dynamic involvement, before the release of a product for testing execution. Powered by the 3-Amigos principle, now the QA Engineer is involved dynamically in the use cases development of a software product which will lead to the test cases preparation and execution thereafter. This approach allows this role to gain a better understanding of the software because he or she will be involved in the discussions of what needs to be done in its early stages. Needless to say, the creation of alternative use case scenarios or questions arising from any given use case can provide important feedback to the other parties involved.</p><p><strong>Use Cases to Test Cases</strong></p><p>Another essential advantage that the 3-Amigos principle offers to the QA Engineer is in creating Test Cases. The QA Engineer can now begin to prepare his or her test cases upon the initial delivery set of the use cases and develop the different scenarios that can lead to additional use cases. This approach is time-consuming for the effort of test case preparation since the entire set of use cases is no longer needed to have the test cases prepared. A basic example of Use Cases is the below picture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/1*2z2MIyPx9S07k3_5AAIOKg.png" /></figure><p><strong>Robot Framework for Automation</strong></p><p>Robot Framework is a popular open-source test automation framework that supports a variety of testing needs, such as acceptance testing, black-box testing, and white-box testing. One of the main benefits of Robot Framework is its simplicity and readability. Test cases are written in a keyword-driven format, which makes them easily understandable by all stakeholders, even those without a technical background.</p><p>In addition, Robot Framework supports a wide range of libraries and plugins, including Selenium for web testing, Appium for mobile testing, and DatabaseLibrary for database testing, among others. This flexibility makes it an ideal tool for end-to-end testing of software products, enabling QA engineers to write and execute tests across different platforms and technologies. When used in conjunction with the 3-Amigos principle, Robot Framework can enhance the communication and collaboration between the BA or Product Owner, developer, and QA Engineer.</p><p>The simplicity of the tool along with the APIs that it offers, allows us to create our Page Object Model design model along with the Selenium API, through a series of test cases containing keywords that describe the actions of the user and the output of the system, in a completely human-readable manner, enabling all parties to understand what we aim to achieve. A basic skeleton which covers all the pages of a web application plus all the actions performed by the user and all the test data of course can be cloned in any other project so that any other QA Engineer can easily use it and create any other test needed. With the assistance of Python whenever required, such as backend testing like DB Testing or API Testing, we can perform end-to-end testing for our systems and also easily maintain them.</p><p><strong>Gherkin as Test Steps in Robot Framework</strong></p><p>Gherkin syntax used in Cucumber.IO employs the so-called BDD (Behavior Driven Development) technique. Under the 3-Amigos principle, we compose the use cases in the BDD context, ensuring a better understanding from all parties involved. Robot Framework offers us this extraordinary flexibility, to create Test Steps for our Test Cases in Gherkin language, thus we can implement in an as-it-is manner, the use cases written step-by-step in test cases. For example, in the image below, we can observe how tests are viewed in Gherkin format within Robot Framework and how the steps in the use cases will be implemented in the test steps:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*g-tJMo2F0BthHv-8Hor7Qw.png" /></figure><p>Moreover, by engaging the QA Engineer in the use case development process, the 3-Amigos principle can ensure that test cases are created earlier in the development cycle, allowing for earlier detection and resolution of defects. This approach can also help to identify any gaps in requirements or design, allowing for these to be addressed before the product is released for testing. The use of Gherkin syntax in Robot Framework’s test cases enables the use cases to be written in a more understandable and consistent format, which helps to reduce misunderstandings and errors.</p><p><strong>Conclusion</strong></p><p>In conclusion, the power that Robot Framework offers, combined with the 3-Amigos principle, can provide business value to the product and minimize communication gaps between all involved parties. Additionally, we can see that the QA role is evolving in all aspects of the software development cycle, both in requirement and analysis, in DevOps by having the QAops role in the CI/CD needs.</p><p><strong>Stay up-to-date with our latest news and insights by subscribing to our </strong><a href="https://engineering.projectagora.com/"><strong>Project Agora blog</strong></a><strong>. And if you’re considering joining our team, don’t forget to check out our current job openings on our </strong><a href="https://careers.projectagora.com/engineering/"><strong>career page</strong></a><strong>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75fbcfc79c3d" width="1" height="1" alt=""><hr><p><a href="https://engineering.projectagora.com/showcasing-the-value-of-3-amigos-principle-in-test-scripts-using-robot-framework-75fbcfc79c3d">Showcasing the value of 3-Amigos Principle in test scripts using Robot Framework</a> was originally published in <a href="https://engineering.projectagora.com/">Project Agora Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Introduction to Pydantic: the powerful Data Validation for your REST APIs]]></title>
            <link>https://engineering.projectagora.com/an-introduction-to-pydantic-the-powerful-data-validation-for-your-rest-apis-a6edfb46b0e8?source=rss-93d6b25eed24------2</link>
            <guid isPermaLink="false">https://medium.com/p/a6edfb46b0e8</guid>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[data-validation]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pydantic]]></category>
            <dc:creator><![CDATA[ProjectAgoraEng]]></dc:creator>
            <pubDate>Thu, 26 Jan 2023 09:20:00 GMT</pubDate>
            <atom:updated>2023-01-26T10:14:42.161Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By </em><a href="https://www.linkedin.com/in/evangelos-soulintzis-ab9b32174/"><em>Evangelos Soulintzis</em></a><em>, Software Engineer</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZaSQdYZIAx3E7Lend2e_fA.jpeg" /></figure><h3>Introduction to Pydantic</h3><p><a href="https://pypi.org/project/pydantic">Pydantic</a> is a Python library created by<em> Samuel Colvin</em> that simplifies the process of data validation. It is a versatile tool that can be utilized in various contexts, such as building APIs, working with databases, and handling data within projects. The library features a simple and intuitive syntax, allowing for easy definition and validation of data models. This includes the ability to specify types, default values, and constraints directly within the code, making it both understandable and maintainable.</p><p>Despite its simplicity, Pydantic is also highly flexible and powerful, providing built-in validators and constraints as well as the capability to create custom validators for further data definition and validation. The library is accessible to both experienced Python developers and those new to the language, offering something for everyone. Join us as we explore the world of Pydantic.</p><h3>Pydantic Features</h3><p>Pydantic is a Python library that provides a range of data validation and parsing features. Some of the main features of Pydantic include:</p><p>1. <strong>Data validation:</strong> Pydantic includes a validation function that automatically checks the types and values of class attributes, ensuring that they are correct and conform to any specified constraints. Custom validation rules using Python-type hinting are also supported, allowing users to define additional constraints on their data.</p><p>2. <strong>Support for complex data types:</strong> Supports a wide range of data types, including lists, dictionaries, and nested models, making it easy to define and validate complex data structures.</p><p>3. <strong>Efficient and lightweight:</strong> Pydantic is designed to be fast and efficient, with a small codebase and minimal dependencies. It is also well-documented and easy to use, making it a popular choice for data validation and parsing in Python.</p><p>4. <strong>Open source:</strong> Pydantic is an open-source library, licensed under the <em>MIT license</em>, and is actively developed and maintained by a community of contributors.</p><h3>Pydantic vs Python Data Classes</h3><p><a href="https://docs.python.org/3/library/dataclasses.html">Python <em>data classes</em></a> are a way to define classes primarily used to store data. They were introduced in Python 3.7 by the @dataclass decorator. When applied to a class, this decorator automatically generates a default __init__ method, a default __repr__ method, and default comparison methods, which simplifies the definition of classes that only need to store data. Additionally, the dataclasses module provides helper functions like dataclasses.asdict() and dataclasses.replace() that can be used to convert a class instance to a dictionary or to create a new instance from the class with updates to the fields respectively. This makes data classes a convenient way to create simple classes that only need to store data and don&#39;t require any other behavior.</p><p>Pydantic is a library that builds on top of Python data classes and adds additional functionality for data validation and parsing. Pydantic provides a special class <em>BaseModel</em> that can be used to define data models and their validation rules. The validation rules can be defined by setting types and constraints on the class attributes and can also be customized with additional rules using Python type hinting. Also provides support for parsing data from sources such as JSON, and can convert the parsed data into an instance of the defined data model. Additionally, can generate a JSON schema from the data model, which can be used to validate input data based on the schema.</p><p>Overall, Pydantic is a library that adds additional functionality for data validation and parsing to Python. While Python <em>data classes </em>provide a convenient syntax for defining simple classes that store data, Pydantic offers a more powerful and flexible solution for validating and parsing data in Python applications. <em>BaseModel</em> class can be used to define data models and their validation rules, it can validate input data and parse it into a Python object and also serialize it to <em>JSON</em> and other formats. This makes Pydantic a valuable tool for ensuring data integrity and consistency in any Python application that deals with data.</p><h3>Examples of using the Pydantic Library</h3><p>One of the main benefits of using Pydantic is that it allows you to define the structure of your data in a declarative way, using Python’s type hints. This means that you can specify the types, default values, and constraints of your data directly in the code, which makes it easy to understand and maintain.</p><p>For example, let’s say you want to define a simple data model for a <em>User</em>, with fields for their username, age, email and password . With Pydantic, you can define this model like this:</p><pre>from pydantic import BaseModel<br><br>class User(BaseModel):<br>    username: str<br>    age: int<br>    email: str<br>    password: str</pre><p>Each field is annotated with a type hint, which specifies the type of data that it can hold. In this case, username , email and password are strings, while age is an integer.</p><p>Pydantic also provides a number of built-in validators and constraints that you can use to further define and validate your data. For example, you can specify that a field is required, or that it must match a certain pattern. You can also specify default values for fields, which can be helpful when working with optional data.</p><p>Here’s an example of how you might use some of these features to further define the User model:</p><pre>class User(BaseModel):<br>    username: str<br>    age: conint(gt=18)<br>    email: EmailStr<br>    password: constr(min_length=8, max_length=16)<br>    phone: Optional[str] = &#39;Unknown&#39;</pre><p>In this example, we’ve used the constr validator to define constrains for the age and password field. that it must be at least 8 characters long. The age field is defined as a conint type with a condition that its value must be greater than 18, and the password field is defined as a constr type with two conditions, the value must be at least 8 characters long and at most 16 characters long. We&#39;ve also used the EmailStr type to specify that the email field must contain a valid email address. Finally, a phonefield has been added which is <em>Optional </em>and has a default value of ‘Unknown’.</p><p>In addition to the built-in validators and constraints provided, you can also create your own custom validators. This can be helpful when you need to enforce more specific or complex rules on your data that are not covered by built-in validators.</p><p>To create a custom validator, you can define a function that takes in a value and performs any necessary validation checks. If the value is valid, the function should return it. If the value is not valid, the function should raise a ValidationError with a message explaining the problem.</p><p>Here’s an example of using custom validators for the User model, similar to the previous example, and showing how data are passed into the Pydantic class for validation and what is the Pydantic output.</p><pre>import re<br>from pydantic import BaseModel, validator<br><br>class User(BaseModel):<br>    username: str<br>    age: int<br>    email: str<br>    password: str<br><br>    @validator(&quot;age&quot;)<br>    @classmethod<br>    def validate_age(cls, value):<br>        if value &lt; 18:<br>            raise ValueError(&quot;User must be adult&quot;)<br>        return value<br><br>    @validator(&quot;email&quot;)<br>    @classmethod<br>    def validate_email(cls, value):<br>        if not bool(re.fullmatch(r&#39;[\w.-]+@[\w-]+\.[\w.]+&#39;, value)):<br>            raise ValueError(&quot;Email is invalid&quot;)<br>        return value <br><br>    @validator(&quot;password&quot;)<br>    @classmethod<br>    def validate_password(cls, value):<br>        password_length = len(value)<br>        if password_length &lt; 8 or password_length &gt; 16:<br>            raise ValueError(&quot;The password must be between 8 and 16 characters long&quot;)<br>        return value<br><br># Valid User<br>valid_user = {<br>    &#39;username&#39;: &#39;test_name&#39;,<br>    &#39;age&#39;: 20,<br>    &#39;email&#39;: &#39;name@test.gr&#39;,<br>    &#39;password&#39;: &#39;123456789&#39;<br>}<br><br># Invalid User<br>invalid_user = {<br>    &#39;username&#39;: &#39;test_name&#39;,<br>    &#39;age&#39;: 16,<br>    &#39;email&#39;: &#39;name_test.gr&#39;,<br>    &#39;password&#39;: &#39;1234&#39;<br>}<br><br>try:<br>    valid_result = User(**valid_user)<br>    print(valid_result)<br>    # Valid Output: username=&#39;test_name&#39; age=20 email=&#39;name@test.gr&#39; password=&#39;123456789&#39;<br>    invalid_result = User(**invalid_user)<br>except ValueError as e:<br>    print(e.errors())<br>    # Invalid Output: <br>    [<br>      {<br>          &quot;loc&quot;:&quot;(&quot;&quot;age&quot;,&quot;)&quot;,<br>          &quot;msg&quot;:&quot;User must be adult&quot;,<br>          &quot;type&quot;:&quot;value_error&quot;<br>      },<br>      {<br>          &quot;loc&quot;:&quot;(&quot;&quot;email&quot;,&quot;)&quot;,<br>          &quot;msg&quot;:&quot;Email is invalid&quot;,<br>          &quot;type&quot;:&quot;value_error&quot;<br>      },<br>      {<br>          &quot;loc&quot;:&quot;(&quot;&quot;password&quot;,&quot;)&quot;,<br>          &quot;msg&quot;:&quot;The password must be between 8 and 16 characters long&quot;,<br>          &quot;type&quot;:&quot;value_error&quot;<br>      }<br>    ]</pre><p>Custom validators can be very powerful and can help you create data models that are tailored to your specific needs. Whether you’re working with simple data types or more complex structures, custom validators can help you define and validate your data in a flexible and reliable way.</p><h3>Pydantic and FastAPI</h3><p>Pydantic is a popular choice for defining data models in <a href="https://pypi.org/project/fastapi">FastAPI</a>, a high-performance Python web framework for building APIs. FastAPI is built on top of <a href="https://pypi.org/project/starlette">Starlette</a>, a lightweight ASGI framework, and integrates seamlessly with Pydantic to provide a fast and easy-to-use way to build APIs.</p><p>One of the main benefits of using Pydantic with <em>FastAPI</em> is that it allows you to define your API’s request and response data in a simple and intuitive way. You can use Pydantic’s data models to define the structure of your request and response data, and <em>FastAPI</em> will automatically validate and convert the data to and from <em>JSON</em> as needed.</p><p>For example, let’s say you want to build an API that allows users to create, retrieve, update, and delete tasks. With <em>FastAPI</em> and Pydantic, you can define the data models for the tasks like this:</p><p>Let’s use the User model from the previous section to define the <em>FastAPI</em> routes for creating, updating, retrieving and deleting a User.</p><pre>from fastapi import FastAPI<br><br>app = FastAPI()<br><br>@app.post(&quot;/users&quot;)<br>def create_user(user: User):<br>    pass<br><br>@app.get(&quot;/users/{username}&quot;)<br>def get_user(username: str):<br>    pass<br><br>@app.put(&quot;/users/{username}&quot;)<br>def update_user(username: str, user: User):<br>    pass<br><br>@app.delete(&quot;/users/{username}&quot;)<br>def delete_user(username: str):<br>    pass</pre><p>In this example, we’ve defined four API routes: one for creating a new user, one for retrieving an existing user, one for updating a user, and one for deleting a user. Each route takes in a request payload (if applicable) and uses Pydantic’s<em> </em>data models to validate and convert the data as needed.</p><p>With <em>FastAPI</em> and Pydantic, you can quickly and easily build APIs that are simple, intuitive, and easy to work with. Whether you’re building a small API or a large-scale application, <em>FastAPI</em> and Pydantic can help you define and validate your data in a reliable and efficient way.</p><h3>Comparison to other data validation libraries</h3><p>There are several other Python libraries that provide similar functionality to Pydantic, such as <em>Marshmallow</em>, <em>Cerberus</em>, and <em>Voluptuous</em>. Here are some ways in which Pydantic compares to these libraries:</p><p><a href="https://pypi.org/project/marshmallow"><strong>Marshmallow</strong></a> is a library for data serialization and validation, it allows to define fields, validations, and other complex data structures which can be used to validate, deserialize and serialize data in various formats.</p><p><a href="https://pypi.org/project/Cerberus"><strong>Cerberus</strong></a> is another library for data validation in Python. It has a declarative syntax similar to marshmallow, but it is focused more on data validation than serialization. One advantage of Cerberus is that it is lightweight and has a smaller footprint than Pydantic or marshmallow. However, Pydantic provides a wider range of features and is more powerful overall.</p><p><a href="https://pypi.org/project/voluptuous"><strong>Voluptuous</strong></a> is a Python library for data validation that is similar to Cerberus in terms of its focus on data validation. It has a declarative syntax and is lightweight, but it does not provide as many features as Pydantic or marshmallow.</p><p>Overall, Pydantic is a more feature-rich library compared to Cerberus and voluptuous, and it provides a different approach to data validation and serialization compared to marshmallow. It may be a good choice depending on the specific needs of your application.</p><h3>Conclusion</h3><p>In <strong>Project Agora</strong>, we use <strong>Pydantic</strong> as our preferred data validation library for REST APIs because of its ease of use, flexibility, and power. With Pydantic’s simple and intuitive syntax, it is easy to define and validate data models, ensuring that the data received by the API is consistent and accurate. Additionally, Pydantic’s built-in validators and constraints, as well as the ability to create custom validators, provide a robust means of defining and validating data. This leads to more reliable and secure APIs that can handle and process data with confidence.</p><p>The official Pydantic documentation can be found <a href="https://docs.pydantic.dev/">here</a>.</p><p><strong>Keep abreast of our latest developments and insights by subscribing to our </strong><a href="https://engineering.projectagora.com/"><strong>Project Agora blog</strong></a><strong>. If you’re inspired to join our team, be sure to check out our current job openings on our </strong><a href="https://careers.projectagora.com/engineering/"><strong>career page</strong></a><strong>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6edfb46b0e8" width="1" height="1" alt=""><hr><p><a href="https://engineering.projectagora.com/an-introduction-to-pydantic-the-powerful-data-validation-for-your-rest-apis-a6edfb46b0e8">An Introduction to Pydantic: the powerful Data Validation for your REST APIs</a> was originally published in <a href="https://engineering.projectagora.com/">Project Agora Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Project Agora’s DevOps Journey]]></title>
            <link>https://engineering.projectagora.com/project-agoras-devops-journey-3764986d6913?source=rss-93d6b25eed24------2</link>
            <guid isPermaLink="false">https://medium.com/p/3764986d6913</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[adtech]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[ProjectAgoraEng]]></dc:creator>
            <pubDate>Mon, 19 Dec 2022 10:15:04 GMT</pubDate>
            <atom:updated>2022-12-19T11:20:14.483Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By </em><a href="https://www.linkedin.com/in/giannis-sialmas/"><em>Giannis Sialmas</em></a><em>, Senior DevOps Engineer</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ERNeV4zp4DpQc_10he4tnA.png" /><figcaption>A DevOps journey to increase the velocity and quality of our products</figcaption></figure><p>Exploring the evolution of CI/CD pipelines and the challenges we faced along the way on our Engineering Team</p><p>In 2018, we started our DevOps journey to increase the velocity and quality of our products.</p><p>At that time, our source code was hosted on Bitbucket and our runtime environments included developers’ laptops and two Kubernetes clusters for staging environment and production. The staging environment was used for integration testing, as well as by the product team for UATs when the development and testing were done by the development team.</p><p>Our first iteration of the CI/CD process used a Jenkins server that responded to webhooks from Bitbucket. We had two long-live branches in our git repository: <strong>dev</strong> and <strong>release</strong>. Developers created feature branches from the dev branch, and when the feature was ready, they merged back to it. The Jenkins server built the microservices, uploaded the generated artifacts in GCP’s Container Registry, templated the image version in kubernetes yaml files hosted in the same repository using bash commands, and finally deployed them to the staging environment. When the development and product teams were done with the staging environment, the dev branch was merged into the release branch, and the process was repeated to deploy to production. The Jenkins server used the git commit hash as the tag for the docker images generated.</p><p>This process had a major flaw because the artifacts tested in the staging environment were not the same as those deployed to production. This was because the docker images were rebuilt in the production pipeline. This caused errors in production, often due to the set of external libraries the application code depends on. In cases where a specific version was not specified for the external libraries, a newer version was used in some production builds, which resulted in either the production build failing or in worse cases runtime errors.</p><p>That led us to the first refactor of our CI/CD pipeline, and thus the second iteration, which introduced 2 new concepts:</p><ul><li>build once deploy many to avoid the above issues</li><li>version artifacts with semantic versioning tags</li></ul><p>In the second iteration of our CI/CD process, we continue to use Jenkins and Bitbucket. Our git strategy included the following branches:</p><ul><li>the <strong>develop </strong>branch, which acted as a buffer branch from which feature branches were started and merged when done.</li><li><strong>release</strong> branches of the format release/&lt;major&gt;.&lt;minor&gt; which were started from development when a release to staging was planned.</li><li>the <strong>master </strong>branch representing the production environment.</li></ul><p>Developers started new feature branches from the development, and senior developers reviewed changes in Pull Requests (PR) to decide which features would be included in the next release.</p><p>At the end of each sprint, a release branch was created from the development (e.g., 2.1). The Jenkins server identified the microservices that had been changed on the release branch, built the respective microservices, and added semantic version tags using the release branch name of the major and minor parts and an auto-incrementing integer for the patch part. The Jenkins server then uploaded the artifact to GCP’s Container Registry and deployed them to the staging environment by injecting the semantic version into the microservices manifest files using bash commands.</p><p>After the development and product team completed testing and UATs, the release branch was merged with to master, the generated git tags were transferred to the master branch, which triggered the production pipeline. The production pipeline deployed the microservices to production without rebuilding them.</p><p>The second iteration worked well for a while, but as the demand for new features increased, we encountered a new problem. Multiple features were included in the release branches and were coupled for release together. Sometimes the features’ deployment was blocked for business reasons, which required us to spend a lot of time using git to remove the specific feature from the release branch.</p><p>The complexity increased exponentially with the addition of the QA team.</p><p>That led us to the third iteration of our CI/CD pipelines.</p><p>After much trial and error, we have decided to adopt GitHub flow as our main git strategy. GitHub flow allows us to keep all of our features separate, and release them as needed. In GitHub flow, the only long-lived branch is the main branch. All feature branches start from the main and are only merged back into it after deployment to production. Hotfixes are treated the same as features.</p><p>When a developer believes that he has completed the requested functionality in their branch, a Pull Request is initiated from the feature branch to the main. When the Pull Request is opened or updated, the first CI pipeline triggers.</p><p>This first pipeline detects the changed microservices in the feature branch in regards to the main, builds them to ensure the new code has not broken the build process, runs unit tests, calculates code coverage and finally runs a Sonarqube scan. The results of the sanity build and Sonarqube analysis are automatically posted as a comment by a Github “service account” on the open Pull Request, which helps with code review. If the reviewer requests changes, the developer can push a new code to the feature branch and the above steps are repeated.</p><p>Once the reviewer is satisfied with the feature branch, a semantic versioning git tag is applied to the HEAD of the feature branch. This triggers the second CI pipeline which builds changes microservices, tags the generated artifacts (docker images, pip packages, npm packages, zips, etc.) with the semantic version, and pushes the artifacts to the company’s central artifact registry in GCP.</p><p>The CI pipelines have been built upon GCP’s Cloudbuild product, and the artifact registry uses a combination of the GCP’s Storage and Artifact Registry products.</p><p>To be more flexible, we have decoupled the CI and CD pipelines. Our main runtime environments are development, QA, non-production and production, but we may need additional environments depending on the circumstances. As a rule of thumb, we use a separate GCP project for each product and environment within a single GCP Organization.</p><p>A separate repo exists in GitHub which hosts Terraform configuration and is used to deploy changes to our GCP product projects. Each product has its own CD repository with several branches, most of which represent a runtime environment. When we need to deploy to an environment, we commit the new code to the appropriate branch. The new commits are picked up by our CD pipeline which applies the Terraform configuration to the corresponding environment. Code commits to non-environment branches plans the Terraform configuration so that an engineer can review it before applying it. The Terraform configuration includes cloud resources needed by the product workloads, as well as the workloads themselves in the form of helm charts or serverless Cloudruns and Cloudfunctions.</p><p>The CD pipelines is also built using GCP’s Cloudbuild product.</p><p>The third iteration of the CI/CD system has been working with efficiency for the past few years. But the journey never ends, as we are always looking for ways to improve and keep up with new business needs and technologies.</p><p><strong>Thanks for reading! Learn more about our Engineering Department </strong><a href="https://careers.projectagora.com/engineering"><strong>here</strong></a><strong>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3764986d6913" width="1" height="1" alt=""><hr><p><a href="https://engineering.projectagora.com/project-agoras-devops-journey-3764986d6913">Project Agora’s DevOps Journey</a> was originally published in <a href="https://engineering.projectagora.com/">Project Agora Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[6 Things to have in mind when starting Unit Testing]]></title>
            <link>https://engineering.projectagora.com/6-things-to-have-in-mind-when-starting-unit-testing-e7d312e30111?source=rss-93d6b25eed24------2</link>
            <guid isPermaLink="false">https://medium.com/p/e7d312e30111</guid>
            <category><![CDATA[unit-testing]]></category>
            <category><![CDATA[unit-test-tools]]></category>
            <category><![CDATA[unit-test-framework]]></category>
            <category><![CDATA[adtech]]></category>
            <dc:creator><![CDATA[ProjectAgoraEng]]></dc:creator>
            <pubDate>Wed, 07 Dec 2022 11:42:41 GMT</pubDate>
            <atom:updated>2022-12-09T10:08:23.490Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By </em><a href="https://www.linkedin.com/in/vasilis-vasilatos-99957460/"><em>Vasilis Vasilatos</em></a><em>, Tech Lead and Engineering Manager of Data</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F3Igv7jVbgRbpekxn64QfQ.png" /></figure><p><strong>1. Introduction and Purpose</strong></p><p>Unit testing is an important piece of the development cycle of a software project/product development. There are multiple methodologies for unit testing and multiple frameworks that support these technologies. The main purpose of unit test is to reduce escaped bugs by testing the lowest partitioning of logic in the code, and the method, and to provide safety when refactoring.</p><p>Speaking of frameworks, for python projects, we use the ‘pytest’ framework for executing the unit tests. We use ‘coverage’ to measure coverage and create relevant reports. So anything that we write should be compatible with these two frameworks.</p><p>Starting from the basics and providing a common structure to help organize the test suites and build the software, all unit tests should be contained in a folder named tests/unit that can be found at the same level as the folder that includes the source code. That is because the tests should be as important as the source code but they should not be part of the source code. By doing this, it is easy to find the tests, but separate them during the delivery of the source code. In addition, it makes it facilitates the configuration of external scanners e.g. static code analysis to scan only the source code for problems and not include the tests as well.</p><p>Now, more specifically, the structure in the tests/unit should be the same as in the source code folder. E.g. if we have a controller.py in the app/controllers folder then we will create a test_controller.py in the tests/unit/controllers folder. This can make searching for a specific test easier and it can support running individual test suites, related to them more easily by running tests in a specific folder.</p><p>Nowadays most frameworks can easily and automatically identify the tests by their names. For that reason, we need to apply a naming convention to our tests so that the tests can be discovered and executed without big effort. So the test should have the same name as the module under test with the test_ prefix.</p><p>Below you can find an example of how the structure should be:</p><pre>my_project/ ├── app/ │ ├── controllers/ │ ├── a.py │ └── b.py └── tests/ ├── unit/ ├── controllers/ ├── test_a.py └── test_b.py</pre><p><strong>2. Test suites inclusion</strong></p><p>Inside the test file, we should include all the test cases needed to have as much coverage as possible. The goal is 100%. However, coverage is not the only important thing. If there are more cases, then they should be added as separate cases, even if they don’t add coverage. Test case names should start with the test_ prefix and the name after the prefix should be descriptive.</p><p><strong>3. What do we test?</strong></p><p>The test to be accurate should only have one subject under consideration at a time. This means that we only need to test one class-module when executing a test case.</p><p>Tests should only check the public methods of the test class-module. It is described as best practice to avoid testing private methods since we want to allow internal refactoring and since the scope of unit test, is to test the output of the unit for a specific input regardless of the internal private handling. However, for internal handling, mutation checking is required, but we will cover that in the next paragraphs. In the case of a complex private method, that we believe, that a unit test is required, we should check if this private method must be extracted in another module as a public method and imported from there. This of course shouldn’t drive the software design but testing a private method should only take place for rare and special cases.</p><p>Constructors should also be tested. They are really important and are part of the public interface of each class module.</p><p>All external dependencies should be mocked. Even the simplest ones since their functionality should not be included in the test. As described above, only one module at a time should be tested. There are other QA methods for end-to-end testing. Unit testing shouldn’t be used for this purpose.</p><p>Another point is that Exceptions and Errors should be tested too. They are also paths of the source code. For that reason, unit tests should cover the raising, handling and messaging/logging.</p><p><strong>4. Mutation Test</strong></p><p>Unit testing should also include testing against the mutation of the code. That means that if the code and the internal private handling change, then the tests should fail. For this using assertions like called_once and called_with for the external/mocked dependencies are suggested. Moreover, there is a need to use values that would protect against cutting edges. for example if in the code there is a statement if x&gt;=5 the test should be provided using the “5” as x to be protected in a mutation like if x&gt;5.</p><p><strong>5. Unit vs Integration Test</strong></p><p>At this point, there is a need to distinguish the unit tests from the integration tests. Take for example testing a client for a database. For the unit test, we DO NOT use wrappers, memory frameworks, or in memory frameworks or other runtime solutions. WE MOCK. The unit tests should depend only on the source code and its own basic dependencies eg mock.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*gXL00Ey6dbDwe_aA.png" /></figure><p><strong>6. Execution of Tests</strong></p><p>Each test case should include all the mocks it requires to run. This is for readability and clearance purposes. For this, we don’t refactor with common private methods and we exclude the test code from duplication checks in sonar. If we have common functionality, we can check the setUp and tearDown methods though.</p><p>Each test case should be totally independent of the rest. The same is for the test suites. They should be able to run in parallel or in a different sequences each time even if they are executed in the same execution series (same process). That is the reason why mocking file I/O is a must. There might be a possibility that two test cases require the same file. This will cause race condition risks that we want to avoid. For that reason, we mock since I/O access functionality usually is an external dependency.</p><p>Dates should be used with caution. Adding a date in the long feature thinking that the test will never make it introduces risks and points out test design problems.</p><p><strong>Feel free to connect and follow our </strong><a href="https://engineering.projectagora.com/"><strong>Project Agora blog page</strong></a><strong> to learn more insights and news from our Engineering Team. Interested in working with us? We’re hiring! Visit our page </strong><a href="https://careers.projectagora.com/engineering/"><strong>here </strong></a><strong>to find our open opportunities!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7d312e30111" width="1" height="1" alt=""><hr><p><a href="https://engineering.projectagora.com/6-things-to-have-in-mind-when-starting-unit-testing-e7d312e30111">6 Things to have in mind when starting Unit Testing</a> was originally published in <a href="https://engineering.projectagora.com/">Project Agora Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Quality matters to the whole engineering department end to end]]></title>
            <link>https://engineering.projectagora.com/quality-matters-to-the-whole-engineering-department-end-to-end-10921fc6e770?source=rss-93d6b25eed24------2</link>
            <guid isPermaLink="false">https://medium.com/p/10921fc6e770</guid>
            <category><![CDATA[quality-assurance]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[product-development]]></category>
            <category><![CDATA[product-design]]></category>
            <category><![CDATA[adtech]]></category>
            <dc:creator><![CDATA[ProjectAgoraEng]]></dc:creator>
            <pubDate>Thu, 01 Dec 2022 15:11:22 GMT</pubDate>
            <atom:updated>2022-12-01T15:11:22.012Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By </em><a href="https://www.linkedin.com/in/panagiotisdimitroulas/"><em>Panagiotis Dimitroulas</em></a><em>, Senior Engineering Manager</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*56rX0Gmcj9Km2b3a" /></figure><p>In this article I will combine a range of several processes, not only QA centralized, which increases product quality, minimizing the cost needed to catch the bugs/defects prior to production. We have already implemented some of them in the Engineering Department of <a href="https://projectagora.com/">Project Agora</a> and soon we will be able to reveal more.</p><p>The picture below summarizes how much costs increased the later we found a bug. This is not an easy-to-solve problem but all the below processes are really critical towards achieving our goal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4UFM2OMtS38ZjOq7" /></figure><h4>Requirements Review</h4><p>In order to start, we need documented requirements but do we review them and who is participating in the review meeting?</p><p>Requirements specification document should be prepared by Product Managers or Product Owners and delivered to a committee consisting of Architects, Dev Leaders and QA Leaders for review. A review meeting should have been scheduled in order to enable the participants to study the document and point out their comments/improvements. I am sure that during this process, questions may arise such as; does the implementation of feature1 impact already implemented feature2; should we redesign our system to make it more scalable needed for ferature3; what are the performance/capacity needs etc? Answering these questions and many others generated during the meeting, relative defects will never appear.</p><h4>Feasibility Study Review</h4><p>Or feasibility analysis is the process used to go deeper and closer to the architecture and design phase. This phase’s output is a feasibility study document which describes higher than the design level how the requirements will be implemented. Feasibility documents are prepared by Architect or Dev Leads and again should be reviewed by a committee consisting of POs, Architects, Dev Leaders and QA Leaders following the same as the above process. QA Leaders should use the feasibility documents to prepare the test plans which include the scope, approach, resources and schedule of intended test activities, the features to be tested, the test environment, the test design techniques, and entry and exit criteria to be used for any risks requiring contingency planning. QA Engineers will have the opportunity to share and get knowledge on technical restrictions and risk factors. Test plans are used as input for the test case preparation which is also reviewed by dev leaders.</p><h4>Architecture and Design Phase Review</h4><p>Architecture and design docs are prepared by the Architect and Dev Leads which should be reviewed apart from dev by QA too. QA Leads get the latest info about how the products/features will be implemented and using the already approved test plans are creating the test suites almost in parallel, meaning that design docs and test suits might be completed in the same scrum sprint or really close. Thus, the QA execution begins right after the source code implementation. Furthermore, since QA Engineers know the system end to end, may have comments in the review related to the proposed implementation prior to this which definitely leads to fewer bugs/defects.</p><h4>Unit Tests Scripts</h4><p>A process I really like and have seen that accelerates automation is the use of οf scripts, already prepared by Developers for unit tests automation, by QA. In doing so, QA does not begin from scratch to prepare its own scripts which should be extended to cover more cases on functional, and non-functional levels (both normal and abnormal).</p><h4>Builds/Smoke Tests/Blaze Tests</h4><p>Smoke tests should have passed to meet the minimum requirements so as to promote a build to Dev. For example, there is no gain for dev to get a build which fails to be installed and crashes from the beginning. Doing so, we free up dev’s time and pushes for fixes at an earlier stage. Similarly, with blaze tests, a set of second-level automated tests are executed which exercises basic features of the product which should work in any case so as to be promoted to QA.</p><h4>QAOps Framework</h4><p>It has been proved that a great way to speed up and increase quality is to introduce QAOps framework into QA processes, automation and reporting dashboards enhancing continuous integration (CI), continuous delivery (CD), and continuous testing (CT). Therefore, QA engineers must work closely with software developers and DevOps in the CI/CD pipeline.</p><h4>New Features’ and regression tests</h4><p>New features and regression tests should be automated to a presentence close to 60% or even more. Automated test cases can be executed any time any day and help to release faster but also respond to preparing patches with fixes, thus avoiding long time manual testing. What about the rest 40% or less? It has been proved that we don’t want to automate all test cases either because these are either corner cases or too complicated and need a lot of effort to be automated.</p><h4>Quality Metrics</h4><p>Part of the quality metrics should be the number of defects we face when our product is live and the areas affected by the defects. This is a great input for QA to plan extra customer-related test cases and by automating to add them to the regression lists.</p><h4>RCA</h4><p>An organization that wants to constantly improve should always wonder why the defects slipped and which part of the engineering organization should take the appropriate action to avoid this happening again. This process is called Route Cause Analysis and could take part in defect triage meetings that QA is leading occasionally.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Oi2TGE8Kowh7MBu5" /></figure><h4>Conclusion</h4><p>The aim of this article is to summarize essential processes toward minimizing defects slippage and increasing quality leading to product improvements. It is more than obvious that quality processes do not only affect the QA department. Quality matters to the whole engineering department end to end. This is something we have realized in <a href="https://projectagora.com/">Project Agora</a> and we have taken the needed initiatives to act as pioneers on the higher possible level. Do you?</p><p><strong>Thanks for reading! Learn more about our Engineering Department </strong><a href="https://careers.projectagora.com/engineering"><strong>here</strong></a><strong>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=10921fc6e770" width="1" height="1" alt=""><hr><p><a href="https://engineering.projectagora.com/quality-matters-to-the-whole-engineering-department-end-to-end-10921fc6e770">Quality matters to the whole engineering department end to end</a> was originally published in <a href="https://engineering.projectagora.com/">Project Agora Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>