<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[GreenM - Medium]]></title>
        <description><![CDATA[Shaping Healthcare Technology landscape with data-driven solutions! - Medium]]></description>
        <link>https://medium.com/greenm?source=rss----ac7f8be0ecdb---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>GreenM - Medium</title>
            <link>https://medium.com/greenm?source=rss----ac7f8be0ecdb---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 00:33:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/greenm" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A comprehensive guide to EHR optimization strategies]]></title>
            <link>https://medium.com/greenm/a-comprehensive-guide-to-ehr-optimization-strategies-98a27813f2d8?source=rss----ac7f8be0ecdb---4</link>
            <guid isPermaLink="false">https://medium.com/p/98a27813f2d8</guid>
            <category><![CDATA[healthcare-technology]]></category>
            <category><![CDATA[healthcare]]></category>
            <category><![CDATA[emr]]></category>
            <category><![CDATA[ehr]]></category>
            <category><![CDATA[healthcare-innovations]]></category>
            <dc:creator><![CDATA[Serhii Tereshchenko]]></dc:creator>
            <pubDate>Fri, 09 Feb 2024 16:03:21 GMT</pubDate>
            <atom:updated>2025-11-19T10:42:39.419Z</atom:updated>
            <content:encoded><![CDATA[<p>Would Apple’s performance have been so high if it had not been optimized continuously? Autopilot systems can be put into the same basket, given their accuracy, which allows us to entrust our lives to them.</p><p>The same applies to Electronic Health Records (EHR) that revolutionized the healthcare industry by digitizing patient information, streamlining workflows, and enhancing the quality of <a href="https://greenm.io/shaping-personalized-care-with-generative-ai-in-healthcare/">personalized care</a>.</p><blockquote><strong><em>Some interesting figures</em></strong>: According to a <a href="https://onlinelibrary.wiley.com/doi/full/10.1002/hcs2.21">survey</a>, 82% of physicians who used EHR reported improved clinical decision-making, 92% reported improved communication with other providers and their patients, and 82% reported reduced medication errors. As per another <a href="https://www.beckershospitalreview.com/ehrs/96-of-us-hospitals-have-ehrs-but-barriers-remain-to-interoperability-onc-says.html">report</a>, 96% of US hospitals have EHRs, but barriers remain to compatibility.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NrzWiXVCH8EtegMx3GpTGQ.jpeg" /></figure><p>However, despite their many benefits, EHR systems often face challenges such as usability issues, inefficiencies, and <a href="https://greenm.io/the-role-of-data-mapping-in-interoperability-in-healthcare/">healthcare interoperability</a> problems.</p><p>Thus, the mere presence of EHR systems is not enough to guarantee streamlined operations and optimal patient outcomes. To harness the full potential of these digital tools, healthcare organizations must implement EHR optimization strategies.</p><p>In this guide, we discuss the intricacies of EHR optimization, which empowers healthcare providers to unlock the true potential of their digital infrastructure.</p><h3>Why should you do EHR optimization?</h3><p>EHR optimization is essential for delivering high-quality, patient-centered care. By optimizing EHR systems, you can improve efficiency, level up patient care, achieve cost savings, ensure regulatory compliance, contribute to <a href="https://greenm.io/what-is-predictive-analytics-in-healthcare-and-its-efficient-usage/">predictive analytics</a>, gain data-driven insights, facilitate interoperability, and promote patient satisfaction and engagement, that is to say:</p><h4>Enhanced clinical workflows</h4><p>Optimized EHR systems streamline clinical workflows in healthcare, reducing administrative burdens and freeing up clinicians’ time to focus on patient care. Customized templates, automated processes, and integrated decision support tools improve efficiency and productivity across healthcare settings.</p><h4>Improved patient care</h4><p>Optimization of EHR systems leads to better patient care through enhanced clinical decision-making, care coordination, and patient engagement. Access to comprehensive patient information, real-time clinical guidance, and communication tools facilitates more informed and personalized care delivery, ultimately improving patient outcomes and satisfaction.</p><h4>Cost savings</h4><p>Optimized EHR systems can lead to cost savings for healthcare providers by reducing redundant tasks, minimizing errors, and improving resource utilization. Streamlined workflows, interoperable systems, and data-driven decision-making help optimize resource allocation, leading to more efficient and cost-effective healthcare delivery.</p><h4>Patient satisfaction and engagement</h4><p>Optimized EHR systems contribute to <a href="https://greenm.io/how-to-enhance-customer-satisfaction-with-machine-learning/">increased patient satisfaction</a> and engagement by providing convenient access to health information, communication channels with healthcare providers, and tools for self-management. Patient portals, secure messaging, and online appointment scheduling empower patients to take an active role in their healthcare, leading to stronger patient-provider relationships and improved outcomes.</p><h4>Regulatory compliance</h4><p>EHR optimization ensures that healthcare providers remain compliant with regulatory requirements and industry standards. Regular updates, security enhancements, and adherence to privacy regulations such as HIPAA help protect patient information and mitigate compliance risks, avoiding potential fines, penalties, and legal liabilities.</p><h4>Healthcare data-driven insights</h4><p>Optimized EHR systems provide access to valuable data insights that can inform clinical decision-making, quality improvement initiatives, and strategic planning. Data analytics tools enable healthcare providers to identify trends, monitor key performance indicators, and track outcomes, leading to evidence-based practices and continuous performance improvement.</p><h4>Interoperability and care coordination</h4><p>Optimizing EHR facilitates interoperability and seamless data exchange between different healthcare settings and systems. Improved care coordination, information sharing, and communication among care teams enhance continuity of care, reduce medical errors, and promote collaboration across the healthcare continuum.</p><h3>What exactly are you supposed to do in EHR optimization strategies?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xmY2ZTpZFagaNYAYSfRdsQ.jpeg" /></figure><h4><strong>1. Conduct an EHR assessment</strong></h4><ul><li>Before implementing any optimization strategies, it’s essential to conduct a thorough assessment of your current EHR system.</li><li>Identify areas of improvement, user pain points, workflow inefficiencies, and system performance issues.</li><li>This assessment will serve as a baseline for developing targeted optimization strategies tailored to your organization’s specific needs.</li></ul><h4><strong>2. Data integration</strong></h4><ul><li>Integrate data from various sources such as laboratory systems, imaging systems, and external databases, including <a href="https://medium.com/greenm/7-strategies-for-strengthening-healthcare-iot-security-5e9e8510b3cd">IoMT devices</a> and external systems, taking into account security.</li><li>Ensure that all relevant patient information is accessible within the EHR system.</li><li>Take seamless <a href="https://greenm.io/key-considerations-for-successful-data-integrations/">data integration steps,</a> which can also include data preparation and cleansing, to reduce duplicate data entry and enhance the completeness and accuracy of patient records.</li></ul><h4><strong>3. Customize EHR templates and workflows</strong></h4><ul><li>Customize templates to specific specialties or clinical settings to capture relevant patient information efficiently while minimizing data entry burden.</li><li>Work closely with clinicians and staff to design workflows that align with their needs and optimize processes for maximum efficiency.</li><li>Streamline data entry processes to reduce documentation burden.</li></ul><h4><strong>4. Optimize system performance</strong></h4><ul><li>Fine-tune hardware infrastructure to improve system speed and responsiveness.</li><li>Optimize software configurations for optimal performance.</li><li>Implement performance monitoring tools to identify and address bottlenecks promptly.</li></ul><h4><strong>5. Implement Clinical Decision Support (CDS) tools</strong></h4><ul><li>Integrate CDS tools into the EHR system to provide real-time clinical guidance and evidence-based recommendations at the point of care.</li><li>Customize CDS alerts and reminders to align with clinical guidelines and best practices.</li><li>Continuously update CDS algorithms to reflect the latest medical evidence and guidelines.</li></ul><h4><strong>6. Enhance interoperability</strong></h4><ul><li>Implement standards-based interoperability solutions to enable seamless data exchange between EHR systems, other healthcare systems, healthcare providers, and other stakeholders.</li><li>Ensure compliance with industry standards such as HL7, FHIR, and DICOM.</li><li>Establish secure data-sharing agreements with external healthcare organizations to facilitate care coordination.</li></ul><h4><strong>7. Foster user engagement and feedback</strong></h4><ul><li>Solicit feedback from clinicians and staff on EHR usability and functionality and get it analyzed with a <a href="https://trevise.ai/">sentiment analytics tool</a>.</li><li>Create user feedback mechanisms such as surveys, focus groups, and user forums to gather insights into user experiences, challenges, and suggestions for improvement.</li><li>Use user feedback to prioritize optimization initiatives and drive user adoption.</li></ul><h4><strong>8. Continuously monitor and evaluate</strong></h4><ul><li>Establish key performance indicators (KPIs) to measure the effectiveness of optimization strategies and track progress over time.</li><li>Regularly review performance metrics, user feedback, and workflow data.</li><li>Identify areas for further improvement and refinement based on evaluation results.</li></ul><h4><strong>9. Leverage data analytics and reporting</strong></h4><ul><li>Utilize data analytics tools within the EHR system to gain insights into patient populations, clinical outcomes, and operational performance.</li><li>Generate reports to monitor quality metrics, track key performance indicators, and identify trends, involving healthcare data science.</li><li>Use data-driven insights to identify trends, monitor quality metrics, and drive evidence-based decision-making across the organization.</li></ul><h4><strong>10. Stay updated with industry trends and best practices</strong></h4><ul><li>Stay updated with industry trends, including <a href="https://greenm.io/top-8-data-trends-after-the-covid-19-crisis/">data-driven trends</a>, regulatory changes, and best practices related to EHR optimization.</li><li>Participate in professional networks, conferences, and educational events to exchange knowledge and stay ahead of the curve.</li><li>Collaborate with EHR vendors and industry experts to leverage new technologies and innovations in EHR optimization.</li></ul><h4><strong>11. Implement healthcare customer relationship management</strong></h4><ul><li>Utilize EHR systems to implement healthcare customer relationship management strategies.</li><li>Engage with patients through patient portals, appointment reminders, and health education resources embedded within the EHR platform.</li><li>Personalize patient interactions and improve communication to enhance patient satisfaction and loyalty.</li></ul><h4><strong>12. Integrate healthcare technology management</strong></h4><ul><li>Incorporate healthcare technology management practices into EHR optimization efforts.</li><li>Collaborate with healthcare software development companies to customize and maintain EHR solutions.</li><li>Ensure compliance with regulatory standards and implement data security measures to safeguard patient information.</li></ul><h4><strong>13. Ensure data governance</strong></h4><ul><li>Establish <a href="https://greenm.io/why-data-governance-is-important-in-healthcare/">data governance</a> policies and procedures to ensure the quality, integrity, and security of EHR data.</li><li>Define roles and responsibilities for data management, access controls, and data stewardship within the organization.</li><li>Implement data privacy measures and adhere to regulatory requirements to protect patient confidentiality and privacy.</li></ul><h4><strong>14. Promote user training and teamwork</strong></h4><ul><li>Offer thorough training programs for clinicians and staff on EHR system functionalities.</li><li>Provide ongoing training and support to keep users updated on system updates and enhancements.</li><li>Encourage collaboration and teamwork among clinicians, staff, and IT personnel involved in EHR optimization efforts.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QdW7BqKaC2ceK76moIYEuA.jpeg" /></figure><h3>Conclusion: your successful EHR optimization is around the corner</h3><p>Now you are one step closer to unlocking the full potential of your EHR system. It turns out that EHR optimization is imperative for delivering high-quality, patient-centered care with efficiency, precision, and cost-effectiveness.</p><p>It leads to streamlining clinical workflows, improving patient outcomes and satisfaction, ensuring regulatory compliance, fostering interoperability, data-driven insights, seamless care coordination, and engagement facilitated by comprehensive patient information and communication tools.</p><p><a href="https://greenm.io/contacts/"><strong>Get in touch with us</strong></a><strong> </strong>if you need more support in optimizing EHR or enhancing your healthcare domain generally. Stay tuned. See you soon!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=98a27813f2d8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/greenm/a-comprehensive-guide-to-ehr-optimization-strategies-98a27813f2d8">A comprehensive guide to EHR optimization strategies</a> was originally published in <a href="https://medium.com/greenm">GreenM</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[7 strategies for strengthening Healthcare IoT security]]></title>
            <link>https://medium.com/greenm/7-strategies-for-strengthening-healthcare-iot-security-5e9e8510b3cd?source=rss----ac7f8be0ecdb---4</link>
            <guid isPermaLink="false">https://medium.com/p/5e9e8510b3cd</guid>
            <category><![CDATA[healthcare-iot-security]]></category>
            <category><![CDATA[healthcare-technology]]></category>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[iomt]]></category>
            <category><![CDATA[healthcare]]></category>
            <dc:creator><![CDATA[Serhii Tereshchenko]]></dc:creator>
            <pubDate>Wed, 07 Feb 2024 15:16:12 GMT</pubDate>
            <atom:updated>2025-11-26T17:44:24.313Z</atom:updated>
            <content:encoded><![CDATA[<p>In the modern world, we cannot imagine everyday life without communication devices such as smartphones or computers, as they are indispensable for staying connected, accessing information, and conducting daily tasks.</p><p>Likewise, as healthcare embraces the symphony of connected devices through the Internet of Things (IoT), the healthcare industry cannot function to the fullest without utilizing IoT/ Internet of Medical Things (IoMT) devices.</p><p>With them, it is possible to envision a seamless network of interconnected devices — Healthcare IoT — a web of information facilitating swift diagnoses, <a href="https://greenm.io/what-is-predictive-analytics-in-healthcare-and-its-efficient-usage/">predictive analytics</a>, and <a href="https://greenm.io/shaping-personalized-care-with-generative-ai-in-healthcare/">personalized patient care</a>.</p><p>Yet, as this digital frontier expands, so do the concerns surrounding the security of this interconnected ecosystem.</p><blockquote><strong><em>Some interesting figures</em></strong>: According to the <a href="https://www.cynerio.com/insecurity-of-connected-devices-in-healthcare-2022?submissionGuid=1456c36f-b0d0-4912-a7e1-870cabf42ef8">report</a> by the Ponemon Institute, among the top threats to IoT and other connected devices were lack of visibility into IoT networks (45%), phishing (45%), zero-day attacks (41%), and ransomware attacks (39%). 43% of healthcare organizations in the US experienced an average of at least one ransomware attack.</blockquote><blockquote>56% of healthcare organizations faced at least one cyberattack in the last 24 months involving an IoT/IoMT device. 43% of respondents suffered at least one data breach in the prior 24 months. 71% of healthcare providers rated the risk as high or very high (7 or higher), but only 21% of healthcare providers reported having proactive security actions in-house.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HLxCBfQ88mry9UcYoqBMqA.jpeg" /></figure><p>This means a greater number of healthcare providers could have implemented security measures. Since IoT became one of the <a href="https://greenm.io/top-8-data-trends-after-the-covid-19-crisis/">data-driven trends after COVID-19</a>, its infrastructure shall be secure enough.</p><p>This article navigates the currents of Healthcare IoT security, unraveling strategies to safeguard patient confidentiality and the integrity of medical data in this dynamic age of technological advancement.</p><h3>What are IOMT vulnerabilities?</h3><p>The integration of <a href="https://greenm.io/what-impact-will-ai-and-iot-have-on-healthcare-data-security/">IoT certainly impacts healthcare data security</a>. At the same time, IoT itself is exposed to security vulnerabilities and risks. Let’s consider them more deeply:</p><ul><li><strong>Lack of security standards</strong>: Many IoT healthcare devices lack standardized security protocols and guidelines, leading to inconsistencies in security implementations across different devices and manufacturers. This lack of uniformity makes it challenging to assess and address security vulnerabilities effectively.</li><li><strong>Inadequate authentication mechanisms</strong>: Some IoMT devices may employ weak or default authentication mechanisms, such as hardcoded passwords or lack of multifactor authentication (MFA). This leaves devices vulnerable to unauthorized access and exploitation by malicious actors.</li><li><strong>Insufficient encryption in healthcare</strong>: Data transmitted between IoMT devices and backend systems may be inadequately encrypted, making it susceptible to interception and unauthorized access. Weak encryption algorithms or improper key management practices can compromise the confidentiality and integrity of sensitive patient information.</li><li><strong>Wireless communication vulnerabilities</strong>: IoMT devices often rely on wireless communication protocols, such as Wi-Fi, Bluetooth, or Zigbee, for connectivity. These wireless technologies may be susceptible to eavesdropping, jamming, or man-in-the-middle attacks, particularly if not properly configured or secured.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JGVp1KlpkGP629J9feKgYA.jpeg" /></figure><h3>What are healthcare IoT security challenges?</h3><ul><li><strong>Data privacy and confidentiality:</strong> Healthcare IoT devices collect and transmit sensitive patient data, ranging from medical histories to real-time health metrics. Ensuring the privacy and confidentiality of this information is paramount to maintaining patient trust and compliance with regulatory requirements.</li><li><strong>Healthcare cybersecurity threats:</strong> The interconnected nature of Healthcare IoT systems renders them susceptible to cyberattacks, including malware infections, ransomware attacks, and data breaches. These threats can disrupt healthcare operations, compromise patient safety, and result in significant financial losses.</li><li><a href="https://greenm.io/the-role-of-data-mapping-in-interoperability-in-healthcare/">Interoperability in healthcare</a>: Integrating IoMT devices with existing healthcare IT systems and networks can introduce interoperability challenges, hindering seamless communication and data exchange. Incompatible protocols, misconfigured interfaces, and data format discrepancies may expose devices to exploitation and compromise.</li><li><strong>Regulatory compliance</strong>: Healthcare organizations must navigate a complex landscape of regulatory requirements and industry standards governing the security and privacy of patient data. Non-compliance with regulations such as the Health Insurance Portability and Accountability Act (HIPAA) can result in severe penalties and reputational damage.</li></ul><h3>How can these challenges be addressed with security solutions for IoT?</h3><p>Securing IoT devices requires best practices. By implementing comprehensive security solutions tailored to the unique needs of IoT environments, healthcare organizations can mitigate risks and ensure the integrity of their IoT deployments.</p><p>From smooth <a href="https://greenm.io/key-considerations-for-successful-data-integrations/">data integration</a> and <a href="https://greenm.io/why-data-governance-is-important-in-healthcare/">data governance</a> to <a href="https://greenm.io/the-greenm-healthcare-data-platform-framework/">designing a data platform</a> and <a href="https://greenm.io/how-to-start-a-telemedicine-practice-main-software-challenges/">starting a telemedicine practice</a> involving IoT devices, these solutions encompass a range of technical, policy, and organizational measures aimed at enhancing visibility, strengthening authentication, encrypting data, and proactively monitoring for security incidents.</p><p>Let’s explore these solutions in detail to understand how they can effectively mitigate the risks associated with healthcare IoT security challenges.</p><h4><strong>1. Data encryption and privacy:</strong></h4><ul><li>Encrypt sensitive data both at rest and in transit to protect it from unauthorized access and interception.</li><li>Implement HIPAA encryption standard strong algorithms and key management practices to safeguard patient information and maintain data confidentiality.</li><li>Utilize secure communication protocols, such as Transport Layer Security (TLS), to ensure the integrity and confidentiality of data transmitted between IoT devices and backend systems.</li></ul><h4><strong>2. Strong authentication and access control:</strong></h4><ul><li>Enable Multi-Factor Authentication (MFA) and require users to authenticate their identity using multiple factors, such as passwords, biometrics, or one-time codes, to access IoT devices or sensitive systems. MFA adds an extra layer of security and helps prevent unauthorized access in case of compromised credentials.</li><li>Enforce the least privilege access controls to limit user permissions and restrict unauthorized access to sensitive data and functionalities.</li><li>Utilize role-based access controls (RBAC) to define and enforce granular access policies based on users’ roles and responsibilities.</li><li>Employ network access controls: implement access control mechanisms, such as virtual LANs (VLANs) and firewall rules, to restrict unauthorized access to IoT networks and devices.</li></ul><h4><strong>3. Risk assessment and vulnerability management</strong>:</h4><ul><li>Conduct regular risk assessments, compliance and security audits to identify potential security vulnerabilities and threats within Healthcare IoT environments.</li><li>Implement vulnerability management programs to prioritize and remediate identified vulnerabilities promptly.</li><li>Employ automated scanning tools and penetration testing to assess the security posture of IoT devices and networks.</li></ul><h4><strong>4. Continuous monitoring and threat detection:</strong></h4><ul><li>Deploy intrusion detection and prevention systems (IDPS) to monitor network traffic and detect anomalous behavior indicative of security breaches or cyberattacks.</li><li>Implement security information and event management (SIEM) solutions to aggregate and analyze log data from IoT devices, network infrastructure, and applications for real-time threat detection and incident response.</li><li>Utilize endpoint detection and response (EDR) solutions to monitor the behavior of IoT devices and identify malicious activities or compromise indicators.</li><li>Implement anomaly detection algorithms and <a href="https://greenm.io/how-to-enhance-customer-satisfaction-with-machine-learning/">machine learning techniques</a> to identify deviations from normal patterns of behavior and detect potential security incidents.</li><li>Deploy intrusion detection systems (IDS) and anomaly detection solutions to monitor IoT device behavior and detect abnormal or suspicious activities indicative of zero-day attacks. These systems can analyze network traffic, system logs, and device telemetry data to identify potential security breaches.</li><li>Deploy email filtering and anti-phishing tools to detect and block malicious emails before they reach users’ inboxes. These solutions can identify phishing attempts based on known patterns, malicious URLs, or sender reputation.</li></ul><h4><strong>5. Incident response and disaster recovery</strong>:</h4><ul><li>Develop and maintain incident response plans and procedures to guide healthcare organizations’ response to security incidents involving IoT devices, such as data breaches, malware infections, or device compromises.</li><li>Establish protocols for containing security incidents, mitigating their impact, and restoring normal operations in a timely manner, minimizing downtime and disruption to patient care.</li><li>Implement network segmentation by dividing IoT devices into separate network segments based on their function or security requirements. Implement network segmentation to contain ransomware infections and prevent lateral movement. This limits the scope of potential attacks and reduces the impact of security breaches.</li><li>Implement robust backup and disaster recovery strategies to ensure the availability and integrity of critical healthcare data in the event of a security breach or system failure involving IoT devices. Maintain offline backups to prevent ransomware attackers from encrypting or tampering with backup files.</li><li>Utilize endpoint protection platforms (EPP) and endpoint detection and response (EDR) solutions to detect and block ransomware attacks targeting IoT devices. These solutions can monitor device behavior, detect suspicious activities, and automatically quarantine infected devices.</li></ul><h4><strong>6. Secure software development practices:</strong></h4><ul><li>Adopt secure coding practices when <a href="https://greenm.io/common-challenges-in-developing-custom-software/">developing custom software</a> for IoT devices and perform rigorous security testing throughout the software development lifecycle to identify and remediate vulnerabilities in IoT device firmware and software applications.</li><li>Implement a <a href="https://greenm.io/sdl-secure-development-lifecycle-and-why-you-should-stick-to-it-in-healthtech-development/">secure development lifecycle</a> and follow industry best practices, such as the OWASP IoT Top 10, to mitigate common security risks and design resilient IoT solutions.</li><li>Establish secure update mechanisms to ensure timely deployment of patches and security updates to IoT devices, addressing known vulnerabilities and reducing the risk of exploitation.</li></ul><h4><strong>7. User awareness and training:</strong></h4><ul><li>Educate users about healthcare IoT security best practices: provide comprehensive security awareness training to IoT device users, emphasizing the importance of adhering to security policies, recognizing phishing attempts, and reporting suspicious activities. Offer training modules, workshops, and simulated phishing exercises to reinforce security awareness.</li><li>Provide comprehensive cybersecurity training and awareness programs for healthcare staff, emphasizing the importance of security practices and policies when using IoT devices and accessing healthcare IT systems.</li><li>Promote secure password practices by encouraging users to create strong, unique passwords for IoT devices and systems and avoid reusing passwords across multiple accounts. Implement password management solutions and by enforcing password complexity requirements to enhance security.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Rp_c73LGgR2KdDL3xWL2w.jpeg" /></figure><h3>Conclusion: healthcare IT solutions are a smart move for securing IoT healthcare devices</h3><p>That’s it. Now you can secure IoT devices to a higher level by implementing IoT device security concepts, such as strong authentication, data encryption, and device hardening.</p><p>This can mitigate vulnerabilities and enhance the resilience of the IoT infrastructure against cyber threats. In case you have any further questions to that end, <strong>do not hesitate to </strong><a href="https://greenm.io/contacts/"><strong>contact GreenM</strong></a>, where I have the honor to work as a content creator and expertise of which includes, inter alia, healthcare. We are here to help you out. See you soon!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e9e8510b3cd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/greenm/7-strategies-for-strengthening-healthcare-iot-security-5e9e8510b3cd">7 strategies for strengthening Healthcare IoT security</a> was originally published in <a href="https://medium.com/greenm">GreenM</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to run isolated tests using Docker Compose on AWS]]></title>
            <link>https://medium.com/greenm/how-to-run-isolated-tests-using-docker-compose-on-aws-8f1228e6eb5f?source=rss----ac7f8be0ecdb---4</link>
            <guid isPermaLink="false">https://medium.com/p/8f1228e6eb5f</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-testing]]></category>
            <dc:creator><![CDATA[Illia Saveliev]]></dc:creator>
            <pubDate>Mon, 01 Mar 2021 11:42:44 GMT</pubDate>
            <atom:updated>2021-03-01T11:42:43.981Z</atom:updated>
            <content:encoded><![CDATA[<p>Nowadays, even the simplest application cannot be imagined without using data. For developers, this means writing complex systems for accessing data storage and information in them. For organizations that want to be data-driven, it is important to skillfully build such data operations and ensure their efficiency and reliability. In the following article, we will talk about how to ensure business logic accuracy by setting up a testing system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gfL2T6fvQ16Egm7yWvf0mQ.jpeg" /></figure><p>Let’s imagine you have the following setup in your project. A project API that communicates with Data API to get data from the database and forwards final results to the web application. We intentionally put Data API between API and database just to make the example a little bit more сomplex.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*6eKudV0yWNlYyr2A" /></figure><p>As you may have guessed, the goal is to cover project API with a set of integration tests that verify business logic and guarantee their stability. Tests should run independently for each new code change. An ideal case is to test the whole data flow from a database to API’s response, including communication via Data API. Of course, all the parts should have their own set of unit tests with mocked dependencies to ensure they work correctly. But the idea here is to check integration between all the components and be sure that the data flow is correct.</p><p>On top of that, we have a set of end-to-end tests that cover the UI part and what data is presented to the user, but we need more prompt feedback in case of broken business logic. Initially, we started with a set of smoke tests that were executed on a live environment and in the end cleaned-up all the test data. This approach worked perfectly to get the general health of the API, but it was hard to cover all required test cases and build a robust comprehensive test suite.</p><p>Moreover, we have to address some important points:</p><ol><li>The main business logic in APIs sits in SQL queries. ISo it is hard to cover them with unit tests, therefore, the ideal solution for this case is a suite of functional integration tests.</li><li>API work with the Vertica database, which is used for analytical solutions. The data there is prepared for reporting and doesn’t have individual row identifiers, like general OLTP databases. Also, it is not suitable for multiple updates or delete operations. Thus, we cannot execute tests on the same database and just rollback all transactions.</li></ol><p>As a result, we came up with an idea to create a completely isolated test environment for our application. Long story short, the ideal candidate for this purpose is Docker Compose, those unfamiliar with it can read the great <a href="https://docs.docker.com/compose/">Overview of Docker Compose</a> from the official docs. The Compose file defines all parts of our environment and fires it on your local machine. Then we can execute a suite of functional tests against it. This approach solves all the aforementioned challenges and allows automated test execution, including embedding it into the CI/CD process.</p><h4><strong>Local Docker Compose file</strong></h4><p>The local Compose file looks pretty straightforward. We define here all our components and settings for them.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ed6ae4446a13692b4094b981d869c782/href">https://medium.com/media/ed6ae4446a13692b4094b981d869c782/href</a></iframe><p>Then you should run the <strong>docker-compose up</strong> command, and the API will be available for testing at <a href="http://localhost:8080/">http://localhost:8080/</a>. Next, you point local tests to this URL and execute them. The magic starts happening; you are running API tests against a real application locally without any connection to your infrastructure.</p><p>It looks simple, but what if you want to automate this test process, and let’s say run tests every night. This is not a problem, we can use <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_CLI.html">AWS ECS CLI</a> for this purpose. AWS has a great <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-cli-tutorial-fargate.html">tutorial</a> on how to run Docker Compose on ECS, but as always, there are some practical nuances and issues that happen in the real world. We will share our experience.</p><h4><strong>Adding a Tests Runner</strong></h4><p>During the previous step, we executed our integration tests locally, from IDE. Now we needed to run our tests on AWS, therefore we introduced a new component and called it <strong>Tests Runner. </strong>It is a simple Docker image where your tests will be executed. In our project, we use Groovy plus Spock to write integration tests, so our Tests Runner image uses Gradle to fire test execution using <strong>gradle test</strong> task. Actually, it is pretty easy to create a test runner for your project, you just need a Docker with required packages and some script that launch tests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/0*IQYy_N4jFAW4bVkh" /></figure><p>As a result, you can include your Docker Tests Runner into the Docker Compose file and deploy it together with other application components.</p><h4>Creating a test database</h4><p>A few notes on working with databases and how to create a test database instance. Overall, the steps are the following:</p><ol><li>Find a public Docker image for the database and create a custom image from it.</li><li>Use shell scripts in the custom image to create a database and initialize its schema. We store database scripts for a schema definition in a separate GitHub repository and create a new image version as soon as a schema is updated, and publish it to our internal Docker registry.</li><li>In the Test Runner we have a task, which waits for the database and starts test execution only when the database is up and running and a schema is created.</li><li>Each test is responsible for its data preparation and cleanup. We have multiple insert scripts that fill database tables with required data.</li></ol><p>Finally, you can reuse this approach to build multiple Docker images for each database type that you have. We use Vertica database, but it will work for all other database types, including MS SQL, PostgreSQL and even Amazon DynamoDB and Elasticsearch.</p><h4><strong>AWS ECS CLI process and files</strong></h4><p><strong>AWS ECS CLI </strong>requires you<strong> </strong>to create two main files: a <strong>Compose file</strong> with a description of your application and an <strong>ECS parameters file </strong>with settings required to run your app on AWS ECS.</p><p>You can reuse your local Compose file and modify it for the ECS version. There are several steps that you should do:</p><ol><li>Add the logging part. It will create corresponding CloudWatch log groups and streams. This will allow you to read logs from application containers.</li><li>Adjust ports. Ports should be the same in the compose file and container for awsvpc network mode. There is no way to overwrite ports as we did in local env, e.g., 8081:80 like we had for Data API. Therefore you should update your container to run API on 8081 and use 8081:8081 mapping.</li><li>Replace service names with the localhost. AWS has a service discovery mechanism, but it is a little bit too complex for our needs, — this way we can go with a simple approach. You can find a detailed overview of service discovery <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-cli-tutorial-servicediscovery.html">here</a>.</li></ol><p>You can examine the final <strong>docker-compose-ecs.yaml</strong> file below. For additional details, read the <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cmd-ecs-cli-compose-parameters.html">official docs</a> about supported Docker Compose syntax.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0de961abbe784b22a70aca13bbcd7a7a/href">https://medium.com/media/0de961abbe784b22a70aca13bbcd7a7a/href</a></iframe><p>The <strong>ecs-params.yaml </strong>defines additional parameters related to the AWS ECS configuration. You can find more details here<strong> </strong><a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cmd-ecs-cli-compose-ecsparams.html"><strong>Using Amazon ECS Parameters — Amazon Elastic Container Service</strong></a><strong>.</strong></p><p>Please note that the following example has placeholders for <strong>TaskExecutionRole</strong>, <strong>Subnet</strong> and <strong>SecurityGroup</strong> parameters. Adjust them to correspond to your infrastructure.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d97a90867b6a50b0e54a0de861d5819f/href">https://medium.com/media/d97a90867b6a50b0e54a0de861d5819f/href</a></iframe><h4><strong>Launch integration tests using AWS ECS tasks</strong></h4><p>Now we have everything to run your tests in AWS ECS, and we will use Fargate for this purpose.</p><p>First of all, you have to <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_CLI_installation.html">install</a> and <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_CLI_Configuration.html">configure</a> ECS CLI. After this, you can work with AWS ECS.</p><p>The first step is to configure the ECS cluster via <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cmd-ecs-cli-configure.html%22%20/t%20%22_blank">ecs-cli configure</a> command, where you define <strong>cluster </strong>name and <strong>config-name</strong> parameters.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0670d758d5e7b9fdaa77fb3d01a2e733/href">https://medium.com/media/0670d758d5e7b9fdaa77fb3d01a2e733/href</a></iframe><p>Then you can start the real AWS ECS test cluster using <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cmd-ecs-cli-up.html%22%20/t%20%22_blank">ecs-cli up</a> command for defined network details like <strong>vpc, subnets </strong>and<strong> security-group.</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6c99684526652d9f9812854334633f40/href">https://medium.com/media/6c99684526652d9f9812854334633f40/href</a></iframe><p>Now the cluster is created and you are ready to launch the integration tests in it. The <a href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cmd-ecs-cli-compose-up.html%22%20/t%20%22_blank">ecs-cli compose up</a> command will create an ECS task with integration tests, as well as all required infrastructure defined in the Compose file and ECS settings file. Just one note here, you should provide the IAM role for the test task. It should be created beforehand and have access to all required resources.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/85d217a2c1c4664bd0edcd8112d7b1be/href">https://medium.com/media/85d217a2c1c4664bd0edcd8112d7b1be/href</a></iframe><p>ECS will then allocate all the needed resources for your services, and all containers from the Compose file will be running simultaneously. In our case, we will execute functional integration tests against a real database instance and two APIs, which is really awesome!</p><p>Finally, you can script all the previous commands and integrate test execution into your CI/CD process to improve your delivery pipeline.</p><h4><strong>This is not the end</strong></h4><p>We hope our example with isolated integration tests illustrated the power of running Docker Compose files on AWS infrastructure. This opens a lot of new possibilities in tasks automation and running complex processes on top of the AWS cloud right from your laptop or CI/CD tool.</p><p>Moreover, Docker and AWS continue their<strong> </strong>partnership to provide native ECS integration to <a href="https://docs.docker.com/engine/context/ecs-integration/">Docker compose</a>, which looks very promising and exciting, and is definitely worth a try.</p><p>If you’re looking for more specific case studies and detailed engineering how-tos, you can find them <a href="https://greenm.io/case-studies/">here</a>. Interested in partnering with a professional team? Check out the GreenM services <a href="https://greenm.io/services/">on our website.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f1228e6eb5f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/greenm/how-to-run-isolated-tests-using-docker-compose-on-aws-8f1228e6eb5f">How to run isolated tests using Docker Compose on AWS</a> was originally published in <a href="https://medium.com/greenm">GreenM</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[HIMSS Europe 2020: Key Takeaways and Virtual Road Notes]]></title>
            <link>https://medium.com/greenm/himss-europe-2020-key-takeaways-and-virtual-road-notes-7f3e2fceec44?source=rss----ac7f8be0ecdb---4</link>
            <guid isPermaLink="false">https://medium.com/p/7f3e2fceec44</guid>
            <category><![CDATA[healthit]]></category>
            <category><![CDATA[himss20]]></category>
            <category><![CDATA[healthtech]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[himss]]></category>
            <dc:creator><![CDATA[Irina Pashkova]]></dc:creator>
            <pubDate>Mon, 01 Feb 2021 15:15:45 GMT</pubDate>
            <atom:updated>2021-02-01T15:15:45.784Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>7–11 September 2020, we immersed ourselves in the cooperative environment of knowledge sharing, expert insights, and innovation in healthcare digitalization at the HIMSS &amp; Health 2.0 European Digital Event.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zni78a-cJM1bYcs6PuiZcg.jpeg" /></figure><p>Domain leaders, entrepreneurs, and investors from about 100 countries gathered on the virtual platform that welcomed 4K of attendees, over 150 speakers including the WHO, the US ONC, and of course the key Partners from Finland (who are supporting this Conference) along with nearly 140 virtual booths at the exhibition floor.</p><p>The event talks, panel discussions, keynotes, and the latest tech solutions showcasing sessions clearly pivoted where European healthcare is going and needs to go.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Kew3RYosgCMeTdt9ta1YA.jpeg" /><figcaption>HIMSS Europe 2020: Exhibit Hall Screenshot</figcaption></figure><p>It was a pure delight to see the formed Ecosystems that include government, public authorities, healthcare professionals, and tech experts.</p><p>The discussions, while having technology at their core, provided an integral vision for each issue highlighting the importance of continuous education, the leadership role of healthcare workers, the necessity of institutional support, and organizational networks.</p><h3>HIMSS Europe 2020 discussions</h3><p>The major topics are listed below, and we’ll make sure to shed some light and reflect on the most interesting aspects during the following weeks:</p><ul><li>AI: Beyond the Buzz</li><li>Building the Digital Maturity and Resilience of European Healthcare Systems</li><li>Building the European Healthcare Data Space</li><li>Critical Care in Critical Settings: Lessons Learned From COVID-19</li><li>Innovation Forum and VentureConnect</li><li>Securing Data, Protecting Privacy, Building Trust for Innovation and Action</li><li>Silver Health and Economy: New Demographics, New Opportunities</li></ul><p>As expected, the 2020 COVID-19 pandemic shaped the agenda and a lot of attention was paid to the lessons learned so far and steps to be taken to make the healthcare sector more resilient and prepared for tomorrow’s challenges. Remote care, self-service, optimization of supply chain and in-hospital activities, eliminating unnecessary healthcare, mitigating risks of medical errors, an individual-tailored approach — all these initiatives empowered by modern technology will continue to play a big role in that. They’ve already reached a good maturity level and will be developed further in the directions of standardization, governance, interoperability, and predictive analytics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*74ciBz-bf7O9qP2a9EMTzg.jpeg" /><figcaption>HIMSS Europe 2020: Session Screenshot</figcaption></figure><p>The attendees were among the brightest international minds. We simply must mention HIMSS Chief Clinical Officer — Charles Alessi who led a number of panel discussions and his vibe and passion energized everyone.</p><p>We are looking forward to continuing the dialog at the face-to-face event in Helsinki in June 2021.</p><h3>HIMSS 2020 virtual experience</h3><p>A few words for the platform which HIMSS prepared for this event in a very limited time window. The organizational committee did their best to ensure the experience was seamless, they prepared all necessary guidance in advance, provided support before and on the event, and were just excellent hosts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2u16Ir8RdhhdFUS6Oupmhg.jpeg" /><figcaption>HIMSS Europe 2020: Lobby Screenshot</figcaption></figure><p>While the experience on the digital conference is definitely different from the real-life event, there are some definite plusses:</p><ul><li>No worries if you missed a session, as it is available on-demand online the next day;</li><li>You can instantly communicate with speakers on the platform, and connect aside as the majority offered their Lin contacts in the session profiles;</li><li>Networking is simplified with the lounge area and the list of active attendees available for more target conversation via private chat;</li><li>Digital Materials that companies have in their booths can be easily stored or shared via the platform functions.</li></ul><p>The platform will be enhanced going forward, and the major suggestions from attendees were relevant to instant video chat capability and keeping the chats history.</p><h3>Conclusion</h3><p>For GreenM, it was the first experience of exhibiting at a virtual event of such a scale. Due to pandemic time, we are forced to live with the new reality of virtual conferences working, networking, and cooperating “from home.” COVID-19 has become a great challenge for society and the healthcare industry. It exposed systemic problems in our approaches to digitization, population health, and digital literacy. At the same time, the coronavirus has given us the chance to streamline the digital revolution here and now, and together — breaking the barriers between countries and societies.</p><p>Would you like to discuss healthtech trends further? Let us know how <a href="https://is.gd/T6XylN">GreenM </a>can help you — our top-notch engineering team would be pleased <a href="https://greenm.io/contacts/">to schedule a free online meeting</a> to devise the best solution with you.</p><p><strong>References:</strong></p><p><em>Key topics of the the HIMSS &amp; Health 2.0 European Digital Event 2020</em></p><p><a href="https://www.himss.org/event-himss-europe-digital">https://www.himss.org/event-himss-europe-digital</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f3e2fceec44" width="1" height="1" alt=""><hr><p><a href="https://medium.com/greenm/himss-europe-2020-key-takeaways-and-virtual-road-notes-7f3e2fceec44">HIMSS Europe 2020: Key Takeaways and Virtual Road Notes</a> was originally published in <a href="https://medium.com/greenm">GreenM</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to migrate your app to Kubernetes?]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/greenm/how-to-migrate-your-app-to-kubernetes-612ce01c6c9?source=rss----ac7f8be0ecdb---4"><img src="https://cdn-images-1.medium.com/max/1024/1*tLDvB00prLT-8hwF1CDq-A.jpeg" width="1024"></a></p><p class="medium-feed-snippet">Everyone wants to have their app running on Kubernetes. But how to start this process? What steps should be performed? How to estimate the&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/greenm/how-to-migrate-your-app-to-kubernetes-612ce01c6c9?source=rss----ac7f8be0ecdb---4">Continue reading on GreenM »</a></p></div>]]></description>
            <link>https://medium.com/greenm/how-to-migrate-your-app-to-kubernetes-612ce01c6c9?source=rss----ac7f8be0ecdb---4</link>
            <guid isPermaLink="false">https://medium.com/p/612ce01c6c9</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Illia Saveliev]]></dc:creator>
            <pubDate>Mon, 15 Jun 2020 15:40:31 GMT</pubDate>
            <atom:updated>2020-06-15T15:40:31.541Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Secure Service to Service Communication in AWS]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/greenm/secure-service-to-service-communication-in-aws-914f335b01ed?source=rss----ac7f8be0ecdb---4"><img src="https://cdn-images-1.medium.com/max/2250/1*1sRE4QKVSFnlrsf-5ue-mw.jpeg" width="2250"></a></p><p class="medium-feed-snippet">Everyone knows how to secure user access to resources. It&#x2019;s simple, as we have a person that can enter his login and password into the&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/greenm/secure-service-to-service-communication-in-aws-914f335b01ed?source=rss----ac7f8be0ecdb---4">Continue reading on GreenM »</a></p></div>]]></description>
            <link>https://medium.com/greenm/secure-service-to-service-communication-in-aws-914f335b01ed?source=rss----ac7f8be0ecdb---4</link>
            <guid isPermaLink="false">https://medium.com/p/914f335b01ed</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[api]]></category>
            <dc:creator><![CDATA[Illia Saveliev]]></dc:creator>
            <pubDate>Fri, 10 Apr 2020 12:37:10 GMT</pubDate>
            <atom:updated>2020-04-13T10:41:48.980Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Data Warehouse Testing]]></title>
            <link>https://medium.com/greenm/data-warehouse-testing-3c0fb955da1d?source=rss----ac7f8be0ecdb---4</link>
            <guid isPermaLink="false">https://medium.com/p/3c0fb955da1d</guid>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <dc:creator><![CDATA[Irina Pashkova]]></dc:creator>
            <pubDate>Mon, 27 Jan 2020 11:14:35 GMT</pubDate>
            <atom:updated>2020-03-23T16:30:59.705Z</atom:updated>
            <content:encoded><![CDATA[<h4>In the <a href="https://medium.com/greenm/data-pipeline-installation-quality-935fe02d7621?source=collection_home---4------0-----------------------">first part of the story</a> shared with the community on QA Data Monsters Meetup (#DQM) we discussed how to check data pipeline work when data storage requirements remain the same, while technologies standing behind it evolve. We covered ways to ensure the new Pipeline version produces the right data, works fast, is reliable, and easy to use.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mqHwHAQSRyzbsvAnLkcgEA.jpeg" /></figure><p>It’s rare luck when you are given a chance to concentrate fully and solely on improvements. As you may know, often when you change the technological stack, some new functionality is always needed as well.</p><p>We had to add new data module processing to Pipeline to support functionality that was not yet in production.</p><p>This task means we have new data sources and new business transformation rules, and need to update the Data Mart structure to receive the resulting data. It required changes, which means testing of all Pipeline components… including data sources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BNm5Hcs1OH6_hUm1zSpPwg.jpeg" /></figure><h3>Test Data Source</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*kr33_M--H6gX-cTG2szKgA.jpeg" /></figure><p>In this area, risks are often underestimated:</p><ul><li>some specialists consider it a “foreign zone of responsibility”;</li><li>they take it as an axiom that the data in the source has already passed a set of checks and cannot be erroneous;</li><li>or it often happens that there is no real data (e.g. the functionality saving the data to the source is not ready or an “upstream” ETL is not built), and the testers need to recreate the test data from scratch.</li></ul><p>So, how did we cope and find all gaps between the described requirements and expectations?</p><p>1) These are questions for an Expert — how the data gets into the system and whether it’s possible to modify it over time, and just how exactly. Sometimes during such conversations, rethinking comes to our minds, and the source design or ETL requirements change.</p><p>2) Search for any tech possibility of non-compliance with the data source and the requirements.</p><p>Most of the problems that we met were caused by manual edits of data sources, happened by-passing the standard data flow. And due to the wrong design of the tables, the “human factor” appeared, and there was data duplication or a breakdown in the relationships between them.</p><p>Why bother spending time and double-checking? Because a user will see all defects through the BI application and a bug will be escalated to you, detracting from other important things, and only then you will redirect it to guys responsible for the source data. Earlier issue detection makes the feedback loop much shorter.</p><p>Therefore, if we found an issue but did not have the opportunity to change the source, we documented the risks and introduced them to Product Owner and Production Support Group, and included Exceptions processing into the ETL logic (for example, if the data was accidentally duplicated due to the lack of the correct key, we can use the record that was added later).</p><h3>Test Target Data Model</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*Jd38UFR5lMWZTbX_VTJR3Q.jpeg" /></figure><p><strong>During the checking we:</strong></p><p>1) verified that “common decency rules” are followed and the same entities and attributes are called identically in different tables (if even one letter is different, you will hate it when writing queries, and if it makes it to production, it will be difficult and expensive to change it later on).</p><p>2) verified that the new structure was optimal for building reports and graphs, i.e. possible preaggregation, denormalization, etc. were introduced. For example, the metric is set up as a field in the database, rather than at the visualization level for faster chart loading, or, if necessary, the data in a table is multiplied by parameters for faster filtration.</p><p>3) check, where applicable, if the table contains fields that facilitate verification of complex calculations (you can always ask to add those) as well and logging fields (the date when the record was saved).</p><h3>Functional ETL Testing</h3><h4><strong>Data Quality tests</strong></h4><p>We started checking the accuracy of the data processing with quick and straightforward tests localized to the Target Data Mart.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*mi4WiN0gnpw3wbhT4YQbEA.jpeg" /></figure><p>At first, we wrote a Smoke suite, and next we created Data Quality tests that check Target Data Mart for compliance with the requirements, including all sorts of implicitly designated logical restrictions.</p><p>For automation, we used the Fitnesse framework and one of the <a href="http://dbfit.github.io/dbfit/">DBLib </a>features — “Query”. The parameter of which is a query and an expected result can be set up as 0 records.</p><p>For example, for the initial check of the patient age with the absence of the date of birth field, you can see that there are no data with zeros, negative values, and there are no people with the age of more than 125 years.</p><p>Such simple tests can identify many ETL process errors without the need to compare data from the source with the target Data Mart.</p><h4>Balancing Tests</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*LmfOouWakKboflFqfoMT0Q.png" /></figure><p>Finally, we wrote Balancing Tests that simulate the ETL process to see that all data from the sources was correctly reflected in Data Mart, and there is nothing extra in it.</p><p>These tests are implemented using the so-called Minus queries, where the results of the two subqueries are compared.</p><p>The first is a request to data sources that mimics business requirements, and the second is a request to Data Mart.</p><p>To write such tests, you need to clearly understand how the data in the source and Data Mart are related (1: M, N: 1, M: N), how the entities correspond to each other, what types of transformations are applied to the data and whether all the data in Data Mart is updated or not.</p><p>Requests that we wrote must be checked for their ability to detect errors. You can do it through intentionally corrupting data in Data Mart (e.g. add or delete records, etc.) and check that a mistake has been found. Actually, this rule applies to all tests — checks must be checked.</p><p>Finally, each Balancing test must be bilateral. You check that all the right data from source made it to the target and that all data in the target is the right one.</p><p><strong>Balancing Tests challenges</strong></p><p>It may be difficult to create such tests because all the data is on different servers in different systems, and it’s impossible to access them in one request.</p><p>It was no longer the case for us, as we gathered it all in one place — Data Lake — data from all sources as well as the results of the transformation. These are files in Amazon S3 that can be used for tables creation via Amazon Athena, an interactive query service, or data catalog set up via Amazon Glue. Thus we can reach them in one request and write Minus queries. And of course, keep test suites in FitNesse for convenience.</p><h4>Most Common bugs</h4><ul><li>Count Mismatch (including Duplicates);</li><li>Anomalies issues: Null or Length relevant;</li><li>Date relevant calculations.</li></ul><p>Which bugs found by Minus queries are most common?</p><p>Often data is lost, or there is erroneous extra one in the target, e.g. duplicates due to improperly applied conditions of joins or filtering.</p><p>NULL values are not processed correctly. For example, they are replaced by an empty string or are not taken into account in the transformation condition.</p><p>We also should be attentive regarding the dates, especially when it comes to various Time Zones (whether unification is necessary or not), and non-standard fiscal periods (if the formula uses the current date, you need to check whether this formula will work correctly in the first and last month of the year).</p><h4>ETL Testing Challenges</h4><p><strong>What other difficulties may arise?</strong></p><p>Even the simplest queries in Athena can “freeze” when Amazon did not have enough resources (for such requests, resources are allocated according to the residual principle).</p><p>Some of the transformations were quite complex, and the creation of an exhaustive test set is equal to ETL creation.</p><p>Also, sometimes, it’s impossible to verify the entire data set due to the large amount of it.</p><p>To solve this, we needed to apply the methods mentioned earlier:</p><p>1) Start with the easiest tests and then move on to the heavy ones. At first, we compare the number of records, the minimum, maximum, average values in the field, checksums, and finally, the content.</p><p>2) Limit the data. For example, look at one period. In other words, we take data for a year or data of just one client.</p><p>3) Simulate a test set.</p><h4>Visualization in Data QA</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cF770uCrC2KtrvpNOCt-BA.jpeg" /></figure><p>You can use visualization with the products that are already used on the project to analyze and model test data, and draw up the correct set of checks — for example, Tableau.</p><p>Also, you can compare the results of two different ETLs if they work with the same data (we had Elastic Kibana).</p><p>The last product we used was Apache Zeppeling, which helped to create a cluster for running tests, thereby solving the Athena resource limitation problem and allowed us to use standard visualizations.</p><h4>Ongoing Support</h4><p>The task of monitoring the Pipeline operation in production was transferred to Production Support Group. The testers made sure that alerts are set up for critical events and processes to respond to the Pipeline failure or delay.</p><p>In addition, Pipeline was incorporated into a Data Integrity project, which collects checksums and uses New Relic for log analysis.</p><h3>Finish</h3><p>ETL testing, like other types, is the one where you need to check the result of the system against the setup requirements.</p><p>There is no graphical interface with intuitive functions, and the tester should be well versed in data and be able to write queries.</p><p>However, if you know the right approaches and have a basic understanding of Amazon services, you can choose the right strategy.</p><p><em>And in conclusion.</em></p><h3>What is the difference between ETL testing and data warehouse testing?</h3><p>Pipeline quality assurance practices described above and those listed in the first article (<a href="https://medium.com/greenm/data-pipeline-installation-quality-935fe02d7621?source=collection_home---4------0-----------------------">Data Pipeline Installation Quality</a>) are of course still just a part of data warehouse testing, which goes beyond that.</p><p>When you build a system from scratch, as you also need to verify a data management product by doing performance, stress and recovery testing for it and make sure that front-end applications marry it nicely and of course address security.</p><p>If you have any questions, topic suggestions, or want to discuss the Data Warehouse testing in your organization, <a href="https://greenm.io/">contact GreenM here</a>.</p><p><em>This article was originally published on </em><a href="https://greenm.io/data-warehouse-testing/"><em>GreenM</em></a><em>.</em></p><p>More articles:</p><p><a href="https://medium.com/greenm/how-to-rapidly-increase-the-value-of-you-business-using-data-you-already-have-1a4534b4bc0d?source=collection_home---4------0-----------------------">A Quick Way to Refine Raw Data Into Dollars A Quick Way to Refine Raw Data Into Dollars</a></p><p><a href="https://medium.com/greenm/how-to-test-a-serverless-api-using-aws-serverless-gradle-test-runner-b2c806fba04b">How to test a serverless API using AWS serverless Gradle test runner</a></p><p><a href="https://medium.com/greenm/deploy-evolution-serverless-feedback-platform-a76be810f72">Deploy Evolution: AWS Serverless Feedback Platform</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c0fb955da1d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/greenm/data-warehouse-testing-3c0fb955da1d">Data Warehouse Testing</a> was originally published in <a href="https://medium.com/greenm">GreenM</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS IAM: Permission Management is as Simple as Donuts. Concepts and Policies]]></title>
            <link>https://medium.com/greenm/aws-iam-permission-management-is-as-simple-as-donuts-concepts-and-policies-58924504a15c?source=rss----ac7f8be0ecdb---4</link>
            <guid isPermaLink="false">https://medium.com/p/58924504a15c</guid>
            <category><![CDATA[aws-iam]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[policy-management]]></category>
            <category><![CDATA[permission]]></category>
            <dc:creator><![CDATA[Anna Korkoshko]]></dc:creator>
            <pubDate>Thu, 09 Jan 2020 15:16:55 GMT</pubDate>
            <atom:updated>2020-01-09T15:16:55.669Z</atom:updated>
            <content:encoded><![CDATA[<h4>Every project journey with AWS starts from resource creation. Sure, what you want to do is dive deep into exciting serverless, powerful high-availability or even build your brand new rocketship right away. But first, you will definitely come face to face with AWS Identity and Access Management Service.</h4><p>AWS IAM is tightly coupled with every resource creation and should be provisioned wisely to avoid errors or security issues. In this series of articles, we will start from the basics and will gradually move on to subsequent parts.</p><p>This article describes basic IAM concepts and IAM policy building components flavored with tasty donuts examples. So go on, maybe stop by the bakery first, and enjoy!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EjJDsZLXeYzWpgg_CAsVtg.jpeg" /></figure><h3>Understanding IAM Components</h3><p>First of all, let’s imagine you’re a policeman. So you come to the police station and see a bunch of donuts lying on the dinner table. Can you eat them? Are they shared? Or somebody bought them just for you? Or they might be your chief’s donuts and you’d better not even look or sniff in that direction. How do you know?</p><p>Right, there should be a donuts policy somewhere. It might be in the Code of Conduct or just in your head. But everyone should know if they are allowed to eat a donut or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*vwzFXDYSefoZQas1dSlBVA.png" /></figure><p>So what are AWS IAM components from the donut perspective?</p><p>An <strong>AWS IAM resource</strong> is a donut you may eat. Or any other thing you can interact with in the office.</p><p>An <strong>AWS IAM policy</strong> is a document, which declares some permission statements. In a minute, we will see what it can declare and how.</p><p>AWS Identities:</p><p>An <strong>AWS IAM user</strong> is just a person like you, your colleague or your boss.</p><p>An <strong>AWS IAM group</strong> is a collection of IAM users united on a permission basis. There can be such groups in a police department as police officers, detectives, crime scene investigators. Or: Majors, Captains, Lieutenants. One IAM user can be a part of several groups.</p><p>But here is another identity type — <strong>AWS IAM Role</strong>. I need a bit more of your imagination at this point.</p><p>Let’s assume that you own a donut bakery-shop aside from your fascinating policeman’s job.</p><p>There are three roles in your donut shop: a cleaner, chef, and cashier. Also, there is a policeman’s role. <strong>To assume the role</strong> is to act according to the permissions and restrictions this role entails. So let’s become a chef by assuming the “chef role.” Only a chef can go to the kitchen and create donuts, but they can not sell donuts and talk to customers.</p><p>They can’t arrest people as well! So to become a policeman again we need to leave the “chef role” and <strong>assume</strong> “policeman role.” Wow! Now you have a badge again! Moreover, you can eat some donuts! But you’d still better check those donut policies to know for sure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*kX0gDZ1W3Jp8mU6yvIzrrw.png" /><figcaption>So <strong>to assume a role,</strong> is like wearing a magic chef’s hat and becoming a chef for a while with all the chef’s privileges and prohibitions.</figcaption></figure><h3>AWS IAM Policy Syntax</h3><p>So let’s move to the most exciting part — AWS Policies. We already know that there is some document (just a JSON actually) that regulates permission. You can create json policy by yourself or use a visual editor in the AWS Management Console. So let’s look closer.</p><p>A policy consists of statements, there can be one or more statements in one policy.</p><p>Here is a simple S3 bucket policy example:</p><pre>{<br>{<br>  &quot;Version&quot;:&quot;2012-10-17&quot;,<br>  &quot;Statement&quot;:[<br>   {<br>    &quot;Effect&quot;:&quot;Allow&quot;,<br>    &quot;Principal&quot;:{<br>     &quot;AWS&quot;:&quot;arn:aws:iam::111122223333:root&quot;<br>    },<br>    &quot;Action&quot;:[<br>     &quot;s3:PutObject&quot;,<br>     &quot;s3:PutObjectAcl&quot;<br>    ],<br>    &quot;Resource&quot;:[<br>     &quot;arn:aws:s3:::examplebucket/*&quot;<br>    ],<br>    &quot;Condition&quot;:{<br>     &quot;IpAddress&quot;:{<br>      &quot;aws:SourceIp&quot;:&quot;54.240.143.0/24&quot;<br>     }<br>    }<br>   }<br>  ]<br> }<br>}</pre><p>Please notice parts starting with “ARN”, which stands for Amazon Resource Name and means a unique identifier for any resources you create in AWS.</p><h3>Let’s discuss statement components</h3><p><strong>Effect.</strong> You can have two different intentions when you are building your policy: to grant or restrict access to actions/resources. So you should specify <strong>Effect </strong>— “Allow” or “Deny.”</p><p><strong>Principal (NotPrincipal).</strong> The entity that is allowed or denied access. E.g. user, role, AWS service. Principals are only required in resource policies (We are going to cover policy types in the next article)</p><p><strong>Action (NotAction).</strong> Here you declare the type of access that you allow or deny. It consists of two parts: AWS Service and action available for this service.</p><p>Assume that we have a “food” service in the office. It allows buying and eating some food. It allows changing the food provider as well.</p><p>So the office manager will be allowed to</p><pre>“Action”:[”food:buy”, “food:changeProvider”]</pre><p>Then employees would have permissions to</p><pre>“Action”:[“food:eat”]</pre><p>Please note the difference between</p><pre>“Effect”:”Deny” <br>“Action”: [“food:eat”]</pre><p>and</p><pre>“Effect”:”Allow”<br>“NotAction”: [“food:eat”]</pre><p>The first example explicitly denies the action ‘eat’ from the food service. However, the second example ALLOWs all the actions from the service ‘food’ except action ‘eat.’</p><p><strong>Resource.</strong> The Amazon resource(s) which is a target for an action.</p><p>Actually, the resource here is our tasty donut. Now we can combine the resource and the action:</p><pre>“Action”:[“food:eat”],<br>“Resource”:”pinkDonut”</pre><p>This simply means: it is allowed to eat a pinkDonut using food service.</p><p>Note, that actions are valid only for specific types of resources. You can’t use action “food:eat” with a “car” resource, but probably there is a service somewhere to drive it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*wqLEDaFN1lmIXyMjit23Hw.png" /></figure><p><strong>Condition.</strong> The conditions under the defined access is valid.</p><p>There are a lot of conditions available for policies. You will need them when you use “but only” phrase when creating a policy. There are some life examples:</p><p>I want to allow my employees to eat pink donuts, <strong>but only</strong> if they are “lieutenants”. (Only they deserve)</p><p>I want to allow my employees to eat pink donuts, <strong>but only</strong> till 2pm. (Seems likely you care about their health)</p><p>I want to allow my employees to eat pink donuts, <strong>but only</strong> if they finished their daily reports. (Encourage fast reports)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*UG7w-VIUkoBty8Dx1UTw0w.png" /></figure><h3>Wildcards</h3><p>You can use wildcards for some elements of the policy. Let’s look at some examples:</p><pre>&quot;Resource&quot;: &quot;arn:aws:s3:::*&quot;</pre><p>This means the policy will be applied to all resources in s3.</p><pre>&quot;Resource&quot;: &quot;arn:aws:s3:::corporate*&quot;</pre><p>This policy will be applied to all buckets starting with “corporate.”</p><pre>&quot;Action&quot;: &quot;s3:*&quot;</pre><p>This policy will provide access to all actions from s3 service.</p><pre>&quot;Action&quot;: &quot;ec2:*KeyPair*&quot;</pre><p>This policy will provide access to all actions available with KeyPairs: DescribeKeyPairs, CreateKeyPair, DeleteKeyPair, ImportKeyPair.</p><p>Please note that you should be extremely careful with wildcards and think twice before you use them. Being specific in policies is a best practice that should be followed to build a secure environment.</p><p>Do you really want to give permissions to terminate instances to QA engineers when writing “Action”: “ec2:*”? Do you really want to give anybody access to all s3 resources by writing “Resource”: “arn:aws:s3:::*”? Despite one of the buckets contains some sensitive data? Currently, the ec2 service has more than 350 actions. Are you sure that you know all of them and want to give your developers access to all the actions?</p><p>And there is an even more important point. When you use a wildcard, you give access not only to existing actions/resources but also to those which will be added <strong>in the future</strong>.</p><p>So “wildcards” is a very useful feature but please use it responsibly.</p><h3>A bit of real-world practice</h3><p>Now, we are going to get our hands dirty and build a simple real-world policy example. Assume, we have an environment with EC2 Instances and S3 buckets. Our task is to build a policy for developers. They should be able to see all instances, stop and start instances with a special dev tag (from the office only) and have access to S3 buckets with dev resources.</p><p>First, let’s allow users to “Describe Instances” using ec2 service, which is really simple:</p><pre>{<br> &quot;Effect&quot;:&quot;Allow&quot;,<br> &quot;Action&quot;:&quot;ec2:DescribeInstances&quot;,<br> &quot;Resource&quot;:&quot;*&quot;<br>}</pre><p>Next, we will need to allow starting and stopping instances, but only if the user accesses AWS from the office IP. Rember? “But only” phrase means using Conditions. Additionally, we are going to use Condition for resource tag to allow interaction only with dev instances.</p><pre>{<br> &quot;Effect&quot;:&quot;Allow&quot;,<br> &quot;Action&quot;:[<br>  &quot;ec2:StartInstances&quot;,<br>  &quot;ec2:StopInstances&quot;<br> ],<br> &quot;Resource&quot;:&quot;*&quot;,<br> &quot;Condition&quot;:{<br>  &quot;StringEquals&quot;:{<br>   &quot;ec2:ResourceTag/env&quot;:&quot;dev&quot;<br>  },<br>  &quot;IpAddress&quot;:{<br>   &quot;aws:SourceIp&quot;:&quot;210.75.12.75&quot;<br>  }<br> }<br>}</pre><p>And finally, we want to allow users to list the content of the dev buckets; PutObject and Get Object from them. So we will use actions from the s3 service.</p><p>Let’s assume that all our dev buckets’ names start with “devdata” prefix. This is a good place to make good use of wildcards.</p><pre>{<br> &quot;Effect&quot;:&quot;Allow&quot;,<br> &quot;Action&quot;:&quot;s3:ListBucket&quot;,<br> &quot;Resource&quot;:&quot;arn:aws:s3:::devdata*&quot;<br>},<br>{<br> &quot;Effect&quot;:&quot;Allow&quot;,<br> &quot;Action&quot;:[<br>  &quot;s3:PutObject&quot;,<br>  &quot;s3:GetObject&quot;<br> ],<br> &quot;Resource&quot;:&quot;arn:aws:s3:::devdata*/*&quot;<br>}</pre><p>That’s it. Here you can see the full policy content. Easy, right?</p><pre>{<br> &quot;Version&quot;:&quot;2012-10-17&quot;,<br> &quot;Statement&quot;:[<br>  {<br>   &quot;Effect&quot;:&quot;Allow&quot;,<br>   &quot;Action&quot;:&quot;ec2:DescribeInstances&quot;,<br>   &quot;Resource&quot;:&quot;*&quot;<br>  },<br>  {<br>   &quot;Effect&quot;:&quot;Allow&quot;,<br>   &quot;Action&quot;:[<br>    &quot;ec2:StartInstances&quot;,<br>    &quot;ec2:StopInstances&quot;<br>   ],<br>   &quot;Resource&quot;:&quot;*&quot;,<br>   &quot;Condition&quot;:{<br>    &quot;StringEquals&quot;:{<br>     &quot;ec2:ResourceTag/env&quot;:&quot;dev&quot;<br>    },<br>    &quot;IpAddress&quot;:{<br>     &quot;aws:SourceIp&quot;:&quot;210.75.12.75/16&quot;<br>    }<br>   }<br>  },<br>  {<br>   &quot;Effect&quot;:&quot;Allow&quot;,<br>   &quot;Action&quot;:&quot;s3:ListBucket&quot;,<br>   &quot;Resource&quot;:&quot;arn:aws:s3:::devdata*&quot;<br>  },<br>  {<br>   &quot;Effect&quot;:&quot;Allow&quot;,<br>   &quot;Action&quot;:[<br>    &quot;s3:PutObject&quot;,<br>    &quot;s3:GetObject&quot;<br>   ],<br>   &quot;Resource&quot;:&quot;arn:aws:s3:::devdata*/*&quot;<br>  }<br> ]<br>}</pre><h3>Conclusion</h3><p>As you can see, the AWS IAM basic components are pretty straightforward and intuitive, yet powerful and flexible. You can use hundreds of actions of more than a hundred services with numerous resources.</p><p>Fine-grained access management will help you to follow all the security best practices and achieve a convenient environment for your project teams. You can always find more details on policy creation in the <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements.html">IAM User Guide</a>.</p><p>In the subsequent article, we will cover different policy types and how they interact. In addition, we will look into the policy evaluation process, obviously from the donuts perspective.</p><p><em>This article was originally published on </em><a href="https://greenm.io/contacts/"><em>GreenM</em></a><em>.</em></p><p>Related Articles:</p><p><a href="https://medium.com/greenm/how-to-test-a-serverless-api-using-aws-serverless-gradle-test-runner-b2c806fba04b?source=collection_home---4------0-----------------------">How to test a serverless API using AWS serverless Gradle test runner</a></p><p><a href="https://medium.com/greenm/aws-lambda-or-aws-fargate-the-step-by-step-guide-to-choosing-the-right-technology-925ebcf89b7c?source=collection_home---4------0-----------------------">AWS Lambda or AWS Fargate: The Step-by-Step Guide to Choosing the Right Technology</a></p><p><a href="https://medium.com/greenm/top-4-aws-patterns-of-highly-available-api-d34599bfbb96">Top 4 AWS Patterns of Highly Available API</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58924504a15c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/greenm/aws-iam-permission-management-is-as-simple-as-donuts-concepts-and-policies-58924504a15c">AWS IAM: Permission Management is as Simple as Donuts. Concepts and Policies</a> was originally published in <a href="https://medium.com/greenm">GreenM</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Pipeline Installation Quality]]></title>
            <link>https://medium.com/greenm/data-pipeline-installation-quality-935fe02d7621?source=rss----ac7f8be0ecdb---4</link>
            <guid isPermaLink="false">https://medium.com/p/935fe02d7621</guid>
            <category><![CDATA[etl-testing]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[testing]]></category>
            <dc:creator><![CDATA[Irina Pashkova]]></dc:creator>
            <pubDate>Fri, 20 Dec 2019 12:07:30 GMT</pubDate>
            <atom:updated>2019-12-20T12:07:29.911Z</atom:updated>
            <content:encoded><![CDATA[<h3>AWS Data Pipeline Evolution testing</h3><h4>You asked for it. We delivered! Here’s the testing story shared with the community on QA Data Monsters Meetup (#DQM) of how we verified a new version of Data Import upgraded in the Amazon cloud using its services.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vURmGwqzXdZROmKLGOVcag.jpeg" /></figure><p>The testing approaches and types discussed are universal and can be applied to any project to safeguard data update quality.</p><p>Grab your favorite beverage. Kick back. Enjoy. Write to us, should you have any comments and questions. We’ll be super glad to be of service!</p><h3>Evolution Testing Task</h3><p>A few words about our great client and partner. With the help of our services, one of the oldest USA Healthcare Research corporations does processing and analysis of patient feedback for American clinics, to see whether people are satisfied with medical services, to address any concerns, and thus increase patient loyalty.</p><p>The project is a web application with a set of dashboards and reports based on a relatively small Data Mart, containing data from 300 clinics that are fully refreshed daily by data imports from various operational data sources.</p><p>The story began when the data was growing rapidly and the sales forecast said it would skyrocket soon. By that time, we already struggled to fit the import into the timeframe from a moment when the new data was available to the expected site update at 8 a.m. In addition, we received new business requirements for synchronous updating of all application modules.</p><p>With this in mind, we decided to optimize the import process by transferring it to Amazon services, since the project has already “lived” in the cloud.</p><p>So, summarize.<strong> Project milestones:</strong></p><p>🚩Daily Runs<br>🚩Full Refresh Mode<br>🚩300 Customers<br>🚩~ 500 Mln rec / tab<br>🚩~ 5h ETL time</p><h3>But wait. How Did We Deal?</h3><p><strong>Some Tech Details</strong></p><p>A Data import passes three well-known steps — Extract, Transformation and Load (ETL). On the scheme below you can see the project’s ETL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MHbskNHa6mGXdwnNdUeTwA.jpeg" /></figure><p>In the Amazon cloud, the new process was set up in a way that for extract and transformation we used temporary EMR clusters, and to upload results to data mart Batch service was used.</p><p>The results of each step were saved as files in the S3 catalog to be used on the following step. In this way, we collected data from different sources in one storage — Data Lake.</p><p>It allowed us to build the Pipeline with a flexible architecture, sequential and parallel steps, and introduce dependencies. The code and configuration of the Pipeline are also stored as files in S3.</p><p>Several Lambda services were set up to monitor the current state of the Pipeline and control the start of a desired stage. (You can check out more on that Pipeline architecture in some of our <a href="https://greenm.io/deploy-evolution-aws-serverless-feedback-platform/">other posts</a>, and stay tuned for the upcoming, cause the Pipeline got optimized with the new AWS features!)</p><p>Since the technology and architecture of the ETL process have changed, we needed to verify that the new Pipeline gives a result with the same data quality as then current production ETL version, and be sure that it works faster, and is more convenient and reliable to use.</p><h3>Hold on. What do we have?</h3><p>The data sources didn’t change; the business transformation rules, and the Data Mart schema remains the same. On the other hand, since the whole “filling” was replaced, an error can occur everywhere, so we need to check all the contents of all tables. However, if you do it by comparing data items in Data Mart against the source, it would take a long time.</p><p>That’s why we used ETL Regression Testing, a comparison of two Data Marts: a tested one — created through the new Pipeline and the Reference one- created through the previously thoroughly tested Production ETL version.</p><h3>Regression via Reference Data Schema</h3><p>We imported data into both Data Marts from the same set of sources and then compared these schemas through the table-by-table SQL queries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yQyUso7Nkg_JSYg_2WF9Ng.png" /></figure><p>The essential points you should pay attention to include:</p><ul><li>Before each launch, the tested Data Mart needs to be entirely cleared.</li><li>It’s necessary to exclude in advance any fields that may differ (log fields with the time of adding or modifying a record, or fields for new functionality).</li><li>At first, you have to run the smoke test and check the tables for data and duplicates, next for the number of records, and only then can you compare the entire contents.</li></ul><h4>FitNesse For ETL Regression</h4><p>We use Fitness Web server to automate ETL testing, where each test is an HTML page with an evident structure. You can use a readymade set of checks, the so-called fixtures, or write your own.</p><p>For Regression testing, the developers helped us with the creation of fixtures. And testers for each table created a config file with a table name, a unique record definition, and a list of fields that should be excluded from the comparison — as simple as that.</p><p>In a test, we indicate the following parameters: which engine for checks must be used, and where to connect, determine the test and reference Data Mart schemas and send just one parameter to the fixture — the config file of a table. After a test run, we get a precise result.</p><h4>Regression Challenges</h4><p>We faced three expected issues during the regression testing:</p><p>1) To test a change or fix we had to wait for a while, as ETL took a long time to run;</p><p>2) The tests hanged for large tables;</p><p>3) Data on the Test environment was incomplete. Since we work on a project where the data is protected by law, we didn’t have some parts of source data.</p><h4>What Did We Do?</h4><p><strong>Manual Inspections</strong></p><p>We didn’t wait until the Pipeline run is finished; we began to inspect its configuration and data processing requests code.</p><p>It was very convenient to do thanks to the chosen architecture — everything is very readable, structured, and stored as files in S3 that are easy to download.</p><p>Inspections help to identify errors at the earliest stages, even before the start of import. For example, some changes are made in many tables via copy-paste, or sometimes developers forget to apply them to others.</p><p><strong>Partial Pipeline Run</strong></p><p>Also, we can use the partial Pipeline run to save time because most errors occur at the transformation step. Therefore, we used the output of Extract step from the previous test, and run the pipeline, not from the very beginning. Additionally, we were able to disable synchronization so that different modules do not wait for each other every time.</p><p><strong>Source Data Limitation</strong></p><p>You can limit the amount of imported data by modifying the Pipeline configuration or by physically changing the data sources. Some Test Source Data could be physically deleted, but due to the dependencies, it’s quite a challenging thing to do. Instead, you can find a rule in business logic that filters data, for instance some flag, and change the value of this flag in the source database to limit the imported data volume.</p><h4>Test Data Modeling</h4><p>Also, to solve the issue of missing data, you need to carefully model it by analyzing the possible options. It’s better to do it by using a script that can be reused again.</p><h3>Non-functional Pipeline Testing</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ceU0os6Wf2AGSe6BEE7lNg.png" /></figure><p>So, we checked that the Pipeline is functioning as required, and provides the desired quality of data.</p><p>As for non-functional requirements, we collected statistics on the Pipeline run time during the regression, thereby confirming that its speed is sufficient. Next, a <strong>Performance</strong> check was conducted.</p><p>Also, during the inspections of the Pipeline code during Regression, we made sure the passwords for connecting to the databases are not stored in the configuration files, but rather set up in Amazon’s Secret Manager service, that access to s3 folders and Data Lake is limited — and those were <strong>Security</strong> checks.</p><p>Among other non-functional requirements, let’s take a closer look on <strong>Usability</strong> and <strong>Reliability</strong>. At the end of the day, those two define whether the Support Team, who will manage Pipeline in production, will fall in love with it or not.</p><p>All the rest of the non-functional types of testing are also important, but they require a long separate discussion. In particular, the topic of Pipeline <strong>Scalability</strong> was covered before (You are welcome to read our previous post <a href="https://medium.com/greenm/scalable-data-pipeline-f5d3c8f7a6d9?source=collection_home---4------1-----------------------">Scalable Data Pipeline. Be Ready for Big Changes</a>).</p><h4>Usability Testing</h4><p>To test the usability of ETL, you can partially apply the same heuristics as for app interfaces testing — user awareness of system status and errors, system flexibility, ease of use of various configurations, and the availability of clear documentation.</p><p>During the Regression testing, when we worked with the Pipeline settings, we ran through all expected usage scenarios, and it was easy for us to evaluate the usability and outline the further development tasks to facilitate some operations (for example, to introduce auto-restart for some stages of processing).</p><h4>Reliability Testing</h4><p>We conducted a <strong>Risk assessment</strong> — created a list of possible issues to check the reliability and <strong>simulated the failures</strong> to make sure that Pipeline stops working as expected, i.e., incomplete new data doesn’t get into production, data is not erased, etc.</p><p>Such experiments are very convenient to do in Amazon cloud using its services — the tester can make some data sources or Data Marts unavailable by turning off a hosting server instance, stop the cluster during data processing, specify a nonexistent table in configuration, create a pile of logs in the repository to imitate a rather long history of usage, etc.</p><p>However, there are some pitfalls to keep in mind while playing.</p><p>Reliability Testing Faults</p><p>1) At first, we did not pay enough attention to checking the Amazon service logs during the pipeline testing. In particular, we underestimated the severity of the exceptions generated a couple of times by service responsible for a pipeline step start. As a result, it led to some delays between different stages.</p><p>2) Secondly, we overestimated the reliability of AWS.</p><p>The scenario of a data processing failure due to a lack of Amazon resources was defined as one with very low probability. Thus, the development of functionality responsible for a Pipeline step auto-restart after alike error was not included in the scope of the initial version.</p><p>Unexpectedly, when the pipeline has been in production for quite a long time, we noticed the error telling that we don’t have enough resources. We wrote to Amazon support and received the answer that the problem is on their end and would be resolved, which they did — but we still got a production update delay.</p><p>3) Finally, during Volume testing, we saw that the Pipeline’s work slowed down because of a large number of files from previous launches — the monitoring function was affected. Still it was decided to automate old records archiving in the next Pipeline version, for now transferring this responsibility to the Production Support team as a manual-maintenance task.</p><p>The supporting documentation had relevant instructions, but with no emphasis in big red letters on WHEN exactly it should be done. As a result, the moment was missed, and — what an unfortunate coincidence — monitoring did not work on exactly that day when Amazon ran out of resources.</p><p>Lessons Learned</p><p>Based on our experience, we would recommend the following:</p><p><strong>1) Look behind the scenes</strong></p><p>As in UI testing, you need to check, not only at the actual results, but also make the absence of a logged Exceptions a part of the Acceptance Criteria.</p><p>The monitoring of the services should be constant, accompanying each test run. It’s not a one time event.</p><p>In our case, for development in the Amazon cloud, in addition to usage of ready-made services management and log monitoring tools, it would be ideal to set up up a CloudWatch Alarm that will automatically inform the team about a problem during development.</p><p><strong>2) Do recovery training</strong></p><p>Even if a risk is defined as low, you can not only mention it during acceptance demo, but also arrange a recovery “field training” with the Production Support team. Especially when it is easy as ABC — document, demonstrate and let them try it!</p><p>As otherwise, having no experience, even professionals will hesitate to interfere in the production processes, which means your team will be asked to do so while being distracted from other tasks. And time differences may make an insufficient recovery cost even higher.</p><p><strong>3) The devil is in the detail</strong></p><p>Make sure that Support tasks for a team that will take care of the Pipeline in Production are documented with all the details specified, accepted and relevant processes established with the proper framing.</p><p>“That’s it, we all done!” — we could have said. But that was not the end of the story…</p><p>In an ideal world move to another technology stack is an independent process that must not coincide with changes in functional requirements. In the real world, however, it is not the case — often migration to a new stack is done alone with new module development.</p><p>We will discuss it in more detail in our next article Data Warehouse Testing.</p><p>Thanks for reading! Stay awesome.</p><p>If you have any questions, topic suggestions, or want to discuss the AWS Data Pipeline Evolution testing in your organization, <a href="https://greenm.io/">contact GreenM here</a>.</p><p>More articles:</p><p><a href="https://medium.com/greenm/how-to-test-a-serverless-api-using-aws-serverless-gradle-test-runner-b2c806fba04b">How to test a serverless API using AWS serverless Gradle test runner</a></p><p><a href="https://medium.com/greenm/deploy-evolution-serverless-feedback-platform-a76be810f72">Deploy Evolution: AWS Serverless Feedback Platform</a></p><p><a href="https://medium.com/greenm/aws-lambda-or-aws-fargate-the-step-by-step-guide-to-choosing-the-right-technology-925ebcf89b7c">AWS Lambda or AWS Fargate: The Step-by-Step Guide to Choosing the Right Technology</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=935fe02d7621" width="1" height="1" alt=""><hr><p><a href="https://medium.com/greenm/data-pipeline-installation-quality-935fe02d7621">Data Pipeline Installation Quality</a> was originally published in <a href="https://medium.com/greenm">GreenM</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tableau Conference 2019: a story of the Visualization Future]]></title>
            <link>https://medium.com/greenm/tableau-conference-2019-a-story-of-the-visualization-future-9d079d73c374?source=rss----ac7f8be0ecdb---4</link>
            <guid isPermaLink="false">https://medium.com/p/9d079d73c374</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[tableau-consulting]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[tableau-public]]></category>
            <category><![CDATA[tableau]]></category>
            <dc:creator><![CDATA[Lena Kirienko]]></dc:creator>
            <pubDate>Fri, 06 Dec 2019 14:00:51 GMT</pubDate>
            <atom:updated>2019-12-06T14:00:51.612Z</atom:updated>
            <content:encoded><![CDATA[<h4>Gut feelings, supported by facts, and how they applied to the Tableau Conference 2019 at Las Vegas</h4><p>As GreenM ambassadors, and Ukraine User Group reps, this November we participated in the largest &amp; the most important Analytics Conference in Las Vegas — TABLEAU 2019.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Oita1k7_aCnbgC7UubUu-Q.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5PbBkqlh_ykCvEcM4qRbhg.jpeg" /></figure><h3>True inspiration</h3><p>More than 20K data community members interacted in all possible ways, inspiring and learning from each other in the city that never sleeps. (Can confirm: we needed some time for our vocal cords to recover after so many lively passionate talks, as well as a good nap for the networking which went on from dusk to dawn!).</p><p>The event gathered together the best professionals in Data Analytics like Steve Wexler — the Founder of Data Revelations and Co-author of The Big Book of Dashboards: Visualizing Your Data Using Real-World Business Scenarios. It was a great honor meeting him.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DWChFhBun-IPIrXKbZngYw.jpeg" /></figure><p>Communicating with him really makes a solid impression. The energy and confidence that Steve conveys to his peers are hard to put into words. But believe us, this is a fantastic feeling. We would also like to congratulate Steve as a 2019 inductee to Tableau Zen Master Hall of Fame, which is an absolutely well-deserved nomination.</p><blockquote><strong>Steve Wexler — </strong>The Founder of Data Revelations and co-author of The Big Book of Dashboards: Visualizing Your Data Using Real-World Business Scenarios (Wiley, 2017). He is the recipient of numerous awards including being a five-time Tableau Zen Master, Iron Viz Champion, and Tableau Training Partner.</blockquote><blockquote>Steve’s presentations and training courses put together a high level of product mastery with real-world experience garnered via the development of thousands of visualizations for dozens of clients. Innumerable numbers of professionals in organizations of any size have gained from his lessons. His seminars are never boring, clear, with a sense of patience and dash of entertainment.</blockquote><blockquote><a href="https://twitter.com/DataRevelations">Twitter</a> <a href="http://www.datarevelations.com/">Website</a></blockquote><h3>Wow. That was important. Key points</h3><p>This year <strong>Tableau</strong> defined itself as cloud-friendly AI-powered self-service analytics and self-service data management.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kee9A5_vlaXpQ8YerxAPcg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s-qQfi5sTkjHtDRMxXgtvQ.jpeg" /></figure><p>During the event, it was recognized that Companies progress to a single source of Truth on their transactional data and add more from external sources to build a good foundation for their ML/AI initiatives.</p><p>We were happy to learn that for better Data Management Tableau:</p><ul><li>partnered with Amazon on their Modern Cloud Analytics program;</li><li>developed Tableau Blueprint — approach to work with data;</li><li>introduced Tableau Catalogue.</li></ul><p>As for other AI vectors, it should be mentioned that Ask Data (beta introduced in 2018) received a bunch of new functions and seems to be on its development way to advanced level. And there is Tableau Explains functionality that verbalizes insights from visuals in a text form.<br>One more bit of impressive news is the loooooong awaited Dynamic parameters will be introduced next year!</p><p>What else was interesting at TC2019? Obviously, we have to mention the story of Hesham Eissa, Iron Viz co-champion. Really just an inspiring story. What’s even more amazing is how he became acquainted with the world of Tableau less than a year ago. You had to be there to feel the energy he exudes. Hesham doesn’t just like it, visualization is his big thing. You can follow Hesham on <a href="https://public.tableau.com/en-gb/profile/hesham3827#!/">Tableau Public</a> to see the winning DataViz Diverse Diners and the rest of his work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a4dOPDk46rBk1If5BkQXGA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*rq-nA2g291z1fZdCaVlkqA.jpeg" /><figcaption>Source: tableau.com</figcaption></figure><h3>The Power of Tableau Community</h3><p>This is what makes the conference really stand out, all thanks to Taha Ebrahimi, Caroline Yam, Ella Worsdale and other terrific leaders who contributed their time and expertise!</p><p>As regular attendees, we can say that the spirit of community has become vibrant, attractive and open to everyone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WGSUWfmIRwl3QkVmpiaX0g.jpeg" /></figure><p>It infects you with a sense of positive energy and the unrelenting enthusiasm of Tableau aficionados of all sorts — from alumni to Zen Masters. User Group Leaders provide guidance for local events arrangement. Makeover Monday, Twitter meet-ups, and many others on &amp; offline activities support the dialog and make the knowledge sharing unstopping and unstoppable.</p><p>For those who’d like to check in more on the new features, here’s a review prepared by Jen Underwood we’d recommend having a read: <a href="https://www.jenunderwood.com/2019/11/25/tableau-conference-2019-new-chapter/">Tableau Conference 2019 Starting A New Chapter</a></p><p>Another review we really like was prepared by Tableau Ambassador Ella Worsdale: <a href="https://dataliciously-ella.blog/2019/11/27/data19-tableau-conference-las-vegas-nov-2019/">#data19 — Tableau Conference (Las Vegas — Nov 2019)</a>.</p><p>If you’re looking for a strategic partner to help you get the most from your data visualization, contact us today for a data assessment.</p><p><em>This article was originally published on </em><a href="https://greenm.io/tableau-conference-2019-a-story-of-the-visualization-future/"><em>GreenM</em></a><em>.</em></p><p>Related Articles:</p><p><a href="https://medium.com/greenm/data-visualization-tips-538fd8e95018?source=collection_home---4------3-----------------------">Data Visualization Tips</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d079d73c374" width="1" height="1" alt=""><hr><p><a href="https://medium.com/greenm/tableau-conference-2019-a-story-of-the-visualization-future-9d079d73c374">Tableau Conference 2019: a story of the Visualization Future</a> was originally published in <a href="https://medium.com/greenm">GreenM</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>