<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tushar Gupta on Medium]]></title>
        <description><![CDATA[Stories by Tushar Gupta on Medium]]></description>
        <link>https://medium.com/@tushar17205?source=rss-14da48e0a74------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*wrM_AVbSUho7_w47</url>
            <title>Stories by Tushar Gupta on Medium</title>
            <link>https://medium.com/@tushar17205?source=rss-14da48e0a74------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 14:48:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tushar17205/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[WATCHDOG : A simple audit logging library]]></title>
            <link>https://medium.com/@tushar17205/watchdog-a-simple-audit-logging-library-888154af3fd1?source=rss-14da48e0a74------2</link>
            <guid isPermaLink="false">https://medium.com/p/888154af3fd1</guid>
            <dc:creator><![CDATA[Tushar Gupta]]></dc:creator>
            <pubDate>Sun, 22 Oct 2023 20:14:31 GMT</pubDate>
            <atom:updated>2023-10-31T11:42:08.930Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>WATCHDOG : </strong>A simple audit logging System</h3><p>In large software companies like Deutsche Telekom, a plethora of consumer products are centrally developed, and administrators bear the critical responsibility and authority to configure customer properties in alignment with their specific business needs. When any malicious configuration, whether intentional or accidental, occurs, it can have a profound impact on business operations, necessitating swift resolution.</p><p>For instance, picture this scenario: You and your friends are eagerly anticipating the India vs. Pakistan final match, and suddenly, the app requires a forced update because someone on the app’s side scheduled it at the wrong time. As a result, you’re unable to watch the game until the app updates. Or consider another situation where you’re in the process of purchasing an iPhone, but upon reaching the payment gateway, you discover that debit cards are inexplicably disabled as a payment method. This could be a result of an admin’s inadvertent mistake.</p><p>Resolving such issues entails a thorough investigation to identify who made the changes, what specific alterations were implemented, and then addressing these issues. This often involves extensive searches through logs and databases, a time-consuming process that can stretch on for hours.</p><p>But With the help of Watchdog we are able to resolve this in just a few clicks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tjTDO2aomt1sz9VXysBG5Q.png" /><figcaption>WATCHDOG Architecture</figcaption></figure><p><strong>What is Watchdog?<br></strong>Watchdog is a versatile system designed for micro-services seeking a comprehensive approach to meaningful log auditing. This powerful library leverages Hibernate listeners, including PostUpdate and PostDelete, to meticulously capture and record alterations made to entities and their related data in a MySQL database.</p><p>With Watchdog in action, every database modification is meticulously documented. This encompasses the recording of both the old and new values for each modified field in every row. Moreover, Watchdog goes a step further by saving vital information, such as the user responsible for the change, the microservice’s name, the timestamp, and additional pertinent details, all of which are seamlessly stored within a MongoDB database for thorough audit trail documentation.</p><p>In essence, Watchdog serves as an invaluable tool for auditing and tracking changes across your micro-services, ensuring transparency, accountability, and precise record-keeping for a wide range of activities within your system.</p><p>How do I integrate watchdog in my micro-service?</p><p>Integrating Watchdog is very simple, <br><strong>step 1</strong>: Add the dependency in POM</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xj3PDks5FK46-0X7Rq5BvQ.png" /></figure><p><strong>step 2: </strong>custom configuration: Add this if you want to record additional fields from SQL table in Audit log, for specifiic databases. or skip audit logs</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L4ffRUCwBENN4vkp1pNugA.png" /></figure><p>Thats all you have to do, and Watchdog will do its magic</p><p><strong>Now where can i see audit logs?</strong> that’s where Watchdog service comes into picture.</p><p>The Watchdog service is a multi-tenant, and multi-microservice platform. Its primary role is to empower users with the ability to easily access and navigate through audit logs specific to their respective microservices. It simplifies the process by offering robust filtering options.</p><p>you can also dynamically map SQL fields to more meaningful names. For instance, the “name” field of a “module_table” can be mapped to “module’s name,” and the “name” field of an “Employee_table” can be transformed into “Employee’s name.” This dynamic mapping enhances the clarity and relevance of the audit logs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lCqaNLunASqFi20v9oBdlQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VgoNcrNj0F4An1mQhupd5g.png" /></figure><p>Watchdog’s contribution to our development workflow is invaluable, and it’s not just a time-saver; it’s a key factor in maintaining the reliability and quality of our microservices. We are proudly recording thousands of recrods every hours across all the micro-services. Currently Watchdog is compatible with MySQL Database and in future we will be expanding to support Mongo DB.</p><p>Contributors: Madhav Pareek, <a href="https://medium.com/u/7b111867530a">Shashidhar Singhal</a>, Vivek Tyagi</p><p>Thanks for reading..</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=888154af3fd1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Standardization & Governance of API logs]]></title>
            <link>https://blog.dtdl.in/standardization-governance-of-api-logs-bd5785d6adaa?source=rss-14da48e0a74------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd5785d6adaa</guid>
            <category><![CDATA[api]]></category>
            <category><![CDATA[gdpr-compliance]]></category>
            <category><![CDATA[gdpr]]></category>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[pii]]></category>
            <dc:creator><![CDATA[Tushar Gupta]]></dc:creator>
            <pubDate>Fri, 11 Mar 2022 16:23:59 GMT</pubDate>
            <atom:updated>2022-03-11T16:23:59.823Z</atom:updated>
            <content:encoded><![CDATA[<h4>Central Logging Solution for transaction logs</h4><p><strong><em>Introduction: <br></em></strong>Reading different logs from different microservices can be quite difficult. There may be a lot of problems in terms of semantics and clarity. Leaking of sensitive information in logs also worsens our problems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*V9bfCFw6-Vw6cpxrbuY2Yg.png" /></figure><p>It is said that the lack of proper logging makes everyone upset. How often have you come across the following complaints?</p><ul><li>A person On-Call in a production issue feels lost in logs as everyone has logged anything &amp; everything!</li><li>The DevOps team is unhappy with the increasing size of the logs and wants the application to remove unnecessary logging.</li><li>The Operations Team &amp; Cross engineering teams are lost in the semantics of logging of different microservices. They are unable to do their first-level analysis.</li><li>The security team wants us to ensure that we don’t store any PII (personally identifiable information) in the logs and are able to govern newly added fields.</li><li>The Engineering Leads want us to ensure that any new API/microservice added must also be logged.</li></ul><p>Also, Isn’t it frustrating when one microservice logs the field ‘natcoKey’ and the other logs the field ‘natCoKey’ and some other service logs the field ‘natco_key’ just to capture the same class of data?</p><p>With a growing number of microservices, each service produces its own set of logs that have different information from one another. In the event of tracking and debugging defects spanning across various microservices, the lack of uniformity in the stored logs makes it exceedingly difficult to analyze the problem.</p><p>But do not worry. why fear, TLogger, and footprint is here 😉</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*p8uJNpqghig-UeqqUWLJIw.gif" /></figure><p>In this article, we will be discussing an approach to standardized logging and PII hiding.</p><p><strong>Enters Tlogger and Footprint…</strong></p><h3><strong>TLogger</strong>:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jllPfub1UsBW6e0IaGQmCw.png" /></figure><p>Tlogger(<strong>T</strong>ransactional <strong>Logger</strong>) is a library that is to be used by each microservice, to make its’ transactional logs standard and PII free. Tlogger is a jar that is imported into a spring-based microservice. This jar exposes an annotation that is to be put on top of controller methods, whose transactions are to be logged. Given some configurations, It can do its magic. It polls data from Footprint every 15 minutes to get newly configured data.</p><h4><strong>Key features</strong>:</h4><ul><li><strong>Common semantics: </strong>Every microservice must provide some standard information to the Tlogger via configurations. TLogger enforces a fixed schema for standardization of logged fields, which allows it to use common fields to populate different data for different microservices.</li><li><strong>Request/Response capture</strong>: Tlogger can get all the information related to the request and response of an API call.</li><li><strong>GDPR Formatting</strong>: If there exists any sensitive data in the request header, body, param, or response body of an API call, it can be omitted, hashed, masked in the logs which are printed. Tlogger fetches this information from the <strong>Footprint.</strong></li><li><strong>General Exception Handling</strong>: In case an exception arises on an API call, Tlogger tries to deduce the status and the reason for the exception. This enables any general type of exception to be handled, without confirming any specific type.</li><li><strong>Nested Logging</strong>: Tlogger also can log the information of the southbound(downstream) calls made in an API call.</li></ul><p>Tlogger also ensures that all these tasks are done at high speeds so that the normal operating performance of the application is not affected. A combination of caching, smart data structures, and algorithms allow Tlogger to do so.</p><h3><strong>Footprint:</strong></h3><p>Now comes the task to provide data for Tlogger in a generic and configured format. For this, we need to generate a Swagger file of a microservice for which we want to use Tlogger. Footprint parses this swagger file to provide an interface to configure each field of an API individually 😋. It is a multi-tenant and multi-microservice user interactive platform for API metadata configuration. It allows users to mark every field of APIs’ parameters, Request, Response, and Headers as skip/hash/plaintext/mask as per the requirement. This configuration is stored in a DB and then is later used by Tlogger for logging and masking by fetching this data.</p><h4><strong>Key features:</strong></h4><ul><li><strong>Multi-tenant and multi-microservice: </strong>Since we work for multiple countries, and each country has many microservices, this platform allows a user to choose the country and the respective microservice.</li><li><strong>Configuration:</strong> Parameters of an API can be set as hashed/skipped/masked or plain using the UI.</li><li><strong>Global Masking:</strong> There are a few parameters that we know beforehand is a PII or security concern, like if params contain keywords like Auth, Address, Phone number, etc. It Allows the Security team to Globally add Keywords in the Hash/Skip list, and then every API’s parameter of all tenants having those keywords get hashed or skipped right after uploading swagger.</li><li><strong>Logging: </strong>Select if any specific API should be logged or not and enforce GDPR policy in Tlogger.</li></ul><p><strong>How do I set them up with my existing microservice?</strong></p><p>Well, Integrating TLogger in a microservice is quite simple. The following steps are required to integrate Tlogger in a spring-based microservice:</p><p><strong>Step 1:</strong> Tlogger: Add dependency in POM</p><p>Tlogger is used as a jar. Including that jar in our pom imports Tlogger.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O-_N5VYUGjw9VusOrpO4ug.png" /></figure><p><strong>Step 2:</strong> Tlogger: Add annotation</p><p>Annotate the controller methods which require to be a part of the transactional logs. Additionally, some properties can be set in the annotation itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qRnRzqfa2niOTaHX3loLgw.png" /></figure><p><strong>Step 3: </strong>Tlogger: Some configs in the properties file</p><p>Tlogger needs to know the Footprint endpoints and some basic information to proceed for logging.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/1*WlhKBttwduPbrooJnyEKZA.png" /></figure><p><strong>Step 4:</strong> Footprint:</p><p>Let us take a walk through Footprint</p><ul><li>Select Tenant and microservice to upload swagger</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*xOxYi9ND6jHDzYziHU93RQ.gif" /></figure><ul><li>Fetch Details and Configure API Details</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*-RrJw2gZFU3ZtB-LVWddpw.gif" /></figure><ul><li>Configure API Headers</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*XOgWeZB1GxYBIS2TsBApOQ.gif" /></figure><ul><li>Add keywords in Additional List: Every microservice can also add extra keywords other than the Global list.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9HYtWUe2AAlRgTnyE0lvUg.gif" /></figure><p><strong>Step 5:</strong> Configure File beat, Logstash</p><p>We can configure any file listener which can read the log files from the k8 pods. A log forwarder can pass the logs to our database. In our case, we use Filebeat and Logstash to index our logs in elastic.</p><p>Finally, we can now enjoy viewing our logs on Kibana…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3J4JV_RUBOMZoW503mG6kQ.png" /></figure><p>As you can see in the screenshot above a field named ‘appName’ is indexed in the logs. This field presents different values for different microservices. So, if we want to view the logs for a specific microservice, we can put a filter on this field.</p><h4><strong>Conclusion:</strong></h4><p>With some initial effort of integrating tlogger into our microservices, we experienced a momentous change in the way we look at our logs. This made it extremely easy for us to track issues across various microservices. Maintaining logging code also became quite easy due to the central management of Tlogger jars.</p><p>Also, the PII hiding feature takes away the pain of manually hiding sensitive information application-wise. Integration with Kibana gave us a portal to view indexed logs and visualizations as well.</p><blockquote>Contributors : <a href="https://medium.com/u/55714221e430">Rishabh Sharma</a>, <a href="https://medium.com/u/7b111867530a">Shashidhar Singhal</a>, <a href="https://medium.com/u/91fd67afb877">Rahul</a> and Saurabh Malhotra</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*mVOjZK-gy4nvsiDPmXs7WA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd5785d6adaa" width="1" height="1" alt=""><hr><p><a href="https://blog.dtdl.in/standardization-governance-of-api-logs-bd5785d6adaa">Standardization &amp; Governance of API logs</a> was originally published in <a href="https://blog.dtdl.in/">Deutsche Telekom Digital Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>