<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vikram Mehta on Medium]]></title>
        <description><![CDATA[Stories by Vikram Mehta on Medium]]></description>
        <link>https://medium.com/@vikram.mehta?source=rss-ce8fb4ad09ac------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*q7aK2erJM1uGnP8NmfOMKg.jpeg</url>
            <title>Stories by Vikram Mehta on Medium</title>
            <link>https://medium.com/@vikram.mehta?source=rss-ce8fb4ad09ac------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 00:51:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vikram.mehta/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Best Practices on Account Security]]></title>
            <link>https://tech.makemytrip.com/best-practices-on-account-security-57e16b9ff0f6?source=rss-ce8fb4ad09ac------2</link>
            <guid isPermaLink="false">https://medium.com/p/57e16b9ff0f6</guid>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Vikram Mehta]]></dc:creator>
            <pubDate>Thu, 01 Nov 2018 09:46:27 GMT</pubDate>
            <atom:updated>2018-11-01T09:46:27.498Z</atom:updated>
            <content:encoded><![CDATA[<p>MakeMyTrip is continuously striving to implement best security practices to protect customers from threats such as password cracking and ensure a secure and seamless experience. However it needs vigilance from the user’s end as well, to make sure their account is not vulnerable to such threats. Even though we have been implementing stricter password policies; keeping common, guessable passwords leaves the account vulnerable to misuse since the probability of compromise increases; what makes things worse is; similar passwords used on multiple websites.</p><p>Simple precautions, however, can mitigate such risks. Here are a few pointers that you can keep in mind while creating a password:</p><p>1. Don’t cut it short : A good password is at least 8 characters long, shorter passwords are mathematically more susceptible to guessing attacks.</p><p>2. Names, Addresses, Dates, Contact Numbers are not passwords : Many criminals profile the victim before they begin guessing the password, so chances are that they already know these details about you.</p><p>3. Keyboard Patterns are not passwords either : We all love to roll our fingers over qwerty, 123456, asd123, aaaaaaa etc. but these render your passwords weak.</p><p>4. <strong>Common words are bad</strong> choices for passwords : ‘Password’ is a bad password, period. So is ‘hello’, ‘letmein’ and other commonly used words. Hackers use digital dictionaries of such words while guessing, so make sure yours doesn’t land in them.</p><p>5. <strong>Don’t share passwords</strong> across accounts : This ensures that a <strong>domino effect on other accounts does not happen</strong> if credentials of one of your accounts get compromised.</p><p>6. <strong>Change passwords</strong> (once) in a while : Yes, we love to cling to the old ones and reuse them, but just like everything else passwords have a shelf life too.</p><p>7. Consider a reliable password manager : We know the above steps might be time consuming or tricky to carry out on an on-going basis, reliable password managers can ease out this process.</p><p>The classic way of creating a strong password is to use a combination of alphabets, numbers and special characters. Eg. Bob&amp;Smith@73#.</p><p>Another good alternative is to use a phrase of two/more unrelated words stringed together Eg.sushi_badminton</p><p>Bottom line, passwords are the first line of defense for protecting your accounts. A wise choice of password toes the line of thought ‘A stitch in time saves nine’.</p><p>In addition to all of these preventive measures, be vigilant of the activity that occurs on your account, and in case something looks amiss, immediately bring it to the attention of the concerned organization / website and start changing your passwords.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57e16b9ff0f6" width="1" height="1" alt=""><hr><p><a href="https://tech.makemytrip.com/best-practices-on-account-security-57e16b9ff0f6">Best Practices on Account Security</a> was originally published in <a href="https://tech.makemytrip.com/">MakeMyTrip-Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SOC-X — Part 3 — Bringing it all together with Apache Metron and Blitz at 1500 eps]]></title>
            <link>https://tech.makemytrip.com/soc-x-part-3-building-the-pipeline-with-apache-metron-and-blitz-at-1500-eps-389d529019a2?source=rss-ce8fb4ad09ac------2</link>
            <guid isPermaLink="false">https://medium.com/p/389d529019a2</guid>
            <category><![CDATA[information-security]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Vikram Mehta]]></dc:creator>
            <pubDate>Tue, 29 May 2018 10:33:47 GMT</pubDate>
            <atom:updated>2018-06-01T04:10:54.175Z</atom:updated>
            <content:encoded><![CDATA[<p>We scratched the surface about big data analytics, ML and dataShark in our previous blogs. It’s now time to set the real thing in motion!</p><p>Before we start, we would like to call out some brilliant stuff done by the Hortonworks team in putting together a much-needed SOC enabler, Apache Metron. The intent of this blog-post is not to go over the steps to build a SOC pipeline, since this is already well documented in the Apache Metron docs section (<a href="https://metron.apache.org/current-book/index.html">here</a>). We will try and cite the problem statements we were faced with, optimisation / performance tuning parameters, customisation, alert management and so on.</p><p><strong>**Disclaimers**</strong></p><p>This blog is published basis our experience with Apache Metron, it’s supporting environment, and use cases / problem statements we were trying to solve. There are no guarantees on the approach, and there could be many more possibilities wrt customisation, optimisation and performance tuning.</p><p>For a better understanding of this blog, it’s important to possess prior knowledge of the following:</p><ol><li>Apache Nifi</li><li>Apache Metron</li><li>Apache Storm and it’s performance tuning guides</li></ol><p>Let’s start with some problems that we were trying to solve:</p><ol><li>Ease out data / log ingest, addition of new devices</li><li>Managing high throughput ingest</li><li>Alert management (de-duplication, enrichment, tracking / ticketing)</li><li>Automated / semi-automated incident response</li><li>Security data lake for forensics and analytics</li></ol><p>We’ll now try and walk through some of the specifics that helped us past the above:</p><p><strong>Data ingest:</strong><br>The Apache Metron streaming pipeline expects logs to be written into kafka queues (one for each data source), while most security devices or required log sources ship logs over syslog. There are a few options available here, let’s talk about the following with their pros / cons:</p><ol><li>Apache Nifi — eases out work-flow / log processing immensely, however, there would be performance implications when we expect complex computation (such as parsing, regex) bring processed at a high throughput. Nifi is extremely useful when we would like to process incoming data from varied data sources quickly, or for quick dev / testing, but not the best option when the throughput is high.</li><li>KafkaCat — a better option for high throughput log forwarding from syslog to Kafka, not memory intensive unlike Nifi, but would not work well with complex computation or workflows.</li></ol><p>Below are Nifi / Storm / Metron specific performance parameters that might require tuning:</p><ol><li># of Kafka partitions — ensure there are adequate partitions to take the load and be available for Storm consumers</li><li># of threads / workers — it’s is advisable to distribute parsing topologies for high throughput sensors across multiple nodes, this ensures a single node doesn’t take the beating, also ensures HA</li><li>Nifi — scheduling is an important aspect, a longer run duration helps higher throughput, at the cost of latency though</li></ol><p><strong>Alert management:</strong><br>Assuming the ingest is working as expected, and use cases have been written, we’ll focus a little more on managing alerts. Let’s consider the below elasticsearch document for an alert from snort:</p><pre>{<br> “elapsed”: 161,<br> “code”: 200,<br> “ip_dst_addr”: “199.27.79.73”,<br> “original_string”: “1461576382.642 161 127.0.0.1 TCP_MISS\/200 103701 GET http:\/\/<a href="http://www.cnn.com//">www.makemytrip.com\/</a> — DIRECT\/198.27.12.73 text\/html”,<br> “method”: “GET”,<br><strong> “is_alert” : “true”,</strong><br> “bytes”: 103701,<br> “action”: “TCP_MISS”,<br> “ip_src_addr”: “127.0.0.1”,<br> “url”: “homepage.html”,<br> “timestamp”: 1461576382642,<br> “source.type”: “squid”<br>}</pre><p>Now if this event was generated multiple times, it would create multiple documents with <strong><em>“is_alert = true”</em></strong>, hence increasing the number of alerts being reported. In order to address this, we followed the below approach.</p><p>Create an additional parser with the below configuration:</p><pre>{<br>   “parserClassName”:”org.apache.metron.parsers.json.JSONMapParser”,<br>   “sensorTopic”:”<strong>indexing</strong>”,<br>   “filterClassName” : “STELLAR”,<br>   ”parserConfig” : {<br>         “filter.query” : “<strong>is_alert == ‘true’</strong>”<br>   }<br>}</pre><p>Create a Nifi work-flow to de-duplicate alerts basis varying criteria:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*3uQJpsxIDINkSULSedjxCQ.png" /><figcaption>Nifi Workflow</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/1*PfGldl-6bytfTzgd1Bp8ig.png" /><figcaption>“Generic Mail Handler” processor — offloading alerts to Blitz</figcaption></figure><p>The alerts-indexing topic would now contain de-duplicated alerts for reporting purposes.</p><p>Next-up we present blitz, MMT’s open sourced incident management framework (<a href="https://github.com/makemytrip/blitz/">https://github.com/makemytrip/blitz</a>); which will help in alert enrichment and tracking.</p><p>Let’s assume we require the following actions to be done once a snort alert is generated:</p><ol><li>Enrich the alert with Source WhoIs and Source Violations in Snort</li><li>Raise an incident in JIRA</li><li>Supply actions (via buttons in the email) such as “Close the incident in JIRA” and “Block the source on Firewall”</li></ol><p>We direct the Nifi work-flow to the blitz python script, which supports the above actions.</p><p>The Blitz configuration looks like:</p><pre>“subject”: “soc-x : possible web attack : {ip_src_addr} : {ip_dst_addr}”,<br>“mail_title”: “Possible Web Attack — {hit_rate}”,<br>“enrichment”: True,<br>“enrichment_conditional”: False,<br>“enrichments”: [<br>     {“e_name”: “Bookings Made”, “e_func”: “booking_data”, “e_key”: “ip_src_addr”},<br>     {“e_name”: “Reverse DNS”, “e_func”: “reverse_dns”, “e_key”:      “ip_src_addr”},<br>     {“e_name”: “WhoIs Org”, “e_func”: “org_whois”, “e_key”: “ip_src_addr”},<br>     {“e_name”: “Blacklist Check”, “e_func”: “whitelist_check”, “e_key”: “ip_src_addr”},<br>     {“e_name”: “WhoIs Details”, “e_func”: “whois_info”, “e_key”: “ip_src_addr”}<br>],</pre><p>By the end of it, we have de-duplicated and enriched mail alerts, automated / semi-automated actions via mailers / buttons.</p><p>Enriched Alerts:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/1*-PJ6Ajoit-D-kw6wQ8kvBQ.png" /><figcaption>ReverDNS, WhoIs and internal API enrichment</figcaption></figure><p>Actions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*BlcJK7ZKRs2FJJoPUOhQaQ.png" /></figure><p>Alerts / SOC Analyst Dashboard:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PwDthguzvT22cSTlUpmePg.png" /></figure><p><strong>Security data lake:</strong><br>The metron indexing module enables logs / data to persist at HDFS in batches. Over time, the file system acts as the warehouse for security logs. The key here is, the ability to read, parse and present these logs for forensics or analytics.</p><p>Let’s take a look.</p><p>Sample log file in HDFS:</p><pre>-rw-r--r--   1 storm hdfs     129024 2018-05-24 18:37 /apps/metron/indexing/indexed/alerts/hdfs-hdfsIndexingBolt-27-0-1527164329010.json</pre><p>Creating a table using a JSON Serde in Hive:</p><pre>CREATE EXTERNAL TABLE `alerts`(<br> `ip_src_addr` string COMMENT ‘from deserializer’,<br> `ip_dst_addr` string COMMENT ‘from deserializer’,<br> `host` string COMMENT ‘from deserializer’,<br> `detector` string COMMENT ‘from deserializer’,<br> `alert_title` string COMMENT ‘from deserializer’,<br> `utc_timestamp` double COMMENT ‘from deserializer’)<br>PARTITIONED BY (<br> `year` int,<br> `month` string,<br> `day` int)<br>ROW FORMAT SERDE<br> ‘org.apache.hive.hcatalog.data.JsonSerDe’<br>STORED AS INPUTFORMAT<br> ‘org.apache.hadoop.mapred.TextInputFormat’<br>OUTPUTFORMAT<br> ‘org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat’<br>LOCATION<br> ‘hdfs://10.0.0.1:8020/apps/metron/indexing/indexed/alerts’<br>TBLPROPERTIES (<br> ‘transient_lastDdlTime’=’1527587257&#39;)</pre><p>Running SQL commands:</p><pre>hive&gt; select count(*) from alerts;<br>OK<br>50<br>Time taken: 3.541 seconds, Fetched: 1 row(s)</pre><p>Visualisation using Zeppelin Notebook:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*U9N5UMw086bxH4LsG1uOrA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=389d529019a2" width="1" height="1" alt=""><hr><p><a href="https://tech.makemytrip.com/soc-x-part-3-building-the-pipeline-with-apache-metron-and-blitz-at-1500-eps-389d529019a2">SOC-X — Part 3 — Bringing it all together with Apache Metron and Blitz at 1500 eps</a> was originally published in <a href="https://tech.makemytrip.com/">MakeMyTrip-Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SOC-X : Part 1 — Machine Learning and Big Data Analytics in Security]]></title>
            <link>https://tech.makemytrip.com/soc-x-part-1-machine-learning-and-big-data-analytics-in-security-c33c4aab1918?source=rss-ce8fb4ad09ac------2</link>
            <guid isPermaLink="false">https://medium.com/p/c33c4aab1918</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[information-security]]></category>
            <dc:creator><![CDATA[Vikram Mehta]]></dc:creator>
            <pubDate>Tue, 02 Jan 2018 08:44:36 GMT</pubDate>
            <atom:updated>2018-01-04T07:47:51.550Z</atom:updated>
            <content:encoded><![CDATA[<p>Being an online commerce platform, managing incident response effectively is one of the core responsibilities of MMT’s IT Security Function. This five part blog series focuses on taking a traditional SIEM implementation to the next level by implementing various tools and techniques. It aims at addressing the following technology or operational issues with traditional systems:</p><ol><li>Standard rule based security technology that detects only the “<em>known bad</em>”</li><li>Inability to process big and fast data efficiently</li><li>Time taking forensics investigation and subsequent action</li><li>Manual incident management</li></ol><p>With Part 1, we’ll first focus on #1 from the perspective of website security. Standard security technology such as a WAF, IPS, traditional SIEM are good, but for a purpose — detecting the “known bad”. As long as the OEM or an analyst is able to define a signature for detecting malicious behaviour, such technology would work great in mitigating the associated threat. But what about the “unknown”? Attacks are getting more and more complex, harder to detect and are increasingly targeting mis-configuration or business logic flaws. Detecting such intrusion or mis-use becomes extremely hard to detect using traditional signature based systems since attackers could use legitimate channels of access and exploit available functionality.</p><p>This left us hunting for something more, a technology that will help find a “needle in the haystack”, will help detect behaviour that is “not normal” compared to previous trend. Way back in early 2015, when big data and machine learning was mostly used to address business use cases such as personalization, we stumbled across Apache Spark (PySpark) and felt it could solve such security use cases, we will attempt to take you through our journey with Spark and associated use cases.</p><p>To keep things simple, we will attempt to walk through the following two use cases that leverage big data analytics and machine learning (this assumes the reader possesses basic knowledge of Python and Apache Spark).</p><p><strong><em>Use case 1 : </em></strong><em>Detecting malicious communication from endpoints using Proxy logs</em></p><p>Let’s assume an organisation has standard malware protection technology in place (Anti-Malware, Proxy, IPS etc). This use case attempts to detect communication that has bypassed the afore-mentioned security defence. We’ll use Apache Spark to analyse GBs of proxy logs to find interesting patterns / indicators of malicious communication.</p><p><em>Let’s first dissect a proxy logline:</em></p><pre>1294011828.214 1309 221.164.123.121 TCP_MISS/200 10032 GET <a href="http://wonderproxyblog.com/">http://</a>squid.com vik DIRECT/76.74.254.120 text/html</pre><p><em>Interesting fields for us</em></p><ol><li>User</li><li>URL</li><li>Destination Host / IP</li><li>Bytes transferred</li><li>Time elapsed</li></ol><p><em>Let’s extract them:</em></p><pre>def do_logparse(n):<br>       SQUID_LOG_PATTERN = ‘(?P&lt;timestamp&gt;\d+)\.\d{3}\s+-(?P&lt;time_elapsed&gt;\d+) (?P&lt;dhost&gt;.*) (?P&lt;cache&gt;\w+)\/(?P&lt;httpresponse&gt;\d+) (?P&lt;size&gt;\d+) (?P&lt;uri&gt;.+) (?P&lt;user&gt;\S+) (?P&lt;method&gt;[A-Z]+\/\S+).+!\n’<br>       match = re.search(WEBSENSE_LOG_PATTERN, n)<br>       return Row(<br>             log_type = ‘SQUID’,<br>             d_host = match.group(2),<br>             user = match.group(7),<br>             bytes = match.group(5),<br>             http_method = match.group(8),<br>             resource = match.group(6),<br>             time_elapsed = match.group(1)<br>       )</pre><p><em>What are we looking for?</em></p><p>Communication to domains that belong to the “unknown” category, there are some challenges:</p><ol><li>We might see a ton of IPs instead of hostnames (we’ll convert them to hostnames via SSL and WhoIs lookup)</li><li>We will need to lookup domain categorisation in real-time</li></ol><pre>#Count the number of times a domain has been accessed<br>dhosts = parsed_logs.map(lambda x: (x.d_host, 1)).reduceByKey(lambda a, b: a + b)</pre><pre>#Filter out only un-categorized domains<br>uncategorized_dhosts = dhosts.join(reputation_matrix).map(lambda x: (x[0], x[1][1])).filter(lambda x: x[1] == None)</pre><pre>#Enrich IPs with SSL domain name<br>sdhosts_ssl = uncategorized_dhosts.map(get_ssl_cert)</pre><pre>#Enrich IPs with whois intel<br>dhosts_whois = uncategorized_dhosts.map(get_whois_intel) </pre><p><strong><em>Use case 2:</em></strong><em>Web traffic anomaly detection using Apache HTTPD logs</em></p><p>Consider a website that deploys standard perimeter protection technology such as WAF, IPS that block known attacks such as SQL injection, cross-site scripting. We would like to identify / flag attacks that bypass such defence technology. We’ll use a similar setup to the previous example using Apache Spark to create a model of “known” or “good” IP behaviour and then compare incoming traffic to this model to arrive at an “anomaly score” which would depict how far the IP is from normal.</p><p>Let’s assume we have parsed Apache HTTPD logs available. We will generate a K-Means model using the following dataset:</p><pre>[&lt;source_ip&gt;, &lt;# of 200 hits&gt;, &lt;# of non-200 hits&gt;]</pre><p>We then create the model using one-minute of HTTPD logs as our training set, post which we test incoming HTTPD logs in one-minute batch windows against the trained model. The key here is, to identify the cluster to which the source IP belongs, and how far it is from the cluster centroid. The farther it is, the more anomalous it’s behaviour.</p><p>The below code snippet illustrates this example:</p><pre>def make_features(a, b):<br>    # Add up counts of 2xx and 5xx response against a source IP<br>    twoxx = a[0] + b[0]<br>    nxx = a[1] + b[1]<br>    return [twoxx, nxx]</pre><pre>#Reduce training data to get counts of 200 and 5xx response codes against an IP<br>rawTrainingData = rawTrainingData.reduceByKey(make_features)</pre><pre>#Convert data to K-Means acceptable format<br>training_set = rawTrainingData.map(lambda s: s[1])</pre><pre>model = KMeans.train(training_set, 2) #Train the model</pre><pre>#Apply the model on incoming testing data over a 60sec windo<br>w<br>results = rawTestingData.reduceByKeyAndWindow(make_features, lambda a, b: [a[0] - b[0], a[1] - b[1]], 60, 60).map(lambda s: predict_cluster(s, model))</pre><p><strong><em>Important considerations for ML</em></strong></p><p>There are few aspects to keep in mind that largely govern the effectiveness of an ML use case:</p><ol><li>Feature selection — this is heart of your ML model. Make sure the features (or attributes) selected or generated directly impact and are relevant to the use case. For example, if we’re looking at DNS data exfiltration as a use case, the most relevant features would be [# of DNS requests per host, size of DNS request per host]</li><li>The right algorithm — depending on your use case, seek assistance from data science teams to identify the right algorithm to generate the most effective outcome. For example, whether to use supervised learning as opposed to un-supervised learning.</li><li>Training data — pick the right data for training your model, and make sure there are adequate samples of varying kinds depending upon your use case.</li></ol><p><strong><em>Sky’s the limit!</em></strong></p><p>There are many other use cases that a security function can leverage, listing some below:</p><ol><li>Detecting anomalous network activity</li><li>Detecting DNS data exfiltration</li><li>Detecting bots / scrapers</li></ol><p>We realised, writing all this code time and again could be cumbersome and not really the ideal job-role for a security analyst. To get around this, we built a big data and ML framework called “dataShark” in 2016 and made it available for public consumption as an open source project — <a href="https://github.com/makemytrip/dataShark">https://github.com/makemytrip/dataShark</a></p><p>More on dataShark in “Part 2 — The dataShark Framework”.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c33c4aab1918" width="1" height="1" alt=""><hr><p><a href="https://tech.makemytrip.com/soc-x-part-1-machine-learning-and-big-data-analytics-in-security-c33c4aab1918">SOC-X : Part 1 — Machine Learning and Big Data Analytics in Security</a> was originally published in <a href="https://tech.makemytrip.com/">MakeMyTrip-Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>