<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tower Street on Medium]]></title>
        <description><![CDATA[Stories by Tower Street on Medium]]></description>
        <link>https://medium.com/@towerstreetHQ?source=rss-e150678758d9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*8-O-VjugyXBHeH9s.jpg</url>
            <title>Stories by Tower Street on Medium</title>
            <link>https://medium.com/@towerstreetHQ?source=rss-e150678758d9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 16:33:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@towerstreetHQ/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Bootstrapping Datasets with Machine Aggregation and Human Annotation: Cyber Risk Case Study]]></title>
            <link>https://medium.com/@towerstreetHQ/bootstrapping-datasets-with-machine-aggregation-and-human-annotation-cyber-risk-case-study-4b9028e2c1ad?source=rss-e150678758d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b9028e2c1ad</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[cyber-risk-management]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[cyber-risk]]></category>
            <category><![CDATA[information-technology]]></category>
            <dc:creator><![CDATA[Tower Street]]></dc:creator>
            <pubDate>Wed, 03 Jul 2019 16:50:54 GMT</pubDate>
            <atom:updated>2019-07-03T16:50:54.666Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ryAGfP5RUNnh3aKA" /></figure><p><strong>Why Build a New Cyber Risk Dataset</strong></p><p>After we introduced a new and more complete <a href="https://medium.com/@towerstreetHQ/the-worlds-most-complete-breach-dataset-for-cybersecurity-risk-models-b5e51c6a2efb?postPublishedType=repub">cyber risk dataset</a> for cyber risk quantification and underwriting, a number of folks in the community asked for more details on how we’ve built the annotation process and tooling for the data we’re collecting.</p><p>As mentioned in the previous post, cyber risk quantification currently lacks the necessary volume of data and rich set of features that are useful for cyber risk modeling. Most datasets have only basic information about the attack and don’t expand on the state of the organization at the time of the incident. Often we have information about when the incident occurred and/or when the incident was made public, but we don’t have information about the financial state of the company, the state of its security, or how the attacker exploited it. There is occasionally some information about the attack, but the labels are not well defined and don’t follow an established standard framework.</p><p>Another challenge occurs when cross-referencing the organizations with other data sources because company names can be written in several different ways.</p><p>In order to have a more complete view over security incidents, we built a dataset where we could see what security controls were affected in the attack, which losses did the company incur as a result of the incident, what was the financial state of the company and finally which record types were affected.</p><p><strong>Scalable Aggregation with an Automated Pipeline</strong></p><p>We take security incident information from 16 different sources — 12 state attorney general listings and 4 public aggregators. As for financial and organizational information on the companies, we use information from public data sources such as Yahoo Finance, but also 14 additional proprietary sources.</p><p>We uniquely identify an organization using Named Entity Recognition (NER). This allows us to identify the company’s name by recognizing the identifiable portion of the organization’s name — e.g. ‘Apple’ in ‘Apple Computers’ or ‘Apple, Inc.’.</p><p>Given the records from different sources, we then deduplicate them and merge records connected to the same event using a model over announcement, incident date, and other features — different events are unlikely to have the same times and other properties co-occur without being about the same event.</p><p><strong>Quality Control with Manually Deduplicated Records</strong></p><p>The aggregation engine still generates some errors and incomplete data. Often there are different events occurring in short intervals or dates from the sources are mixed up, creating records that are erroneously merged. We needed to improve this if we want to have a robust enough dataset for modeling and taking principal risk. We also wanted additional features providing more information about the incidents in order to test hypotheses from the team and security advisory board, so we decided to start manually annotating.</p><p>We could not find any tools or services that made this task efficient enough, so we built a custom toolchain and methodology to handle the annotation, and we built an annotator workforce and training program.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*g1UdukVApOyzkdrx" /><figcaption>Deduplicating and merging incidents from different sources in the annotation toolchain</figcaption></figure><p>Annotators are given rows of information from different sources about one company. Each row has the original source of information, plus often any external source that was attached to the original source. The annotators read through this data, identifying incidents that are duplicates. The UI allows for drag and dropping into buckets representing the same incident.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SIqovLzQuEnu6y3m" /><figcaption>Annotating essential information about the security incident</figcaption></figure><p>For each of the deduplicated events, the annotator initially identifies the essential information of the event — the date when it was published, when the incident occurred, the total number of records and their type. Occasionally this information is present in the original data source, so the job of the annotator is to find inconsistencies in the data, and to add missing information.</p><p><strong>Affected Security Controls Annotation using the CIS Framework</strong></p><p>In our <a href="https://medium.com/@towerstreetHQ/bayesian-cyber-risk-quantification-with-industry-specific-models-749766443247">modeling</a>, we use the <a href="http://cisecurity.org/">CIS framework</a> to establish the security profile of a company. Even though the detailed security state of the company at the time of the incident isn’t mandatory to disclose publicly, it is often possible to infer which <a href="https://www.cisecurity.org/controls/cis-controls-list/">CIS controls</a> were affected in the incident, since some details about the attack are provided to the public. We, therefore, ask the annotators to identify the CIS controls that could have prevented the incident or significantly mitigated its impact. This gives a proxy to which controls were missing or not working properly at the target organization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ekoEpoPRFs1fJiml" /><figcaption>Identifying the CIS controls that were affected in the incident</figcaption></figure><p>We’ll publish the detailed methodology for annotating each of the CIS controls soon. In this post we give an example using our scenario-based methodology so that the annotator can identify the affected controls by understanding the scenario of the attack.</p><p>Let’s say a phishing attack from a human error resulted in a theft of credentials — a user imputed his credentials to a phishing site. The credentials were then used to log in to a remote machine and later the attacker found out that POSs (pay terminals) were located on the same network. The attacker then used this access to steal credit card information from the terminals.</p><p>In this scenario, we can reason that the employees of this company were not trained sufficiently (<em>CIS 17: Implement a Security Awareness and Training Program</em>). The ability of the attacker to log in remotely with stolen credentials implies that multi-factor authentication was missing (<em>CIS 16: Account Monitoring and Control</em>), and finally, the missing separation of sensitive applications on the local network implies <em>CIS 14: Controlled Access Based on the Need to Know</em> was either missing or not implemented correctly.</p><p><strong>Incurred Losses</strong></p><p>For researching the losses incurred due to the incident, we read the original sources available about the attack and researched the costs using specific detailed queries that we’ve trained the analysts on. We also review financial statements for publicly listed companies to look for concrete evidence of these costs. We annotate only concrete numbers we can find mentioned, we do not allow annotators to project estimates — only directly citable figures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*F5QbOVYFsqCC4lXA" /><figcaption>Annotating the costs that the company had as a result of the security incident</figcaption></figure><p>During our research, we defined a Loss Graph, which contains generalized events such as Customer Credit Monitoring and Notification, and Third Party Litigation. Annotators look for evidence of a total cost and also look to link losses associated with each specific category in the Loss Graph.</p><p><strong>Financial and organizational company profile</strong></p><p>The dataset has several financial indicators available at the time of the publicly disclosed incident. These values are taken directly from the automated pipeline aggregating structured information, so they are not annotated manually.</p><p>These values include (all at the nearest time of the announcement) the price of stock, revenue, net income, number of employees, ICB industrial sector, operating expenses, among many others. In the initial batch of events we’ve annotated, these values are available for approximately 60% of the events.</p><p>If you’re interested in more details about the data, review the original intro post on the <a href="https://medium.com/@towerstreetHQ/the-worlds-most-complete-breach-dataset-for-cybersecurity-risk-models-b5e51c6a2efb?postPublishedType=repub">cyber risk dataset</a>. We welcome any experts who can be helpful in scrutinizing the data, suggesting new sources to aggregate, improving its quality, and recommending better methods for aggregation and annotation. We have a limited release private beta, so please include a bit about your background and use case to: miguel.pinheiro@towerstreet.co</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b9028e2c1ad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Which CIS Controls Matter: An Empirical Analysis of Cyber Insurance Risk Assessment]]></title>
            <link>https://medium.com/@towerstreetHQ/which-cis-controls-matter-an-empirical-analysis-of-cyber-insurance-risk-assessment-172b9118210b?source=rss-e150678758d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/172b9118210b</guid>
            <category><![CDATA[dataset]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[risk]]></category>
            <category><![CDATA[insurance]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <dc:creator><![CDATA[Tower Street]]></dc:creator>
            <pubDate>Wed, 26 Jun 2019 15:55:30 GMT</pubDate>
            <atom:updated>2019-06-26T16:10:12.631Z</atom:updated>
            <content:encoded><![CDATA[<p>The CIS standard is a set of <a href="https://learn.cisecurity.org/cis-controls-download">Critical Security Controls</a>. The recently announced <a href="https://medium.com/@towerstreetHQ/the-worlds-most-complete-breach-dataset-for-cybersecurity-risk-models-b5e51c6a2efb">Cyber Risk Dataset</a> is a historical breach dataset with manual annotations for which CIS controls were affected during a breach. This allows us to conduct a new analysis of which CIS controls insurers look at, what the dataset says, and what security experts express through CIS prioritization. We find a stark contrast between the priorities set by the CIS controls and those insurers focus on, which may be related to the availability of data.</p><h4><strong>The CIS standard is prioritized by security experts</strong></h4><p><a href="https://learn.cisecurity.org/cis-controls-download">CIS Critical Security Controls</a> is a framework developed by the <a href="https://www.cisecurity.org/">Center for Internet Security</a> (CIS) and represents a prioritized set of best practices designed to prevent the most pervasive and dangerous cyber attacks today.</p><p>We use the CIS standard because unlike the other more comprehensive standards such as NIST or ISO2700x, the CIS Top 20 framework was developed to provide organizations with a more concise set of <strong>prioritized</strong> controls. It was developed by security experts from different industries to help organizations reduce risk and establish a baseline cybersecurity program.</p><h4><strong>Annotating historical CIS controls from breach data</strong></h4><p>For risk quantification, we need to derive weights for the importance of individual controls. In an ideal world, we would like to get information about the state of the security controls at the time of the breach. However, this information is not widely available since companies are not compelled to disclose details about their security posture at the time of the breach.</p><p>Our security team has developed a scenario-based methodology for each CIS control and trained human annotators to be able to manually impute the affected CIS controls for each breach. From the first batch of 2,750 largest and most notable breaches, we managed to attribute at least one CIS control to approx. 1,000 cases.</p><p>Let’s say a phishing attack from a human error resulted in a theft of credentials, e.g. a user imputed his credentials to a phishing site. The credentials were then used to log in to a remote machine and later the attacker found out that POSs (pay terminals) were located on the same network. The attacker then used this access to steal credit card information from the terminals.</p><p>In this scenario, we can reason that the employees of this company were not trained sufficiently (<em>CIS 17: Implement a Security Awareness and Training Program</em>). The ability of the attacker to log in remotely with stolen credentials implies that multifactor authentification was missing (<em>CIS 16: Account Monitoring and Control</em>), and finally, the missing separation of sensitive applications on the local network implies <em>CIS 14: Controlled Access Based on the Need to Know</em> was either missing or not implemented correctly.</p><h4><strong>Identifying the most important CIS controls</strong></h4><p>Our dataset is heavily skewed towards consumer data breaches, given that companies are compelled to disclose these breaches. With this in mind, we can still perform a simple aggregation and count how many times each CIS control occurred in the annotated dataset. You can see the result in Figure 1 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/949/1*xm5FolfUT02b6a7k9zR12g.png" /><figcaption><em>Figure 1: Shows the total number of times a CIS control could have prevented a cyber breach</em></figcaption></figure><p>The 3 most frequently affected controls from our analysis are:</p><p><strong>CIS 14 — Controlled Access Based on the Need to Know:</strong> This covers all the cases when the network was not properly segmented based on application and data sensitivity, e.g. cases when retailer’s Point Of Sale (POS) devices were on the same network as regular employee endpoints. It also includes cases when shared folders were not properly protected with access controls and unauthorized people had access to sensitive data such as IP, PII, PHI, PFI, etc. Finally, scenarios such as unencrypted hard drives lost during transport by 3d parties, stolen unencrypted laptops, and disk drives.</p><p><strong>CIS 13 — Data Protection:</strong> This control covers all scenarios related to data stolen from undocumented or misplaced storage locations (laptops, network drives, 3d party cloud providers, etc.), data backups, legacy databases, and applications. Additionally, it includes cases when raw data in the clear text were exfiltrated without detection.</p><p><strong>CIS 17 — Implement a Security Awareness and Training Program:</strong> Covers all cases of fishing and more general cases when the attacker requested an employee to make some action such as making a wire transfer, sending a tax form or other sensitive information. Any unintentional disclosure of sensitive data to the attacker is included as well.</p><p>There is a stark contrast between the chart above and the priorities set by the CIS controls. That doesn’t mean the CIS priorities are invalid, but rather shows how limited inferences from biased public data can differ from the consensus of security professionals. The limitation of breach data is one of the reasons security risk research is so prone to bias and where existing data needs to be combined with <a href="https://medium.com/me/stats/post/749766443247">expert knowledge</a> in order to build representative models.</p><h4><strong>Do insurers focus on the right controls during the risk assessment and underwriting?</strong></h4><p>Let’s compare the results above with the analysis of insurance questionnaires conducted in 2017[1]. This study maps questions from common insurance application forms to individual CIS controls. Figure 2 below is credited to the authors and shows the total number of sub-controls addressed per control for each CIS Control.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1BD6ibk45JuVOQt3ppglNw.png" /><figcaption><em>Figure 2: Showing the total number of sub-controls with a question in a form corresponding to that sub-control [1]</em></figcaption></figure><p>There are obvious overlaps in the two charts, most notably the high occurrence of consumer-related breach controls — <em>CIS 13: Data Protection </em>and <em>CIS 14: Controlled Access Based on the Need to Know</em>, and <em>CIS 16: Account Monitoring and Control</em> and <em>CIS 17: Implement a Security Awareness and Training Program</em>.</p><p>On the other hand, there are also two noticeable differences between the two charts, specifically the relatively high scores of <em>CIS 8: Malware Defenses</em> and <em>CIS 10: Data Recovery Capabilities</em> in insurance applications. The authors of the paper explain that CIS 8 scores high because one of the most common questions asked by the insurers is whether anti-virus and personal firewalls are running on all work stations of the target organization (considered an easy win). Whereas <em>CIS 10: Data Recovery Capabilities</em> is mostly relevant for scenarios relating to business interruption losses and data availability such as ransomware, which insurers want to minimize. This control is tricky because usually the information we have available isn’t about whether the company was able to recover the data (the incident response), but rather which controls were affected during the attack.</p><p>The authors conclude that there is a worrying discrepancy between the security experts opinions and the focus of insurers in the underwriting process and that one explanation may be that insurers focus on these controls more because they mitigate the risk they are liable for. Our analysis above suggests another explanation — perhaps insurers are more biased toward modeling breaches with accessible public data rather than on modeling the overall quality of the security programs implemented by the insureds. If the latter, this would be an interesting example where data limitations bias us against robust global financial resilience from cyber attacks.</p><p><em>Note: The cited article uses CIS controls version 6.0, while the annotation methodology uses version 7.0. You should keep this in mind when comparing the two charts.</em></p><h4><strong>Feedback on the dataset</strong></h4><p>This is the most complete dataset for breaches that we’ve found available, but it is far from complete. We want your feedback — how can the dataset be extended or enriched to make it more representative and useful? Are there other sources of data we can aggregate to add other breach types or annotate other security-related metadata?</p><p>If you have any feedback feel free to contact us at miguel.pinheiro@ceai.io</p><h4><strong>References</strong></h4><p>[1] Daniel Woods, Ioannis Agrafiotis, Jason R. C. Nurse, and Sadie Creese, Mapping the coverage of security controls in cyber insurance proposal forms, Journal of Internet Services and Applications 8 (2017), no. 1, 8. <a href="https://link.springer.com/article/10.1186/s13174-017-0059-y">link</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=172b9118210b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bayesian Cyber Risk Quantification With Industry-Specific Models]]></title>
            <link>https://medium.com/@towerstreetHQ/bayesian-cyber-risk-quantification-with-industry-specific-models-749766443247?source=rss-e150678758d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/749766443247</guid>
            <category><![CDATA[insurance]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[risk-management]]></category>
            <category><![CDATA[cyber-insurance]]></category>
            <dc:creator><![CDATA[Tower Street]]></dc:creator>
            <pubDate>Thu, 20 Jun 2019 16:55:47 GMT</pubDate>
            <atom:updated>2019-06-21T23:14:40.035Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Isl7omC6Rgq9ThMT" /></figure><h4><strong><em>The problem of cyber risk quantification for the security, finance, and insurance ecosystems</em></strong></h4><p>Cyber risk modeling consistently presents researchers with two major challenges; the data available for modeling is incomplete and sparse, and cyber risk requires rich models to capture the potential impact of events that occur in different industries.</p><p>Cyber risk modeling with small, sparse data requires modeling concepts from advanced statistics rather than deep neural networks trained on large, dense data. Today’s cyber risk modelers often work around the data issue by using simple average-cost-per-record models for data breach losses, despite the well-known problems with this approach [1]. Another approach is to limit large variance in the data by reducing the scope of study to a specific subset of events, e.g. Ponemon is focusing on small breaches only (less than 100k records affected).</p><p>Companies are forced to disclose consumer data breaches, but there is a lack of metadata about the incidents; especially around the state of the company’s security at the time of the incident. The companies are not compelled to disclose how the attack happened or the financial impact. Most industries are more concerned about incidents resulting in business interruption. Until recently, the companies didn’t have to make these events public, even when the interruption was visible to end users. To address these issues, we have built <a href="https://medium.com/@towerstreetHQ/the-worlds-most-complete-breach-dataset-for-cybersecurity-risk-models-b5e51c6a2efb">a dataset</a> that includes which CIS security controls were affected and what losses the company suffered.</p><h4>Being industry-specific because you need the granularity</h4><p>Although the folks modeling cost per record, per industry might be missing richness in their models, they are right to look at the different properties on a per industry basis. Different industries have different business models, different security threats against their assets and profits, and different kinds of downstream losses after cyber incidents.</p><p>Our approach leverages industry-specific scoping and subject matter expertise (SME). Careful analysis of the regulatory landscape, historical incidents, and financial expertise allowed us to effectively map out potential losses that can occur after a cyber incident.</p><p>Being industry-specific requires strong SME time investment; a full-time, two-person SME team (one for accounting, one for security) will invest a month or more building out a single industry-specific loss taxonomy. We’ve developed what we call the loss graph, in which our internal SMEs worked with additional industry experts to map out more than 50 different loss scenarios (and growing) that are now included in the loss graph and are grouped in the following 9 loss areas:</p><ul><li>Replacement or Upgrade of Faulty Controls</li><li>Determination of Cyber Event Extent Review of Necessary Legal &amp; Regulatory Action</li><li>Notification, Credit Monitoring, Credit Restoration</li><li>Regulatory &amp; Vendor Fines &amp; Penalties</li><li>Third Party Litigation &amp; Damages</li><li>Loss of Gross Profit</li><li>Contributing Costs</li><li>Extraordinary Costs</li></ul><p>We are working with the security community in a similar way and we build pre-configured security assessment tooling stets for industry-specific streamlined onboarding into security risk quantification and underwriting.</p><h4><strong><em>Being Bayesian because you need the practicality</em></strong></h4><p>At Tower Street, we utilize Bayesian GLMs for the frequency model and a set of calibrated loss distributions for the severity/loss model. These two models in conjunction represent our end-to-end cyber risk quantification model. There are two important pieces of missing historical information that are fundamental for cyber risk modeling and lead to considering a Bayesian approach; information about the historical security posture of companies, and losses that were incurred after a cyber incident.</p><p>The practical way to attack this problem of missing security posture and loss information is to build a Bayesian model, which lets you use as much historical data as you can and initialize the rest of the model based on elicitation from a panel of experts. Then you get out there and start underwriting based on your model. These models allow: (a) expert initialization in the form of informed priors, (b) new data in form of evidence can be used later to update the model and obtain a new posterior distribution — you can make it more empirical over time as you observe more security assessments and downstream losses.</p><p>The process of converting expert knowledge into numerical estimates is called expert knowledge elicitation and there is a big body of research focused on designing the most effective elicitation methodology for the given family of models [2]. Tackling a big problem such as cyber risk is far too complex for any single expert, so we reduce the scope of the analysis to separate elements for which expert inputs can be incorporated effectively.</p><p>There have been many books written on this topic; probably the most well known is Douglas Hubbard’s book How to Measure Anything (and the follow-up How to Measure Anything in Cybersecurity Risk).</p><h4><strong><em>The Pain of Historical data</em></strong></h4><p>Historical community-driven breach datasets are incomplete and sparse, but at the same time, there is a lot of publicly available information that can be aggregated to fill these holes and provide a better foundation for modeling. Most of the publicly available data come from attorney general websites where companies are compelled to disclose consumer data breaches, but also data aggregators. However a lot of information about the incident is typically lacking, and aggregators often only portray data breaches and not other types of attacks.</p><p>We’ve written up the adventure of <a href="https://medium.com/@towerstreetHQ/the-worlds-most-complete-breach-dataset-for-cybersecurity-risk-models-b5e51c6a2efb">building a breach dataset</a> with automated aggregation and human annotation. We’ve built a human annotation toolchain from scratch with a custom app just for the annotators, layered on top of a scalable scraping system and data aggregation pipeline with a bit of Natural Language Processing to ensure quality aggregation of records.</p><p>Using this approach of aggregation and annotation, we gain two major advantages — increasing the number of records in one dataset and the amount of metadata in each record. The largest currently publicly available dataset has around 9.5K incidents, while we have gathered information about more than 30K incidents. We also annotate better metadata about security controls and losses which allows us to perform richer modeling.</p><h4><strong><em>The Rise of Inside-out Assessments, Active Security Testing and GRC Platforms</em></strong></h4><p>The most easily obtainable security data comes from outside-in assessments. These are insufficient for larger companies, where the infrastructure complexity and deception tactics leave outside-in assessment with a more limited view. Based on our experience, the predictive power of this data decreases for larger and more complex companies.</p><p>Our security team, advisors, and insurance partners also cautioned us from the beginning that outside-in data is of limited or no value for larger companies, and that the only real security posture information comes from legitimate inside-out testing of controls.</p><p>Most insurers today rely on a simple security questionnaire when assessing a client’s security posture. While asking some questions is good, it’s important to test the controls and conduct a gap analysis. This missing feedback loop in security is being filled these past few years by terrific new startups that can continuously test the effectiveness of a company’s security program and the state of its security controls — we’d especially draw your attention to SafeBreach and AttackIQ.</p><p>Increased compliance pressure is forcing companies to undertake regular security audits that all usually boil down to a single questionnaire. Companies trying to escape the “excel hell” are quickly moving towards more mature solutions in the form of GRC (e.g. LogicGate, Reciprocity, etc.) and 3d-party risk assessment platforms (CyberGRX). This again provides a great opportunity for streamlining more rigorous risk assessment by ingesting existing questionnaires, and we’ve already demonstrated a 20-minute enterprise-grade risk assessment by ingesting existing questionnaires and quickly filling in anything else needed for the risk quantification.</p><h4><strong>References</strong></h4><p>[1] Edwards B., Hofmeyr S., Forrest S. (2015) Hype and heavy tails: a closer look at data breaches. Workshop on the Economics of Information Security.[2] O’Hagan, A. et al. (2006) Uncertain Judgements: Eliciting Experts Probabilities, book.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=749766443247" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The World’s Most Complete Breach Dataset for Cybersecurity Risk Models]]></title>
            <link>https://medium.com/@towerstreetHQ/the-worlds-most-complete-breach-dataset-for-cybersecurity-risk-models-b5e51c6a2efb?source=rss-e150678758d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5e51c6a2efb</guid>
            <category><![CDATA[insurance]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[cybercrime]]></category>
            <category><![CDATA[risk-management]]></category>
            <dc:creator><![CDATA[Tower Street]]></dc:creator>
            <pubDate>Wed, 19 Jun 2019 16:14:02 GMT</pubDate>
            <atom:updated>2019-06-19T16:28:05.087Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6A1wHLtD2tV3XBVoDENBuQ.png" /></figure><p><strong><em>The pain of historical security incident data</em></strong></p><p>When we conducted our technical due diligence on available historical security incident data sets, we found them to be incomplete. They are missing many incidents, and also missing a lot of interesting data about each incident.</p><p>Breach Level Index has records from 2013, and it has redacted organization names. Identity Theft Resource Center has a truncated set of incidents in 2012 and 2013. The public Verizon dataset (VERIS) has been reducing operations since 2013. None of these projects are staffed or supported adequately from the community to be comprehensive enough.</p><p>All US states now have privacy laws that force companies to disclose breaches of consumer data to the respective attorney general where each consumer resides. Many of those attorney general offices publish those notifications, therefore giving some visibility to the security incidents happening in the US.</p><p>There are many other interesting sources of public data one would also like to aggregate and merge with the incident data, so we built a machine aggregation system for security incidents, and designed a human annotation layer on top for a final pass of incident metadata enrichment.</p><p><strong><em>Show me the data</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xftp37iga8078jU3jQjF2g.png" /><figcaption>Number of incidents throughout the years in our automated pipeline (‘Cyber Risk Dataset’) compared against public aggregators</figcaption></figure><p>We currently have around <strong>37,500 records</strong> automatically identified as unique incidents. From those, we have manually annotated approximately <strong>2,750 records</strong>, which include all security incidents related to publicly traded companies, and all companies which have suffered a data breach of over 1 million records.</p><p>We are releasing the manually annotated data in a small private beta with top academic and industry security risk researchers, to understand how useful this can be to the community, and how we can improve it further.</p><p>The annotated data includes basic information about the incident (organization, publication and incident dates, and the number of records breached), and whether different record types (PII, PHI, PCI, PFI, SSNs, passwords) have been affected in the incident.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/1*YAtcMRvlMzNLI_SWfTbMPA.png" /><figcaption>Number of records breached per record type (vertical axis is on logarithmic scale)</figcaption></figure><p>Since our models are aligned with the <a href="https://www.cisecurity.org/">CIS framework</a> and want to estimate losses, we are also annotating which <a href="https://www.cisecurity.org/controls/cis-controls-list/">CIS controls</a> have been affected in the attack, and the losses in the aftermath of the incident. We have also tried to divide the losses into different loss categories in order to obtain a more meaningful categorization.</p><p><strong><em>The aggregation process and architecture</em></strong></p><p>We aggregate data from attorney general websites from 14 different states, 4 different data aggregators, and our team of manual annotators.</p><p>Although the incident data itself small, we built our pipeline to measure cyber risk globally, so we have aggregated data on over <strong>1.4 million companies</strong>, so the underlying pipeline operates at a larger scale than the private beta incident dataset might imply.</p><p>This pipeline was built in Scala using Spark and Python for modeling. To find duplicates in the data (deduplication), we use Named Entity Recognition (NER) to identify the same company entity when it’s referred differently by different sources — e.g. the company Apple can be referred to as ‘Apple’, ‘Apple, Inc.’, ‘Apple Computers’, etc.</p><p><strong><em>The annotation process and toolchain</em></strong></p><p>Annotating incidents by deduplicating records, together with the enrichment of each attack is quite complex. Annotators need to first select which rows of information refer to the same event, and then add information to all those rows. Since this type of annotation is quite specific and annotating raw data on spreadsheets is borderline impossible to track, we developed an annotation toolchain to speed up the process. Here’s a screenshot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pfu9uKsAshcyOx2M" /></figure><p>Crowdsourcing the annotations didn’t prove successful, which is often the case for such complex deep domain-specific tasks. After iterating the annotating guidelines a few times, we weren’t able to get the accuracy above 25%.</p><p>So we built our own team of annotators which got us to the level of quality we were hoping for. Being able to directly instruct handpicked annotators, combined with developing an intuitive and efficient annotating tool, was the sweet spot that allowed us to more effectively annotate the data. Since we have our own team we were able to personally train them to annotate data that needs expertise to understand, such as the affected CIS controls in each incident, and categorized incurred losses.</p><p><strong><em>Private beta results so far</em></strong></p><p>We carefully selected a set of the world’s best security researchers to participate in the private beta of our data set, and give us feedback on the quality and completeness of the data, as well as our strategy for engaging the community.</p><p>The responses so far have been positive. Many researchers noted the diminishing quality of existing breach datasets and a clear need for a more live-maintained dataset. We also received requests for adding additional metadata such as attack types, actor types, and such — we’re keen to incorporate this into a revision of the annotation methodology.</p><p>We welcome any experts who can be helpful in scrutinizing the data, and improving quality, as well as pointing out better methods for the aggregation or annotation process described above.</p><p>We will have a limited release with a private beta, so you can email your background and use case to: <a href="/service/mailto:miguel.pinheiro@towerstreet.co">miguel.pinheiro@towerstreet.co</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5e51c6a2efb" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>