<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Simon Brooks on Medium]]></title>
        <description><![CDATA[Stories by Simon Brooks on Medium]]></description>
        <link>https://medium.com/@SpeakingFrank?source=rss-7933f0563fc5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ut3ZRcuSDOPvW9XRfHTahw@2x.jpeg</url>
            <title>Stories by Simon Brooks on Medium</title>
            <link>https://medium.com/@SpeakingFrank?source=rss-7933f0563fc5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 01:52:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@SpeakingFrank/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Getting Started with Elasticsearch while on Furlough.]]></title>
            <link>https://medium.com/@SpeakingFrank/getting-started-with-elasticsearch-while-on-furlough-c2251c769cb9?source=rss-7933f0563fc5------2</link>
            <guid isPermaLink="false">https://medium.com/p/c2251c769cb9</guid>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Simon Brooks]]></dc:creator>
            <pubDate>Tue, 12 May 2020 19:07:02 GMT</pubDate>
            <atom:updated>2020-05-12T19:07:02.647Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cYRDVGCslX1G24eYgoJAJw.png" /></figure><p>To say we are in difficult times would be a massive understatement, but while everyone&#39;s situation is different I am lucky that my wife and I can both work from home. That said as you can guess from the title I am currently Furloughed and have been for nearly three weeks now.</p><p>In between being the teacher to our son, I am trying to expand my knowledge of Elasticsearch and the other AWS services that can be used with it. I am interested in understanding how the AWS offerings differs to that of Microsoft, whose tools I have been using for over a decade.</p><p>For anyone trying to learn Elasticsearch you will need to be prepared to write a lot more code, be it cURL or Python, then you would usually have to write when using Microsoft Business Intelligence and data tools. In my opinion I also found it harder to find concise step by step instructions that didn’t require a good back ground in AWS. My aim in this post is to bring together the knowledge I have gained during this process and hopefully save someone else some of my pain.</p><h3><strong>What is Elasticsearch</strong></h3><p>I am not going to write anything on this subject as I would only be paraphrasing the many different web pages I have read over the last month. Here are some links if you would like more info on Elasticsearch (ES). The below is a link to the Elasticsearch product, but you can all just search on Google, other browsers are available, and you will find hundreds of articles and web pages on the subject.</p><p><a href="https://www.elastic.co/">Elastic - The Search AI Company</a></p><p>I am thankful I have access to Pluralsight, which has a few videos that I have used during my learning. The step by step demo’s were very useful, but I found it difficult to follow along with all the demos, either because the required resources were not available, the syntax would not work on windows, or the necessary code was not included in the resources for the course. I did however watch both videos and found the content very useful.</p><ul><li>Searching and Analyzing Data with Elasticsearch: Getting Started (by Janani Ravi)</li><li>Building an Elasticsearch Cluster with Amazon Elasticsearch Service on AWS (By Jordan Yankovich).</li></ul><p>This is not to say that you may find these video’s exactly what you need, it really depends on your use cases and the resources you have access to.</p><h3><strong>Elasticsearch in AWS (Amazon Web Services)</strong></h3><p>There are a number of different ways to get started with ES (Elasticsearch), you can download the open <a href="https://www.elastic.co/downloads/elasticsearch">source code</a> locally and run an instance from a local machine, you can work with a <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.6/docker.html">docker image</a> and finally, starting an ES domain in AWS. I also used some of the resources from this <a href="https://www.logicalfeed.com/posts/1182/upload-bulk-json-data-to-elasticsearch-using-python">page </a>while I was working with the on-premise version.</p><p>I managed to get ES running using all three implementations I mentioned earlier. I started with the local instance and docker versions, but I found it difficult to translate the provided cURL statements so that they would work in Windows. I am sure I could have persevered at this point, but I decided to move onto AWS. I have provided a link to how I got started with the other two methods, but for this post I am going to concentrate in detail on ES in AWS.</p><p><strong>Useful Commands</strong></p><p>Before I move on to AWS I did collect a few nuggets of useful code snippets while I was working with ES on-premise, some of which you may already know, but I will share never the less.</p><p>The first is a very useful command when working on the command line, something in my day job I have very little need to use, other than the occasional ping or ipconfig. The command is <a href="https://stackoverflow.com/questions/18391932/how-can-i-shorten-the-path-of-a-dos-prompt"><em>prompt $g</em></a><em>, </em>which will shorten the current directory to just “&gt;”. This is extremely useful if you are working with long directories, as the example below shows. You now have much more space in your window to type commands. If you are using this method and you need to know the current directory use the “cd” command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*cHonGl0rgkueAXlgyFXX0g.png" /><figcaption>prompt $g — thanks to the post on Stack overflow, link included.</figcaption></figure><p>Another one which a developer recently showed me was how to start a command line session directly from a Windows Explorer directory. All you have to do is navigate to a directory in an Explorer window, then click in the address bar to highlight it, then type <em>cmd</em>, which will launch a command line window in that directory.</p><p>I also picked up a few cURL commands during my learning, but only a few as you will see I moved onto Python when I started working with ES in AWS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*48_wb-85tH5q8OmM2eDRTA.png" /></figure><p><strong>Create Elasticsearch Domain in AWS</strong></p><p>I am not going to walk through the entire process for this as there is already a wealth of walk-through dedicated to this subject, I will however point out the ones I used during my work. Before we get started on ES you will need to have access to an <a href="https://aws.amazon.com/free/?all-free-tier.sort-by=item.additionalFields.SortRank&amp;all-free-tier.sort-order=asc">AWS account</a>. For the purpose of my training I have created an account using my personal email address and I am using the free tier services only.</p><p>After creating your account you can follow <a href="https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg-create-domain.html">this guide</a> to create your first domain. These instructions are also included in the “Building an Elasticsearch Cluster with Amazon Elasticsearch Service on AWS” video on Pluralsight. The only thing you need to do differently from the guide is select the t2.small instance type (Data Node) if you want to stick to the free tier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*u6OrudJGSN5TMz1z8YYuuQ.png" /></figure><p>Please be aware that these settings are only for demo purposes as the correct level of security has not been applied. Once you move on from the basics you will want to look at <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started_create-admin-group.html">IAM </a>and <a href="https://aws.amazon.com/blogs/database/get-started-with-amazon-elasticsearch-service-use-amazon-cognito-for-kibana-access-control/">Cognito</a> authentication to secure your app correctly. Once you confirm your settings your ES domain will be created, but it can take up to 15 minutes for the status to change to active.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*esZEOlP-LTu1SjLlv94bFg.png" /><figcaption>ES Domain Endpoint and Kibana</figcaption></figure><p>Once the domain status is active you can test the domain is active by clicking on the Endpoint link. When I first clicked on the link it launched a new window where I received the following error;</p><pre>{&quot;Message&quot;:&quot;User: anonymous is not authorized to perform: es:ESHttpGet&quot;}</pre><p>The aim of this post is not to go into detail around AWS security, the quickest way I found to get this to work was to modify the access policy and allow your public facing IP address to access the resources. Again I must point out, which I am sure I will do a few more times, that this is only to get started, this policy will need to be revisited at a later point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/312/1*h-bRca-i2oBgJZ9Go_A0EA.png" /></figure><p>You can use the following web site to view your <a href="https://www.whatismyip.com/">public IP</a>, which can change depending on how you connect to the internet. From the drop down select, Custom access policy, then IPv4 and type in your public IP address. Make sure the permission is set to allow and then click submit. This is again going to take a few minutes to process. Once the domain status is Active you should now be able to click the domain endpoint and receive a json message in the window.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZERHoA8soYCOBH-jVPlDVw.png" /></figure><h3><strong>Using Jupyter Notebooks to add, delete and search ES</strong></h3><p>There are lots of different ways and languages that can be used to work with ES, but for the purpose of this post I have written a Python notebook. I tend to write my notebooks in <a href="https://docs.microsoft.com/en-us/sql/azure-data-studio/download-azure-data-studio?view=sql-server-ver15">Azure Data Studio</a>, which is free, mainly because it has a SQL kernel, which allows me to create notebooks for my SQL Server work. You can of course open the notebook inside Jupyter and run it from there. Instructions for working with notebook inside Azure Data Studio can be found <a href="https://www.sqlservercentral.com/articles/using-notebooks-in-azure-data-studio">here</a>.</p><p>You can either copy and paste the sections as I run through them or the better option would be to clone my <a href="https://github.com/Git-Hub-Simon/ElasticSearch-in-AWS">Gitbhub repo</a>.</p><h3>Notebook content explained</h3><ol><li>The first section of the notebook imports the required modules. Each of them allows for different commands to be executed, but the detail of what each one does can be viewed in the code snippets that follow.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/1*npcvbZH2r5d2Tr9ePo9DUg.png" /><figcaption>Importing modules</figcaption></figure><p>2. I use the next section to update the global domain variables which is then re-used in the following code. You will need to copy and paste your domain endpoint from the AWS Console and paste into to the domain variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hsePaGi3BE1geKvkWXT_6Q.png" /><figcaption>Setting Variables</figcaption></figure><p>3. While testing my code I found that I was checking the status of the index a number of times. To streamline my code I have created my own function which will return the status of an index.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*mH3jtfg9shQErSA7C6lHmQ.png" /><figcaption>Functions</figcaption></figure><p>4. Now the initial setup is complete we can start to look at using the ES domain. First lets runs some simple code to return the status of the domain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/453/1*8nYFpNp_MX-tZKEcp5OBcQ.png" /><figcaption>ES Details</figcaption></figure><p>This will return the same output that you should see if you click on the endpoint link in the AWS portal.</p><p>5. You can also check the status of the node(s) and return some high level details of the existing indices, if any exist.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*5snY03T6F5qQvYLvt8sO_Q.png" /><figcaption>Node and Index Status</figcaption></figure><p>This will provide you with some health information about each index, the document counts and some other high level information. Throughout the code I have used print or <a href="https://docs.python.org/3/library/pprint.html">pprint </a>to return the results. Both these functions do a similar job, but there are slight differences. Change which one is commented out and see what the differences are.</p><p>6. Now it is time to create an index. There are a number of different ways to do this, either by creating a blank index, or creating it at the point in which you upload data. The below creates an empty index name “test_index” and then shows you the status of this index.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/504/1*Q1apHdXczpucUg8YGybY_A.png" /><figcaption>Create blank index</figcaption></figure><p>7. For testing purposes I have also included the code required to delete an existing index.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/503/1*PnamzsyyBsBjSdkgCO_J6g.png" /><figcaption>Delete an index</figcaption></figure><p>8. Lets look at inserting a single document into a new index. The code below inserts a very basic document containing three elements, the author, some text and the timestamp. You can also change the index_name and doc_type variables if required. The doc_type variable relates to the document type which is metadata about your document. It is a good way to group documents together, something that is explained in detail <a href="https://logz.io/blog/elasticsearch-mapping/">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*UhPuQYLhMsQTIduH7boWLQ.png" /><figcaption>Adding single document</figcaption></figure><p>9. Once you have data in an index you can then start to use the search features within ES. The most basic search is to return your documents based on the index name. In a notebook the default setting means that only 10 results will be returned, but this can be changed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*lxTUWjwwEtOiOjEXN8DNyA.png" /><figcaption>Searching by index</figcaption></figure><p>10. The next level of search is to also specify the document type. This is not very exciting at the moment as we currently only have one document loaded into one document type. You can see this if you look at the output of the two queries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/1*1koVTGIsWrtCrz0jvXZhng.png" /><figcaption>Searching by index and doc type</figcaption></figure><p><strong>Loading a larger data set</strong></p><p>To really show the power of ES we need to first load a larger data set. For this I have downloaded two samples, the complete works of Shakespeare and some dummy accounts data. The json files can be downloaded from <a href="https://kobelb.github.io/kibana-rbac-docs/tutorial-load-dataset.html">here</a>, but they are also included in the repo under sample files.</p><p>11. This section is for setting the variables that will be required. You need to provide the directory path for wherever you have saved the sample files. If you copy this from an explorer window then you will need to reverse the direction of the slashes. You do not need to change the index_name or doc_type if you are uploading the accounts data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*JkkYRL11i2ItHOgtTwnISg.png" /><figcaption>Set file directory</figcaption></figure><p>12. Running the below code will use the <a href="https://kb.objectrocket.com/elasticsearch/how-to-use-python-helpers-to-bulk-load-data-into-an-elasticsearch-index">bulk </a>upload method to import the data. The file in question has been setup specifically for this type of loading.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*I7vXv7eKrhE6awsefL0v7g.png" /><figcaption>Bulk upload</figcaption></figure><p>13. The next two code blocks are examples of how you can use the search features of ES to return the relevant documents. You can also limit the results and tell the query which data points should be returned.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*6-MoXps-V7Rt_l-hhq2Pwg.png" /><figcaption>ES Search method 1</figcaption></figure><p>You can also use more complicated filters. The below returns a single document where the balance of the account is greater 10,000 and less than 20,000.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/1*3zf1MB3i0ad1dISg14rYkw.png" /></figure><h3>Using Kibana to visualise your data</h3><p>There are a number of different visualisation tools that work with ES, but Kibana is an open source offering which is included in your ES domain, so seems like a good place to start. You can launch Kibana from the ES overview page using the Kibana link.</p><p>I am not going to cover Kibana in any great detail in this post, instead I will just explain the basic steps I took to create a very basic dashboard. Before you can do anything with the ES Data in Kibana you need to create an Index Pattern.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y2KWOxqoTJZ-ZGSCFUuJIg.png" /><figcaption>Create an Index Pattern 1</figcaption></figure><p>Navigate to management menu on the left hand side and then select “Index Patterns, then click on the create new pattern button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hUFr10Wz2DlLqqslnPnCxA.png" /><figcaption>Create an Index Pattern 2</figcaption></figure><p>Use the free text box to search for the index(s) that you would like to include in the pattern. For this demo I have used a single index of accounts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8kCLBd6PMxLYtVTXPNZ6lQ.png" /><figcaption>Create an Index Pattern 3</figcaption></figure><p>There are other settings that you can configure on the second screen, but I left them blank and left Kibana to create a unique id.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o6pSxS452HdU-fhL_s3UeQ.png" /><figcaption>Create an Index Pattern 4</figcaption></figure><p>The above shows all the columns and their types, the main types can not be changed from here, but it is possible to make some changes to how the data is displayed. For example if you are working with a string field you can change it to an URL, you can truncate the value or even do a static lookup. These settings can be changed for each data point using the pencil icon on each of the lines. I am going to leave everything as it is and come back and make changes later if required.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wsQONjGIcyF-JzB-cYxZHA.png" /><figcaption>Create an Index Pattern 5</figcaption></figure><p>You don’t have to but I made this index pattern my default. Now you are ready to start creating visuals that can then be added to a dashboard. This post is primarily about ES so I am not going to cover all the steps required to create a dashboard, but the below shows what I was able to create in less than 30 minutes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fq11MaY8ur7zZD56QsiqRw.png" /><figcaption>Kibana Dashboard</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c2251c769cb9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Integrating Business Intelligence]]></title>
            <link>https://medium.com/@SpeakingFrank/integrating-business-intelligence-70faf2a29ce3?source=rss-7933f0563fc5------2</link>
            <guid isPermaLink="false">https://medium.com/p/70faf2a29ce3</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[bi]]></category>
            <dc:creator><![CDATA[Simon Brooks]]></dc:creator>
            <pubDate>Wed, 15 Jan 2020 14:12:01 GMT</pubDate>
            <atom:updated>2020-01-15T14:12:01.684Z</atom:updated>
            <content:encoded><![CDATA[<p>There is one constant between all the organisations I have worked in; they always think about the Business Intelligence (BI), Reporting, Analytics and MI too late in the process.</p><p>Is this the wrong thing to do, I mean you can’t report on sales, email traffic or meter readings without a system in place. Personally I do believe that this is the wrong thing to do in today’s data driven landscape and hopefully the rest of this blog will explain why.</p><h4><strong>Types of BI</strong></h4><p>To the untrained eye BI is a function of an organisation that sits at the side busily providing reports, but in reality organisations are looking to data more and more to provide quantitative evidence to help them make decisions. To understand this fully you need to consider the different types of Business Intelligence, as shown in the diagram below, and how these are key to making fact based decisions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*JF0yPCSgNC7FhveskNwc8w.png" /><figcaption>Business Intelligence Chains</figcaption></figure><h4><strong>Making Decisions</strong></h4><p>The chains shown above forge the required information for making a number of different types of decisions;</p><ul><li>Which, how and when customers are using our products</li><li>Which features have been successful</li><li>What features should be developed</li><li>Technology performance and scaling</li><li>As a customer “am I getting a good return on investment”</li></ul><p>When you read this list you start to understand how integral data and BI is to building a successful business.</p><h4><strong>Right Time for BI</strong></h4><p>So how am I going to make sure that the BI function within Glasswall can provide the relevant intelligence to the right people at the right time? The primary focus, as mentioned previously, is making sure I am involved as early on in the process as possible. I hear you ask why is this important, well if we take the list of decision points above and relate them to a new feature being developed; You of course need architects, designers and engineers to develop the solution, but you also need to;</p><ul><li>use existing data feeds to prototype the feature</li><li>measure performance</li><li>monitor usage</li><li>capture any related data points consumed or generated by the feature</li></ul><p>If there is no BI involvement at the beginning then the feature will still be delivered, but will you be able to monitor how and who is using this feature. Customers don’t always use software/products as expected and understand how things are used can inform how features are updated or even drive a new feature request. Purchases of the software/product will also have an interest in how and who is using it within there own organisation. This is extremely important as it can be a deciding factor when justifying the financial outlay.</p><p>To achieve this at Glasswall I will be making sure that there is a BI presence in each squad, thus hopefully enabling BI to be considered at the correct time. Glasswall also have a fantastic SRE (Site Reliability Team) who I will continue to work with to bring together the strengths and experience from both disciplines.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70faf2a29ce3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Importance of Backups and how the Cloud makes it so much easier.]]></title>
            <link>https://medium.com/@SpeakingFrank/the-importance-of-backups-and-how-the-cloud-makes-it-so-much-easier-1232f7d96147?source=rss-7933f0563fc5------2</link>
            <guid isPermaLink="false">https://medium.com/p/1232f7d96147</guid>
            <category><![CDATA[backup]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[glass-wall]]></category>
            <dc:creator><![CDATA[Simon Brooks]]></dc:creator>
            <pubDate>Wed, 27 Nov 2019 20:31:18 GMT</pubDate>
            <atom:updated>2019-11-27T20:31:18.213Z</atom:updated>
            <content:encoded><![CDATA[<p>Most people learn the hard way how important database backups are, usually when they delete something they shouldn’t have. You may ask why I am posting about a topic which has literally been blogged about forever.</p><p>Well firstly I believe you can learn a lot from history. The second is because today after 15 years of working with SQL I managed to truncate 12 tables in the wrong database, thankfully only in the development environment. Personally I am blaming SQL Server Management Studio for opening my script against the wrong server, pause for laugh.</p><p>A few years ago I would have been trying to find a DBA who could do the restore for me, but thanks to Azure I was able to trigger a point in time restore. The database was back to normal within 30 minutes. I then did the honest and open thing and let the rest of the developers know what had happened.</p><p>Now this was not a serious incident as it was only Dev, but there are plenty of people out there who would have been concerned about admitting this mistake to their boss, let alone the entire team. I don’t know if it is a sign of the times or maybe just the culture within Glasswall, but I didn’t hesitate to let them know. I always say “You can’t make anything without making mistakes”, maybe someone else has said that before, but I am hoping it is original.</p><p>I will finish with some thoughts and actions from today. Don’t worry about making mistakes, it happens to everyone, we are only human. That said, I don’t see any harm in putting additional measures or installing software that can help mitigate this type of issue. I am seriously going to suggest an add-in for Management Studio that colour codes your tabs based on environment. I know Redgate do one, but I am sure there are other alternatives.</p><p>Maybe one last thing, please make sure you backup your databases.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1232f7d96147" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designing the new Data Warehouse]]></title>
            <link>https://medium.com/@SpeakingFrank/designing-the-new-data-warehouse-e25ed94d345d?source=rss-7933f0563fc5------2</link>
            <guid isPermaLink="false">https://medium.com/p/e25ed94d345d</guid>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <dc:creator><![CDATA[Simon Brooks]]></dc:creator>
            <pubDate>Tue, 26 Nov 2019 22:33:29 GMT</pubDate>
            <atom:updated>2019-11-27T08:26:41.122Z</atom:updated>
            <content:encoded><![CDATA[<p>Business Intelligence (BI) is an interesting term, one that most people don’t really understand. I always dread meeting new people, they often ask me what I do, I find myself leading with BI, but then move quickly to Analytics or Reporting. It seems strange to me that data is at the heart of nearly every business, but yet this term is alien to most.</p><blockquote><strong>Business intelligence</strong> (<strong>BI</strong>) comprises the strategies and technologies used by enterprises for the <a href="https://en.wikipedia.org/wiki/Data_analysis">data analysis</a> of <a href="https://en.wikipedia.org/wiki/Business_information">business information</a>.</blockquote><p>Back on topic, designing a Data Warehouse (DW), in my opinion, requires you to understand a few things;</p><ol><li>Who are the end users and what hats do they wear?</li><li>What questions should it spark or even answer?</li><li>What data are we collecting and will it enable us to do point 1 and 2?</li></ol><p>I think a lot of people start with the data they have, but I believe you have to understand the people and different personas that will be interacting with the BI sitting on top of the Data Warehouse. Without this knowledge you could end up designing a Data Warehouse that is not fit for purpose.</p><p>I have spent the last few months spending time with different stakeholders around the business collecting and understanding their needs and wants. I thought this would be a good subject for my first ever blog post, so please be kind.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q9BfE5DrzAknVAgeL5zCYA.png" /><figcaption>First draft of Data Warehouse</figcaption></figure><p>The above sketch was done on my iPad, where I do some of my best design work. The aim of this diagram is show the logical high level design of the new Data Warehouse I am developing and how each of the entities interact with each other. I will continue to revise this design and share my progress and the final design in future blogs.</p><p><strong>How to read the diagram</strong></p><p>I use the different colours to make it easy to pick out the different aspects of the diagram;</p><p>White – These are the individual entities which will become the fact and dimension tables in the physical model.</p><p>Red — These are the logical keys between the entities.</p><p>Yellow — Details the different types of Dimensions being used.</p><p>Blue — At a basic level they show the relationship between the entities. The different lines provide a little more detail about that relationship.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*052Tcq-4ncvniQ5T4ZIGRA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e25ed94d345d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>