<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Neil Crofts on Medium]]></title>
        <description><![CDATA[Stories by Neil Crofts on Medium]]></description>
        <link>https://medium.com/@neil-crofts?source=rss-7e6334dac289------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*TN8_sY5n8cG8D5JYRaADcw.jpeg</url>
            <title>Stories by Neil Crofts on Medium</title>
            <link>https://medium.com/@neil-crofts?source=rss-7e6334dac289------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 20:41:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@neil-crofts/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Whatever happened to model driven development?]]></title>
            <link>https://medium.com/bbc-product-technology/whatever-happened-to-model-driven-development-ec0175139720?source=rss-7e6334dac289------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec0175139720</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[model-driven-development]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Neil Crofts]]></dc:creator>
            <pubDate>Thu, 15 Oct 2020 12:50:17 GMT</pubDate>
            <atom:updated>2020-10-15T12:50:17.681Z</atom:updated>
            <content:encoded><![CDATA[<p>At the start of the millennium there seemed to be the promise of methods that would change the way we built software.</p><p>Model Driven Development promised an end to much of traditional software drudgery.</p><p>No longer would we have to hand code and manually test our applications. Instead we’d model them, push a button and <em>as if by magic</em> a fully working application would appear.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*lzSIpif1j_7U134X.jpg" /><figcaption>As if by magic…</figcaption></figure><p>Well it hasn’t quite worked out like that has it?</p><p>Writing code is still largely a bespoke manual process. And it is expensive, not just writing the code but testing it, deploying it, documenting it, and so on.</p><p>It got me thinking. What was so good about MDD? Has it died a death? Or has it just morphed into something else?</p><p>Whatever happened to model driven development?</p><h3>Deconstructing the promise</h3><p>Let’s try and answer these questions, a bit like a television chef deconstructs a recipe, by ripping apart the key ingredients of the topic.</p><p>Firstly, it’s worth clarifying what we mean by a <em>model</em>.</p><p>Here we consider a model simply as an abstraction of something — essentially a simplified, blueprint of a more complex entity or process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J3_VKo_0mj-iDWUHX0xZiQ.png" /><figcaption>Model blueprints</figcaption></figure><p>From a blueprint of a house we can follow a set of procedures and build a house. And we can repeat this as many times as we want. Further if we customize our plan or change our procedures we can end up with a radically different type of building, but all built to the same repeatable and trusted standards.</p><p>Suddenly this is starting to sound interesting and the power of having a model at the heart of the development process becomes clearer.</p><h3>A plurality of representations</h3><p>Possibly the most important aspect of MDD is that the model contains enough unique information to be able to derive the concrete things you want to implement.</p><p>All of the key attributes of the entities should be included in the model. But isn’t this just configuration? Sort of, but not quite. Configuration turns into modelling when you use your configuration at the heart of your design but especially when you use it to create multiple outputs.</p><p>Much of the power of MDD then comes from the fact that a single model may be transformed in many ways. As they say in the world of patents, a model transformed into a plurality of representations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1chrqVLSfUbzGwBA.png" /><figcaption>A plurality of representations</figcaption></figure><p>As a minimum the model can be transformed into the target application. Incidentally <em>application</em> doesn’t necessarily mean a full-stack application. It could of course be just a data processing activity that the model is abstracting.</p><p>Further to the main event, significant time and cost benefit can be realized by also transforming the model to other representations such as test suites. Even more value can be extracted from the model by creating textual documentation or graphical views for documenting the system of interest.</p><h3>Model definitions</h3><p>Now we have an understanding of what a model is, the question becomes how can one represent that model? As MDD is a set of concepts rather than a standard, the format of a model rather depends on the nature of the problem and therefore can be in whatever format that best suits a domain. However, there have been attempts to produce common modelling languages that span many domains.</p><p>In the early days of MDD, the Unified Modelling Language came to the fore. Personally, I struggled with UML. More specifically I struggled with the promise of UML. I am sure there will be many out there who <em>got it</em> and some parts of UML have at least perpetuated, but back in the day I just could not get over some key barriers:</p><ul><li>You needed tooling which was either expensive or free but unreliable.</li><li>You got some diagrams from it, but not complete implementations.</li><li>Round trip engineering was frankly a pain.</li></ul><p>I don’t wish to be too dismissive but fundamentally there was a lot of hype around UML. At the time it seemed to promise a complete solution, but in the end, it just appeared to get in the way. Once people start believing that, an approach is going to struggle.</p><h3>Domain modelling</h3><p>However, there have been plenty of examples where a model driven approach has proven successful.</p><p>Where I’ve personally had more success is through modelling with Domain Specific Languages (DSL), especially in modelling the realms of multi-step web applications, voice apps and also with agents/bots on messaging platforms. In these cases, a model that abstracts away the platform specific details can be a huge productivity boost by providing a simple and trusted language for rapidly building fully functional, production ready services across many target platforms.</p><p>A more recent discovery for me is that model driven techniques are a fantastic match for data processing problems. Specifically, in the Big Data world where Extract Transform Load problems are commonplace. The ability to transform and enrich data reliably from one message format to another without having to hand crank complex code over and over again is incredibly powerful.</p><p>One of the common things about DSLs and Data Models is that processing is completely driven by the modelling. Here round-trip engineering no longer means manually changing your code and then reflecting this back in your model. Instead a change to the model is immediately reflected by the implementation and are by nature in sync — a much more satisfying relationship.</p><h3>Code generation</h3><p>Now we have a model in a given format it should be possible to automatically generate code in a target implementation language. The aim of this is to produce exemplar code that you trust and will be guaranteed to work on your target platform.</p><p>The rules for transforming your model into the implementation language are typically encoded as templates or programmatic business rules. The fundamental principle here is that if you produce well tested transformation rules once, they can be used many times to solve related problems.</p><p>Another way I like to think of these transformation rules are as being what <em>copy &amp; paste</em> should have been.</p><p>On those projects I’ve worked on where code generation has been done well, it has resulted in significant productivity and reliability gains. As such it’s hard to understand why we don’t do more of this as an industry. Perhaps though it is simple trust issue? As developers we are happy to trust the code generated by our compiler or interpreter is correct. So why is it we seem much less likely to trust higher level abstractions?</p><p>As a variation on code generation in some situations it is appropriate to write a model interpreter that knows how to execute the model directly. Updates to the model are reflected immediately in the application.</p><p>This can be an incredibly powerful mechanism and works particularly well when the model is written in the form of a DSL that describes a multi-step activity such as the pages of a web site or dialogues in a voice application or bot.</p><h3>Test inference</h3><p>Possibly the feature of MDD that has seen least adoption as far as I can tell is that of automating the testing process.</p><p>If a model describes what it is supposed to do, then it should be possible to generate tests to show what it should do. Just as importantly it should be possible to infer what it should not do.</p><p>A further possibility is that it is also possible to infer and generate suitable input test data for both happy and sad cases. This can be incredibly useful in big data or distributed messaging applications when availability of data at the right volumes is essential.</p><p>Unfortunately, it seems this use of MDD has not really taken a foot hold in our industry yet.</p><h3>Seeds of time</h3><p>Predicting which technologies and techniques will succeed is like looking into the seeds of time. The specific goals of new approaches and tools may vary but in general the outcome we hope for is some combination of <em>faster, cheaper, simpler</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*56t8niU0n9Is7qFB.png" /><figcaption>Whatever happened to MDD?</figcaption></figure><p>As we have intimated some of the early model driven approaches just didn’t seem to make things simpler. As a consequence, development speed and costs suffer and ultimately developer confidence too. The software community reacts accordingly.</p><p>But is model driven development dead? Certainly not, but like the television chef identifying the key elements of a recipe, I have learnt to identify those problems it is best suited for.</p><p>For instance, after everything I have said about UML, I have recently been enjoying somewhat of a renaissance with it after being introduced to C4 modelling (<a href="https://c4model.com/">https://c4model.com/</a>). By scaling back my ambitions and not expecting full round-trip engineering I have found this approach very effective at helping me to understand and communicate the architectures I am working on.</p><p>Further I’ve also been reflecting on the types of problems where I <em>have</em> ended up using a model-based approach. Often these projects have appeared from the outside to be almost magical in that they solve not only the immediate problem, but also a host of other related problems with minimal change. And therein lies one of the major indicators that a problem is ripe for a model driven approach, namely <em>multi tenancy</em>, which in this context is where you deploy or execute multiple models using the same infrastructure. So, if you are in a problem space where you have not just a single problem to solve but many related ones then a model driven approach may well be more than relevant today.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec0175139720" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bbc-product-technology/whatever-happened-to-model-driven-development-ec0175139720">Whatever happened to model driven development?</a> was originally published in <a href="https://medium.com/bbc-product-technology">BBC Product &amp; Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Examining the limits of serverless]]></title>
            <link>https://medium.com/bbc-product-technology/examining-the-limits-of-serverless-9011fe7b46ea?source=rss-7e6334dac289------2</link>
            <guid isPermaLink="false">https://medium.com/p/9011fe7b46ea</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Neil Crofts]]></dc:creator>
            <pubDate>Mon, 08 Jan 2018 10:47:00 GMT</pubDate>
            <atom:updated>2018-01-08T10:47:00.505Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently the team I work for has been working with serverless technologies. Specifically our team has been implementing a new and simpler method of processing the many millions of records generated when our audiences interact with BBC online services. This provides insight into how people are utilizing BBC services with the aim of ensuring that everybody gets value from our output. While the pros of the new approaches are compelling, we have also been reminded to consider the limitations of these technologies and how they may impact the security of our services.</p><h4>Continuous change</h4><p>During our relatively short time on this project the team has come to the practical realization that the rate of change in cloud platform services is such that the traditional model of write something well and then maintain it for years to come no longer holds true. The expectation now is that we are going to need to constantly reinvent how we do things in order to be able to meet the expectations of our stakeholders while at the same time meeting increased volumes yet still reducing costs.</p><p>Typical industry solutions for big data processing include using MapReduce across a set of servers to batch process the data. This involves creating clusters of machines which can be non-trivial, not just in building the solution but also in terms of maintaining it and managing it through its lifetime. However with the advent of serverless technologies such as AWS Lambda and Google’s Cloud Functions new and more cost-effective approaches have become viable.</p><p>The positives of these new approaches are many. From simplified architectures to reduced complexity of the implementation. The big wins for us though are cost reduction and the offer of seamless scalability to meet the demands of <em>big news</em> days without having to reserve otherwise unused capacity. To give some context to this last statement, a big news event in the UK or rest of the world causes a significant uplift in the number of people using our services. For us this is the difference between having to deal with several billions of daily events rather than an average one billion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/0*H0NhJHgtJH2rcdM3.jpg" /></figure><p>To deal with such a range in a traditional context you would require to reserve extra capacity to deal seamlessly with these volumes. The introduction of serverless solutions has made a dramatic difference to the way in which we can cope with the highs and lows of world events and moves us along our journey to the goal of real time processing and analytics.</p><h4>All that glisters</h4><p>That said, all that glisters is not gold, and we need to be mindful of the unexpected effects of new ways of doing things. Threats have a tendency of lurking in dark crevices and can pose a threat to not just the performance but also the security of the service. Specifically we need to consider the new techniques by not just looking at the benefits but also by examining the limits of these new methods. Further we need to explore what happens if they are pushed beyond those limits.</p><p>Before pressing on with an example, I must give credit to Dinis Cruz who has played a key role in helping to shape my thinking on issues such as this. I encourage you to read his <a href="http://blog.diniscruz.com/">blog</a> which has a wealth of security related information.</p><p>One such example of where we can run into unexpected issues comes from the temptation to keep throwing serverless functions at problems. Let’s face it we all like to play with new toys, but not thinking carefully about the effects on the system can be detrimental in unexpected ways.</p><p>Consider therefore a system comprising several independent data flows. Each flow is logically separate from the other and has a similar pattern of receiving data from an external data source which leads to triggering serverless functions to post-process the data before making it available for analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IGqGQnU3MksiA5rl." /></figure><p>In isolation each set of processes seems very simple and can be tuned quite nicely to optimize throughput. As more events occur additional functions are invoked by the cloud service to seamlessly scale. All good, well not quite.</p><h4>Threat modelling</h4><p>After detailed security threat modelling of our new architecture we realized that we had introduced a potential new vulnerability into the system.</p><p>Despite us taking care to logically separate out each pipeline so there were no clear dependencies it turns out that a malicious attack on any individual data flow may actually cause <em>all</em> of the services to fail. Not good.</p><p>What we overlooked was the fact that a service such as AWS Lambda has an account wide limit for the maximum number of parallel invocations. As such, an attack on one independent pipeline could in theory exhaust the supply of Lambda functions and therefore not just interrupt the targeted service, but starve <em>all</em> services within that account.</p><p>We have coined the term ‘cross-pipeline dependency’ to describe the threat. Technically this is not really a new class of problem; systems have always had inter-dependencies. With traditional platforms the limits of systems seem so much more apparent. With modern cloud service platforms though it is frighteningly easy to fall into the trap of assuming near limitless capabilities.</p><h4>Threat mitigation</h4><p>So how do you mitigate this risk? In this case there are several strategies.</p><p>Firstly isolating critical services in separate accounts with individual resource limits is one approach, albeit that introduces more management overhead.</p><p>Another option is to try to prevent the problem in the first place by restricting the number of events that can occur by throttling upstream. Possible, but if the upstream system is not in your control this can be awkward.</p><p>Alternatively, and with impeccable timing, a <a href="https://aws.amazon.com/about-aws/whats-new/2017/11/set-concurrency-limits-on-individual-aws-lambda-functions/">new feature of AWS Lambda</a> has recently been released that allows you to throttle concurrent invocations on a per function basis and consequently gives you the ability to isolate your services within a single account.</p><p>In all cases it is essential to understand how the system performs in practice so you can set the right throttling levels.</p><h4>Great monitoring</h4><p>Clearly with any of the above mitigation strategies it is imperative to understand how the system is actually performing and even more importantly to know when the system is misbehaving. Having great monitoring of your service in place is therefore an essential.</p><p>As a data focused team we are big fans of measuring how our services operate. It has become a prerequisite to build an appropriate dashboard as part of each new piece of work.</p><p>Understanding what <em>normal</em> is for our services has become part of the daily routine. As the project has matured, we have learned more and more about the relative importance of the different metrics we measure.</p><p>We have found increasingly we are treating our dashboards not as something set in stone, but rather as living documents which we modify regularly as we are presented with new and more compelling information about what is really important. For us good metrics are the ones that tell us something useful about the system, anything else is just noise and can be removed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0Py7oL0wVOBkZXky." /></figure><p>Through this data driven approach we are able to tell at a glance if the system is operating within normal limits and be alerted when more or less data than expected is being processed.</p><p>With this infrastructure in place it is relatively straightforward to focus on what is important when it comes to serverless functions, namely the number of parallel invocations, the frequency of invocations and the average duration of each invocation. By continuously monitoring these attributes it becomes clear how the system behaves throughout the peaks and troughs of the news year.</p><p>Feeding this information back into the throttling mechanisms mentioned before then helps mitigate the security risk of cross-pipeline contamination while still allowing us to benefit from the simplicity and reliability of the new serverless approach.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9011fe7b46ea" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bbc-product-technology/examining-the-limits-of-serverless-9011fe7b46ea">Examining the limits of serverless</a> was originally published in <a href="https://medium.com/bbc-product-technology">BBC Product &amp; Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>