A colleague recently pointed me to a blog post: On the Futility of Email Regex Validation. For the sake of brevity, I will refer to it as Futility in this article.

Although I admit that the challenge of writing a regex that can successfully identify whether a string conforms to the RFC 5322 definition of an Internet Message header is entertaining, I feel bound to point out that Futility is not a useful guide for the practical programmer.

This is primarily because it conflates RFC 5322 message headers with RFC 5321 address literals, which, in simple language, means that a valid SMTP email address is different from a valid message header in general. It is also because it incites the reader to become preoccupied with edge cases that are theoretically possible from a standard point of view but which—as I will demonstrate—have an infinitesimal probability of occurring “in the wild.”

This article will expand upon both assertions, discuss a few possible use cases for email regex, and conclude with annotated “cookbook” examples of practical email regex.

RFC 5321 supersedes 5322

The universality of SMTP for email transmission means that, as a practical matter, no examination of email address formatting is complete without a close reading of the relevant IETF RFC, which is 5321. Futility depends entirely upon the Internet message header RFC: 5322.

What do I mean by that? As just one example: 5322 allows strings enclosed in parenthesis, even in a domain name. However, this is superseded in RFC 5321, which explicitly states that the domain name portions of email addresses “are restricted for SMTP purposes to consist of a sequence of letters, digits, and hyphens drawn from the ASCII character set.”

In constructing a regular expression, it is hard to overstate the degree to which this constraint simplifies matters, especially regarding determining excessive string length. Further below, the annotated examples will highlight this. It also implies other practical considerations in the validation context that we will explore further.

Please note that the test suite referenced in Futility includes ten tests that contain comments or diacritical or Unicode characters and incorrectly asserts that 8 of them represent valid email addresses.

Mailbox names in the wild

As per both RFCs, the technical name for the email address portion to the left of the “@“ symbol is “mailbox.” Both RFCs allow considerable latitude in what characters are allowable in the mailbox portion. The only significant practical constraint is that quotes or parenthesis must be balanced, which is a real challenge to verify in vanilla regex.

However, real-world mailbox implementations are the measure that the practical programmer should employ. As a rule, the people who pay us frown upon 90% of our billable hours being directed to resolving the 10% of theoretical edge cases that might possibly not even exist at all in real life. I will now demonstrate that in the case of email addresses actually in use, edge cases do not even amount to 1%. To do that, we will consider the dominant email mailbox providers, consumers, and businesses and consider what types of email addresses they permit.

For consumer email, I did some primary research using a list of 5,280,739 email addresses leaked from Twitter accounts. At the time of writing, there were 115 million Twitter accounts, giving us a 99% confidence level with a 0.055% margin of error over the entire population of Twitter, which is sufficiently representative of the general population of all Internet email addresses. Here is what I learned:

• 82% of addresses contained only ASCII alphanumeric characters
• 15% contained ASCII alphanumeric and dots (ASCII periods)
• 3% contained ASCII alphanumeric, dots, and dashes

This adds up to 100% of email addresses. However, this is a rounded 100%. For the trivia lovers out there, I also found:

• 38 addresses with underscores for 0.00072% of the total
• 27 with plus signs for 0.00051%
• 1 address with Unicode characters representing 0.00002% of the total

Conclusion 1: Assuming that email address mailboxes contain only ASCII alphanumeric, dots, and dashes will give you better than 5 9’s accuracy for consumer emails.

For business emails, Datanyze reports that 6,771,269 companies use 91 different email hosting solutions. Fortunately for us, it is a Pareto distribution, and 95.19% of those mailboxes are hosted by just ten service providers.

Gmail for business (34.35% market share)

When creating a mailbox, Google allows only ASCII letters, numbers, and dots. It will, however, accept the plus sign when receiving email.

Microsoft Exchange Online (33.60%)

Allows only ASCII letters, numbers, and dots.

GoDaddy Email Hosting (14.71%)

Uses Microsoft 365 and allows only ASCII letters, numbers, and dots.

Seven additional providers (12.53%)

Not documented.

Unfortunately, this only allows us to be certain of 82% of businesses, and we need to know how many mailboxes that represents. I wanted to find an alternative methodology that would allow us to calculate a confidence level.

The Twitter email addresses included 173,467 domains. Only 400 domains had more than 100 individual email mailboxes represented, and these 400 domains collectively host 96% of Twitter subscribers.

Conclusion 2: The remaining 173,067 domains and 237,592 email addresses represent business email addresses. It is reasonable to take these 237,592 email addresses as representing a population of 1 billion business email addresses. These population and sample sizes come to a 99% confidence level with a 0.25% margin of error, giving us close to 3 9’s when assuming that an email address mailbox contains only ASCII alphanumeric, dots, and dashes.

Use Cases

With practicality foremost in our minds, let us now consider under what circumstances we might need to identify a valid email address programmatically.

New account creation/user signups

A prospective new customer is trying to create an account in this use case. There are two high-level strategies. In the first case, we attempt to verify that the new user's email address is valid and proceed with account creation synchronously.

There are two reasons why you might not want to take this approach. The first is that although you might be able to validate that the email address has a valid form, it might not exist. The other reason is that synchronous is a red flag word at any scale. This should cause the pragmatic programmer to consider instead a fire-and-forget model where a stateless web frontend passes form information to a microservice or API that will asynchronously validate the email by sending a “magic” link. After the user clicks on this link, it will trigger the completion of the account creation process.

Contact forms

In the case of a simple contact form, of the sort often used to download white papers, the potential downside to accepting strings that look like a valid email but are not is that you are lowering the quality of your marketing database by failing to validate if the email address really exists. So once again, the fire-and-forget model is a better option than programmatic validation of the string entered in a form.

Parsing of referrer logs and other large volumes of data.

This is the real use case for programmatic email address identification in general and regex in particular: anonymizing or mining large chunks of unstructured text.

I first encountered this use case assisting a security researcher who needed to upload referrer logs to a fraud detection database. The referrer logs contained email addresses that needed to be anonymized before leaving the company’s walled garden.

These were files with hundreds of millions of lines, and there were hundreds of files a day. “Lines” could be close to a thousand characters long. Iterating through the characters in a line, applying complex tests (e.g., is this the first occurrence of @ in the line, and is it part of a file name such as imagefile@2x.png?) Using loops and standard string functions would have created an impossibly large time complexity. In fact, the in-house development team of this (very large) company had declared it an impossible task, and so my friend turned to me for help.

I wrote the following compiled regex:

search_pattern = re.compile("[a-zA-Z0-9\!\#\$\%\'\*\+\-\^\_\`\{\|\}\~\.]+@|\%40(?!(\w+\.)**(jpg|png))(([\w\-]+\.)+([\w\-]+)))")

and dropped it into the following Python list comprehension:

results = [(re.sub(search_pattern, "redacted@example.com", line)) for line in file]

I cannot remember just how fast it was, but it was fast. My friend could run it on a laptop and be done in minutes. It was accurate. We clocked it at 5 9’s, looking at false negatives and false positives.

My job was made somewhat easy by the fact as referrer logs, they could only contain URL “legal” characters, so I was able to map out any collisions, which I documented in the repo readme. Also, I could have made it even simpler (and faster) if I had performed the email address analysis and learned with the assurance that all that was needed to get to the 5 9’s target was ascii alphanumeric, dots, and dashes. Nonetheless, this is a good example of practicality and scoping the solution to fit the problem. One of the greatest quotes in all of programming lore and history is the great Ward Cunningham’s admonition to take a second to remember exactly what you are trying to accomplish and then ask yourself, “What is the simplest thing that could work?”

In the use case of parsing out (and optionally transforming) an email address from a large amount of unstructured text, this solution was definitely the simplest thing I could think of.

Annotated cookbook

Like I said at the beginning, I do find the idea of building an RFC 5322-compliant regex amusing, so I took it on as a non-practical challenge for my own amusement.

Below I will show you composable chunks of regex to deal with various parts of the RFC 5321 standard and explain how that chunk of regex works. At the end, I will show you what it looks like, all assembled.

The structure of an email address is as follows:

The mailbox

1. Legal characters
2. Single dots (double dots are not legal)
3. Folded White Space (RFC 5322 craziness)
4. (A complete regex solution would also include balanced parenthesis and/or quotes, but I do not have that yet. And very possibly never will.)

The delimiter (@)

The domain name

1. Standard dns parsable domains
2. IPv4 address literals
3. IPv6 address literals
4. IPv6-full
5. IPv6-comp (for compressed)
6. 1st form (2+ 16-bit groups of zero in the middle)
7. 2nd form (2+ 16-bit groups of zero in the beginning)
8. 3rd form (2 16-bit groups of zero at the end)
9. 4th form (8 16-bit groups of zero)
10. IPv6v4-full
11. IPv6v4-comp (compressed)
12. 1st form
13. 2nd form
14. 3rd form
15. 4th form

Now for the regex.

Mailbox

(?<fullAddress>(?<mailbox>([a-zA-Z0-9\+\!\#\$\%\&\'\*\-\/\=\?\+\-\=\^\_\{\}\|\~\`\’]|(?<singleDot>(?<!\.)(?<!^)\.(?!\.))|(?<foldedWhiteSpace>\s?\&\#13\;\&\#10\;.)){1,64})\s?(?<singleAt>(?<!\@)(?<!\-)(?<!\.)\@(?!\@))

First we have (?<mailbox>. This names the capture group for convenience. Inside the capture group are the three regex chunks separated by the alternate match symbol | which means that a character can match any one of the three expressions. Part of writing good (performant and predictable) regex is ensuring the three expressions are mutually exclusive. That is to say that a substring that matches one, will definitely not match either of the other two. To do this, we use specific character classes instead of .

We also use the “explicit” quantifier {1,64} instead of the the “greedy” quantifier * . This is functionally useful because it enforces the specific mailbox length defined in the RFC. It is also good practice in general because careless use of greedy quantifiers can lead to unexpected results.

To the untrained eye, the regex ".+" appears to search for any character (except a quote) that appears between two quotes. However, apply it to the string this "word" is a "quote", and it will return "word" is a "quote" and not "word" and "quote" .

Finally, it is also important to evaluate the use of quantifiers from a performance perspective. When combined with generic tokens such as . instead of specific characters, such as [a-zA-Z], you can create a “runaway” regular expression because of how “backtracking” works. The regex I wrote for my friend all those years ago would have been ten times as fast if I had known this at the time, and with the volumes of data he had to work with, it would have made a significant difference.

Chunk #1: Unconditionally legal characters

[a-zA-Z0–9\+\!\#\$\%\&\'\*\-\/\=\?\+\-\=\^\_\{\}\|\~\`\']

The first alternate match is a character class enclosed in square brackets, which captures all ASCII characters that are legal in an email mailbox except the dot, “folded white space,” the double quote, and the parenthesis. We excluded them because they are only conditionally legal, which is to say that there are rules about how you can use them that have to be validated. We will handle them in the next two alternate matches.

Chunk #2: singleDot

(?<singleDot>(?<!\.)(?<!^)\.(?!\.))

The first such rule concerns the dot (period). In a mailbox, the dot is only allowed as a separator between two strings of legal characters, so two consecutive dots is not legal. To prevent a match if there are two consecutive dots, we use the regex negative lookbehind (?<!\.), which specifies that the next character (a dot) will not match if there is a dot preceding it. Regex look arounds can be chained. There is another negative lookbehind before we get to the dot (?!^), which enforces the rule that the dot cannot be the first character of the mailbox. After the dot, there is a negative lookahead (?!\.), this prevents a dot being matched if it is immediately followed by a dot.

Chunk #3: foldedWhiteSpace

(?<foldedWhiteSpace>\s?\&\#13\;\&\#10\;.)

This is some RFC 5322 nonsense about allowing multi-line headers in messages. I am ready to bet that in the history of email addresses, there has never been anyone who seriously created an address with a multiline mailbox (they might have done it as a joke). But I am playing the 5322 game, so here it is, the string of Unicode characters that creates the Folded White Space as an alternate match.

No chunk provided: Balanced double quotes and parentheses

Both RFCs allow for the use of double quotes as a way of enclosing (or escaping) characters that would normally be illegal. They also allow for enclosing comments in parenthesis so that they will be human-readable but not considered by the mail transfer agent (MTA) when interpreting the address. In both cases, the characters are only legal if balanced. This means there must be a pair of characters, one that opens and one that closes.

I am tempted to write that I have discovered a demonstrationem mirabilem. However, this probably only works posthumously. The truth is that this is non-trivial in vanilla regex. I have an intuition that the recursive nature of “greedy” regex might be exploited to advantage. However, I am unlikely to devote the time necessary to attack this problem for the next few years, and so in the very best tradition, I leave it as an exercise for the reader.

atSign

\s?(?<atSign>(?<!\-)(?<!\.)\@(?!\@))

This delimiter chunk starts off with the special case \s? because according to Futility test cases, a space is legal just before the delimiter, and I am just taking their word for it. The rest of the capture group follows a similar pattern as singleDot, it will not match if preceded by a dot or a dash or if followed immediately by another @.

Domain name

Here, as in the mailbox, we have three alternate matches. And the last of these has nested in it another four alternate matches.

Standard dns parsable

(?<dns>[[:alnum:]]([[:alnum:]\-]{0,63}\.){1,24}[[:alnum:]\-]{1,63}[[:alnum:]])

This will not pass several of the tests in Futility, but as mentioned earlier, it complies strictly with RFC 5321, which has the final word. Note the explicit quantifiers that enforce the character length limits from the RFC.

IPv4

(?<IPv4>\[((?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\])

There is not too much to say about this. This is a well-known and easily available regex for IPv4 addresses.

IPv6

(?<IPv6>(?<IPv6Full>(\[IPv6(\:[0-9a-fA-F]{1,4}){8}\]))|(?<IPv6Comp1>\[IPv6\:((([0-9a-fA-F]{1,4})\:){1,3}(\:([0-9a-fA-F]{1,4})){1,5}?\])|\[IPv6\:((([0-9a-fA-F]{1,4})\:){1,5}(\:([0-9a-fA-F]{1,4})){1,3}?\]))|(?<IPv6Comp2>(\[IPv6\:\:(\:[0-9a-fA-F]{1,4}){1,6}\]))|(?<IPv6Comp3>(\[IPv6\:([0-9a-fA-F]{1,4}\:){1,6}\:\]))|(?<IPv6Comp4>(\[IPv6\:\:\:)\])|(?<IPv6v4Full>(\[IPv6(\:[0-9a-fA-F]{1,4}){6}\:((?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3})(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\])|(?<IPv6v4Comp1>\[IPv6\:((([0-9a-fA-F]{1,4})\:){1,3}(\:([0-9a-fA-F]{1,4})){1,5}?(\:((?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))\])|\[IPv6\:((([0-9a-fA-F]{1,4})\:){1,5}(\:([0-9a-fA-F]{1,4})){1,3}?(\:((?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))\]))|(?<IPv6v4Comp2>(\[IPv6\:\:(\:[0-9a-fA-F]{1,4}){1,5}(\:((?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))\]))|(?<IPv6v4Comp3>(\[IPv6\:([0-9a-fA-F]{1,4}\:){1,5}\:(((?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))\]))|(?<IPv6v4Comp4>(\[IPv6\:\:\:((?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3})(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\]))

I could not find a good regular expression for IPv6 (and IPv6v4) addresses, so I wrote my own, carefully following the Backus/Naur notated rules from RFC 5321. I will not annotate every subgroup of the IPv6 regex, but I have named every subgroup to make it easy to pick apart and see what is going on. Nothing too interesting, really, except maybe the way I combined greedy matching on the “left” side and non-greedy on the “right” in the IUPv6Comp1 capture group.

The full monty

I have saved the final regex, along with the test data from Futility, and enhanced by some IPv6 test cases of my own, to Regex101. I hope you have enjoyed this article and that it is useful and a time saver for many of you.

AZW

Edited August 17, 2023: Reading https://elliotchance.medium.com/its-impossible-to-validate-an-email-address-ae606ad711b4 made me realize my first mailbox group was missing characters (`’-=^) and so I edited my article (and my regex101 testbed) to correct.

Read my book The Chaos Factory to learn how to improve the success rate 
of enterprise development projects!

On the practicality of regex for email address processing was originally published in Better Programming on Medium, where people are continuing the conversation by highlighting and responding to this story.

I recently read Bryan Bates’ well-researched and balanced article on Citizen Development in the context of Agile practices.

He definitely has the right idea about low-code platforms being the path to relieving the pent-up demand for business applications that talent shortages create and that competitive pressures aggravate. He does a good job tracing the history of no-code, and of describing some of the benefits and constraints.

However the no-code space is vast, and there are options that he does not know of. These options dramatically change some of the assumptions his article is based on.

But first: why is there a picture of a clay model of a car in this article?

I believe it is important to explicitly understand the market dynamics that are playing out in corporate IT and to address the underlying causes more proactively than simply using no-code as described in the article and waiting for someone to build a better no-code system one day.

What are the market dynamics and underlying causes? The almost complete failure to apply industrial production techniques to application development.

Very few people would disagree with the assertion that one cannot possibly hope for a successful application development project if the development team consists entirely of new and/or relatively unskilled developers. Everyone accepts as a given that without at least a few master craftspeople guiding the project, there is no hope of succeeding in creating robust and scalable code.

Compare this to automobiles. If you traveled in a car today, you just trusted your life to an exceedingly complex piece of machinery, that could easily kill you if anything went wrong, but was assembled by someone without an engineering degree, and who might not even understand how most of the machinery in the car works. Yet every day a few hundred thousand motor vehicles are completed, 80 million per year, all due to the magic of industrialization.

There are many competing ideas of what industrialization really means, but I will give you a definition that think is better than any other.

Industrialization is the ability to mass produce products while maintaining very high quality, preferably better quality than would be achievable by hand crafted production.

The way industrialization is achieved is not by mere multiplication of brute force, but by embedding human skill into machinery so that unskilled humans can produce some as if they had the necessary skill.

The thing to note about automobile production is that no one tries to automate the skill in the creative process. Car manufacturers still use clay models. They still hire top-notch engineers to create prototypes and calculate the performance parameters. The part that is highly automated is the assembly of mass-produced components manufactured to very high tolerances so that they fit together the right way every time, no matter how unskilled the person doing the assembly is.

The difference between skill and force

Let’s take a very simple example from your kitchen, the difference between a food processor and a rice cooker.

Food processors multiply brute force. In the time it takes you to dice 1 onion, it can dice 10. It can whip egg whites in a 10th of the time and with almost no physical effort on your part. No doubt, they are fun things to have around. But would a top chef ever use one to cut vegetables for their signature dish? Can a food processor help you make 100 radish roses in the time it would take you to cut 10? No.

Now consider the humble rice cooker. Do you need to watch over it? Does it ever burn the rice? Or undercook it? No. Can someone who has never made rice before in their lives make a perfect bowl of rice just by following simple instructions? Yes. Even top sushi chefs — people notorious for their attention to every detail — use rice cookers. This is because the rice cooker has skill embedded into it. Skill as good as the very best human chefs alive. And a completely unskilled person can use one to create rice as good as if it was made by the best chef alive, which means that there is no longer any practical limit to how much rice can be cooked. It is simply a question of buying more rice cookers and hiring the first person that walks in the door.

Isn’t coding different?

If you believe that this does not apply, or impossible to achieve, in the case of software, then you need to take a closer look at compilers. Once upon a time, top-notch programmers would refuse to use compilers because they could write better code by hand. Today only a lunatic would suggest that all machine code should be written by hand. A compiler will do a much better job, every time.

And compilers are used much more than you think. The web page you are reading now is the result of dozens of intermediary compilation steps. C code was compiled to make the Apache web server this is probably running on. Then there is your web browser. Perhaps you are using Google Chrome, it was written in C++, and compiled to assembly language. Chrome includes the V8 JavaScript engine because this webpage has JavaScript in it. V8 interprets the code at first, then compiles it using JIT (Just In Time) compilation to convert the JavaScript into byte code (very similar to assembly language) for a performance advantage. The JavaScript itself was more than likely originally written in TypeScript and then transpiled into JavaScript. Chrome also does similar interpretation/compiling for the HTML and CSS on the webpage. Again, the CSS was almost certainly first written in another language such as LESS, and then transpiled to CSS.

So programming already uses industrialization in a very limited sense. Every day, billions of lines of machine code are mass-produced by compilers that do the job with higher quality, reliability and safety than if the same code was being written by hand.

But this is only the beginning. There is so much oppotunity to apply industrialization to the rest of application development.

What are these new options?

What IT needs to industrialize is to adopt a similar approach. Currently, we take our best engineers, and we tie them up by putting them on our most important projects. What we should do instead is put them on creating components that can be easily (and safely) assembled by less skilled developers.

Large IT departments should be building their own proprietary no-code platforms, dedicating their top talent to maintaining these platforms, which can then be used by junior programmers, business analysts, and tech-savvy business people to assemble apps out of pre-built, pre-tested and debugged, drag and drop components.

Frameworks already exist to accelerate this process so IT departments do not need to reinvent the wheel. Some of them offer pixel-perfect UI controls, the ability to change the way code is generated, and total source code visibility with no “black boxes” or proprietary libraries to worry about.

These tools do away with the restrictions or concerns Bryan Bates lists: licensing, specialized capabilities, performance tweaking, and top-notch UI/UX because the IT department has the ability to build all of this into a no-code platform that they totally control.

Why should IT want to do this?

The benefits to business are clear, but there are also many benefits for the IT department. A few of the most important are:

• Fewer bugs. Once a component is debugged all future uses of it are safe.
• Truly Agile. With an average dev time of 2–3 days for a new component, who needs project management?
• Lower governance costs. With components being for reuse by definition, all dev work is automatically capitalizable, who needs stage/gate?
• More satisfying work. What senior engineer enjoys moving a button two pixels to the left? Designing for reuse is inherently more interesting than pushing out “compromise code” to “just get it done” by some deadline.

In summary

The real new face of Agility is not bypassing the IT department. Businesses tried that in the 90s and it didn't work. The real new face of Agility is fundamentally changing the way the IT department works: getting it out of the business of projects and into the business of platforms, no-code platforms for internal use.

Adam Zachary Wasserman is the author of The Chaos Factory: The inside story of IT failure, available at www.chaosfactorythebook.com

AI means a lot of things to a lot of people. Usually what it means is not very well thought out. It is felt, it is intuited. It is either adored, worshipped or deemed blasphemous, profane, to be feared.

In this article, I explore what society at large really means by artificial intelligence as opposed to what researchers or computer scientists mean. I want to clarify for the non-technical audience what can realistically be expected from AI, and more importantly, what is just unrealistic pie-in-the-sky speculation.

4 Books that Challenge Your View on Artificial Intelligence and Society | Data Driven Investor

I am worried that blind fear — or in some cases worship — of AI is being used to manipulate society. Politicians, business people, and media personalities craft narratives around AI that stir up deep emotions that they use to their advantage. Meanwhile truth is only to be found in dense technical literature that is out of reach for the ordinary person.

What do we mean by intelligence?

Is intelligence a purely human characteristic? Most people will consider some dogs to be more intelligent than others. Or dogs to be more intelligent than Guinea pigs, so clearly intelligence is something that an animal can have.

If a dog can have intelligence, can a bird? How about an earthworm, or a plant? Where do we draw the line? There are many definitions of intelligence, but one that I like (from Wikipedia) is:

“the ability to perceive or infer information, and to retain it as knowledge to be applied towards adaptive behaviors within an environment or context“.”

Seems reasonable? Some people argue that by this definition even plants can be intelligent So why not computers?

This is the intelligence to which AI researchers generally refer. Yet the reality is that when average people speak of or think about AI, they are not thinking about plants or animals. Most people would not get too excited one way or another about the idea that a computer might be able to operate at the level of a plant, a guinea pig, or even a dog.

Equally, people do not really care if a machine can do what its creator intends it to do; what it is programmed to do. Handguns are created to kill people, and they do, yet no one worries that handguns will develop consciousness and kill all the humans — as they are “programmed” to do.

If we are going to be honest, we must admit that what the average person means by “artificial intelligence” is really “artificially like a human”, and what they worry about (or celebrate) is the possibility that AIs could spontaneously come up with motivations of their own.

The human factor

Motivation is key; human motivation has a different quality than that of all other animals, and this difference is arguably what makes humans unique.

The great mathematician and computer pioneer Alan Turing had suggested that instead of asking what it is to be human — which he considered a practical impossibility — we could simply say: “if it walks like a duck and quacks like a duck.” .

Turing’s proposal was that if a computer program could pass a written interview without the interviewer realizing that they were interacting with a computer, then the computer could be said to be for all practical purposes an artificial intelligence that was human in character. However this does not really answer the question of whether a computer program could make the same kind of complex decisions in real time that a human does, whether it can have motivations that were not preprogrammed into it.

Above all, a written interview encompasses only a very small part of human experience. We live in a world of action, and many actions have very real and immediate consequences. The decision of whether or not to walk down a dark alley is a complex one, and many of the factors that influence our decision are unknown. Every day we make decisions without knowing all the data, and the fact that the human race survives and thrives is indisputable proof that on the whole we are amazingly successful at doing this — if one defines success as survival.

Value making machines

How we pull this trick off is a matter of debate. But most serious scholars would agree that it is almost unavoidably the case that the human propensity for assigning value is at the heart of things.

Humans are “value making machines” and we do it at a speed that no computer network could hope to match. If you hear an unexpected sound, you will have a reflex action in 170 milliseconds. To put this in perspective: it is 30 milliseconds faster than Google’s recommended time for the first byte of data to hit your browser after you click on a link, and Google recommends a 500 millisecond time for the page to finish loading after the first byte.

Loading a web page is a trivial action for a computer, and still it takes more than twice as long for the computer to do this as it does for your brain to figure out whether an unexpected noise is threatening (a snapping twig or a metallic sound) or delightful (ice cream truck bells or a child’s laughter) — which by the way: it does instantly and with virtually no data at all.

Identifying something as threatening or delightful is what we mean by assigning value, and it is something that we do instinctively, automatically, “without thinking”. Yet without doing it, we would be unable to “think” as we know it.

Say you are thirsty; you must solve the problem of what to drink, then you must solve the problem of getting it.

Suppose your choice of what to drink is between pond water and fresh sparkling well-water. You are thinking right now that obviously we want the well-water. But what if you are an escaped prisoner, and the pond water is safely out of site in the forest, and the well is in a town square where you might be seen? Now you are thinking pond water. You are motived by thirst, but you are more highly motivated to remain free. This is because the value you place on staying free is much higher than the value you place on the freshness of your water.

Human motivation is completely dependent upon values. A thirsty dog will simply drink from the first available water it finds, because its motivation is survival, unaffected by values.

Let’s consider an example unrelated to survival: would you push a button that would definitely kill one person, or would you refuse to push it even if it meant that 10 people might die. Your thinking about this question is not in any related to your own physical survival, yet it has a moral urgency that few people could deny. Your thinking will be entirely directed by the values that you assign while evaluating your options.

Where do these values come from?

The truth is that we don’t know.

For some it is a question faith. Sincerely religious people belive that our values are a reflection of God’s will.The few people who truly believe in evolutionary theory and all of its implications would say that our values are that/those which allow(s) us to survive and which consequently perpetuate itself/themselves.

Almost everyone else holds the view that values are a self-evident truth. That we have these values because we all have them, that it is just obviously so. As Sam Harris puts it:

“When we really believe that something is factually true or morally good, we also believe that another person, similarly placed, should share our conviction”.

As far as a claim to understanding goes, this is weak tea. It provides no useful grounding for explaining why some values are culturally dependant, while others seem to be universal or nearly so. It does not explain the origin of values, and so for the most part, the religious and the evolutionists notwithstanding, we have no useful explanation of how values work.

It is in values that we find the consciousness gap. In the above example you are motivated by thirst, a biological factor. An AI, equipped with the appropriate sensors, could also have that sort of motivation, for example the need to charge a battery that is running low.

But what motivates you to take a picture of something beautiful to share with your loved ones, or to argue politics with your friends? What motivates you to watch a scary movie or to learn a sport? What about the button pushing example: on what basis could a computer make a decision like that without a human first providing values such as one life is worth less than ten, or that killing is wrong no matter what the circumstances. Our morality, our ability to assign value in the blink of an eye, has evolved over millions of years. How would a computer — in a single generation mind you, because they have no natural reproductive mechanism — arrive at a moral grounding on its own, without a blueprint first being provided by a human?

Would we even want it to? Human history suggests that tens of thousands of generations were required before we arrived at what we now consider civilized behaviour. Would we really want a race of AIs to make the same slow and painful march towards civilization?

Sure we could give them a “jump-start” but then we are back to the consciousness gap: if we program in the jump start, basically programming in the values of some human, can the AI be said to truly have human-like consciousness, capable of spontaneous motivations? As you may have guessed; I believe the answer is probably not.

Wherein the danger

This is not to say that AI is totally and completely uninvolved with anything dangerous or any area for concern. The use of AI is no more and no less subject to the law of unintended consequences than any other field of human endeavour. We can be absolutely certain that the use of AI for things such as curating the content of your social media feed will lead directly to unforeseen results that many people do not like.

This is not due to to any inherent quality of AI. It is the nature of the world in which we live. It is inherent to human decision making, and whether that decision is to import Cane Toads into Australia (to control pests), or to use AI to make automated stock purchases, the overwhelming probability is that catastrophe will ensue.

Party tricks

We cannot imagine trying to present the “button problem” to a dog. We have no reason to believe that a dog would have any moral framework within which to make the problem relevant, no reason to believe that dog would care one way or another.

We have no more reason to believe that we could present this problem to an AI than to a dog. We have no evidence at all that any AI anywhere has a moral framework guided or informed by the sort of spontaneous value-making that marks humans. It is irrelevant that computers can calculate faster than a human or can predict certain classes of problems faster or better. Winning a chess game against a human is a landmark in programming and computer science, but it is of little consequence in the real world. The ability to calculate a very large number of permutations within a very strict, and very limited, set of rules is in no way indicative of general intelligence or the ability for a computer to develop consciousness.

Computers being able to triage and diagnose medical conditions as well as — or possibly better than — human doctors is also not quite as impressive as it sounds. To be sure it is very helpful to automate checklists and decision trees; in a medical emergency a computer is faster than cracking a book. And it is true that without these checklists, humans are prone to all sorts of perceptual and cognitive biases, but you would be very wrong if you assumed that the AI that can do triage can also decide if the shadow in front of your car is a cardboard box or a child on a tricycle.

Sensationalism sells. The sky is always falling, and the failure of last week’s prophecies of Armageddon (the “Y2K bug” or New York City’s West Side Highway under water by 2019) never seem to slake people’s thirst for this week’s prediction of impending doom. The same can be said for sensationalist idealism. The repeated failure of utopian ideologies to actually produce the predicted earthly paradise seems in no way to hinder the convictions of the true believers that this time. ..

AI is nowhere near as exciting, or mysterious, or dangerous, or magnificent, as it promoters and detractors would have you think. It has become for the most part a branch of the mathematics of probability, glorified actuarial work. It has all the sex appeal of an otaku or anorak lecturing on their favorite subject. It is a one trick pony.

In short: it is nothing to worry about or get excited about. We return you to our regularly scheduled program.

Read my book about fixing IT

Hi! This is Ankita, community manager of DDI (https://medium.com/datadriveninvestor), one of Medium’s fastest-growing publications. We are a community of over 25,000 industry experts in Tech, Finance and Entrepreneurship. Are you interested in publishing with us? here’s how you can submit an article for review:

AI and the Consciousness Gap was originally published in DataDrivenInvestor on Medium, where people are continuing the conversation by highlighting and responding to this story.

Some data suggests not…

Current common wisdom has it that everyone will be losing their job to automation real soon now. AI is going to eat the world, and the US needs universal basic income because all the truck drivers are all going to lose all the jobs this week, or next week at the latest.

I have my doubts.

I suspect that many readers will be difficult to convince, so before I get to my argument I’ll just put forward a few facts.

A Few Facts About Employment

In its 2017 executive summary, the International Federation of Robotics informs us that 35% of the world’s industrial robots are purchased by the automotive industry putting it ahead of even the semiconductor and electronics industries. Auto manufacturers’ use of robots has been steadily increasing since 2010. Let’s take a look at what the US Department of Labor has to say about employment in that industry since 2010…

Note that these numbers are for jobs in motor vehicles and parts manufacturing, the jobs that would be most affected by automation. It does not include the jobs that would not be affected by automation: wholesale, retail, or repair and maintenance jobs. In fact, the 2017–2018 State of the Region report by the Detroit Regional Chamber notes that “mechanical and industrial engineers saw the largest growth”.

On the flip side, In the Chinese factory city of Kunshan, Foxconn claims it has reduced its workforce by 60% through automation.

A Few Facts About Wages

Now let’s turn to South Korea. In its Executive Summary World Robotics, the International Federation of Robotics observes that:

“The countries with the highest robot densities by far are the Republic of Korea (710 robots per 10,000 employees) and Singapore (658 robots). The Republic of Korea has had by far the highest robot density in the manufacturing industry since 2010.”

What impact is all this automation having on wages? Has it turned South Korea into a worker’s dystopia with starvation-level wages? It turns out that South Korea’s minimum wage is 8,350 won ($ 7.37 USD) per hour, 12¢ higher than in the USA.

In Singapore, the year over year growth in median income has outstripped western countries (showing US, UK, Germany and Canada for comparison) by a considerable margin since 2010.

Some Old(er) Facts

Peter Lindert and Jeffrey Williamson produced a rigorous analysis of Earnings and Cost of Living indices in the United Kingdom pre and post Industrial revolution. Their conclusion is that both wages and standard of living increased dramatically post Industrial Revolution. People made more money, and the essentials of life cost less.

Even their harshest critics, such as Charles Feinstein, do not dispute the increase, but merely the size or rate of the increase.

Clark Nardinelli, author of Child Labor and the Industrial Revolution, and professor at Clemson University writes:

“The standard-of-living debate today is not about whether the industrial revolution made people better off, but about when”

Interpreting the Facts

The numbers from the US auto industry are unambiguous. Increased automation simply has not resulted in fewer jobs. With the data I have, it is impossible to claim with certainty that automation has increased jobs (I cannot establish a causal link), but since jobs have increased, it is clear that nothing, not automation or anything else, has resulted in job losses.

As for the Foxconn claim, I would remind the reader that this is a PR release by a commercial entity, not statistics from a government agency. What the government stats do show (in the same article from South China Morning Post) is that the number of laptops produced in Kunshan is down to 51 million from a high of 120 million due to declining sales. It is a reasonable possibility that Foxconn may have simply reduced its workforce as a result of lower sales volumes. Of course, framing it as the result of modernization makes for the better press, so I remain skeptical that a 60% labour force reduction is due to automation when a 60% decrease in demand is a much more likely explanation.

I tend to favor the analysis that attributes increased wages and standard of living in Industrial Revolution Era, England to a virtuous circle of automation leading to affordable products of a standard quality leading to increased production, leading to higher wages and increased labor force which in turn leads to greater disposable income and higher demand for the aforementioned affordable quality products.

This latter fact is undeniable. From the first truly automated production — the manufacturing of naval block and tackle in the Portsmouth Naval Yard using machines designed by Samuel Bentham, Henry Maudslay, and Marc Isambard Brunel — automated manufacturing has led directly to lower prices, fewer defects, and better quality-to-price ratios.

I am not trying prove conclusively that automation will always increase employment and improve the standard of living using only two examples.

I am making the point that there reasons to question the current narrative and view it critically. These examples, and many others, are proof that automation does not always create job loss. As to whether it sometimes does, that is too extensive a question to handle in one small article.

Enough About Manufacturing

So far I have focused on manufacturing. So what about service jobs? Wait staff and drivers?

It is instructive to note that since the introduction of the Airbus 320, modern aeroplanes are entirely capable of automated take-off and landing, and preventing certain types of accidents due to pilot error. Yet the presence of a human pilot is considered essential. And in certain documented cases, the automation causes accidents instead of preventing them.

A commercial airliner takes off and lands in a highly controlled environment. The runways are restricted, only one aeroplane at a time uses them, there are no children, dogs, or other drivers to worry about. Nevertheless “Automatic landings probably account for less than 1% of all landings on commercial flights” according to FlightDeckFriend.com (a website built by Pilots for Pilots). This is the best that can be achieved for machines that cost billions of dollars to design and certify, and that sell for hundreds of millions of dollars.

There are roughly 3.5 million commercial truck drivers in the US. They drive in heavy traffic surrounded by unpredictable drivers, and innocent pedestrians. Not only that, but a tractor-trailer cab sells for only $100,000, not $200,000,000. How are we to imagine millions of robotic trucks safely navigating infinitely more complex traffic situations than a runway, using robotic systems that will have to cost a thousand times less than the ones on a commercial jet? I hope the reader will understand my skepticism.

Fred LeFranc, the founder and CEO of Results Thru Strategy, which advises hospitality firms says:

“We are at the place where the most repetitive tasks can be performed by a robot, and they don’t call in sick, they don’t talk back.”

As they used to put it rather quaintly in my hometown of New York: It is to laugh.

Robots = mechanics + computer + programming. How could that possibly go wrong? My computer doesn’t call in sick. Instead it crashes when I am on the last paragraph of a 1000 word essay, and then when I reboot it, my file is corrupt and unusable. My washing machine does not talk back. Instead it just stops working until I call a service tech — who takes two weeks to show up and then does not have the necessary part with him when he does. Yeah, robot employees would be just delightful. At least when human calls in sick you can guess when they will be better, and even if they talk back, at least they on the job… which is more than I can say for my washing machine.

The McKinsey Global Institute claims that fewer than 5% of occupations could be completely automated (A Future That Works: Automation, Employment, and Productivity)

Obviously I am inclined to believe them, and I think you should too. Politicians are attempting to manipulate the public through fear, and that is not a good thing. Social engineering experiments produce unintended consequences far more frequently than they work out exactly as planned.

I encourage the reader to take heart in the fact that despite the non-stop predictions of the end of the world, civilization, and humankind, the world keeps turning and the overall condition of humanity is increasingly better over time. Some individuals will definitely lose their jobs to automation, and it will be difficult for them, but civilization is not coming to end — not for this reason in any case — and we should not overreact. When someone tries to convince you that it is, I believe you should demand the same kinds of facts and analysis that I have provided here instead of just taking them on faith.

I suggest that the most likely reaction of society and the job market in the face of this latest round of automation will be the same as it has been for millennia: better goods, more affordability, increased overall employment, and a higher standard of living. If you do not believe me; ask yourself if you would rather be living in pre-industrial England with a life expectancy of 40 years and one out ten children dying before they turned one year old.

My book The Chaos Factory: The inside story of corporate IT failure is available here at Amazon.

Is Automation Going to Eat Your Job? was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.

CPUs speak only one language; they don’t care what language you write in

Programmers tend to make a big deal over the supposed difference between compiled languages and interpreted ones. Or dynamic languages vs. statically typed languages.

The conventional wisdom goes like this: A compiled language is stored in machine code and is executed by the CPU with no delay, an interpreted language is converted to machine language one instruction at a time, which makes it run slowly. Dynamic languages are slower because of the overhead of figuring out type at runtime.

The reality is nothing like this

A long long time ago, when programmers wrote to the bare metal and compilers were unsophisticated, this simplistic view may have been somewhat true. But there are two things that make this this patently false today.

The first thing is that all popular programming languages these days — Java, JavaScript, Python, Scala, Kotlin, .NET, Ruby — all run on virtual machines. The virtual machines themselves are written in a variety of languages*.

The second thing is that VMs make it easier to easier to observe the program’s execution, which in turn makes JIT (Just in Time) compilation possible.

So how about a real example that illustrates why this matters: Java.

Java is compiled. Or is it? Well… sort of not really. Yes there is a compiler that takes your source and creates Java bytecode, but do you know what the JVM does with that bytecode as soon as it gets it?

First it build a tree. It disassembles the Java bytecode and builds a semantic tree so that it can figure out what the source code was trying to do. In order to accomplish this it must undo all of the snazzy optimizations that the compiler has so carefully figured out. It throws away the compiled code.

That sounds crazy! So why does it do this? The answer to that question is the key to understanding the title of this article.

The best way to understand code is to watch it running

This applies to humans, but it applies just as well to compilers. The reason why the modern JVM undoes the compilation and the optimizations is that “conventionally” compiled Java bytecode runs too slowly on the JVM. To attain the speed of execution for which Java is known these days, the JIT has to throw away the code that was statically compiled (and statically optimized) and “watch” the code running in JVM, and make optimizations based on the code’s actual behaviour at run time**.

Don’t go around saying things like “[insert language name here]is too slow because it is interpreted” because that is simply not true. Languages are just syntax and can have many implementations (for example there are several implementations of Java). You could say “[insert language name here] is slow because the interpreter/VM doesn’t have JIT” and that might be true.

Languages are not fast or slow

C++ is not a faster language. C++ runs fast simply because more hours have been spent on the compiler optimizations than for any other language, so of course they are better. It is worth noting that programs compiled using PGC++ and Clang are regularly 2 or 3 times faster than the same source code compiled using the AOCC compiler. This is proof that it is the compiler and its optimizations — not the language itself — that dramatically affect execution performance.

Java is generally considered next fastest, and that is because it has had more hours invested in its JIT compiler than anything except C/C++.

Framework or Language redux

But it is not all down to the compiler. I have already written about the dangers of unexamined libraries and frameworks. The article could also have been titled Syntax or Library: Do you know which one you are using? I asked a trusted friend to review this article, and brought up the very good point that memory access patterns are the big performance culprit overall. He made the point that C programs benefit from the fact that…

“C only has primitive arrays and structs (but it’s very liberal with pointers so you can pretty much wrangle anything into these primitive structures with enough patience). Working with hashmaps or trees can be very painful, so you avoid those. This has the advantage of nudging you towards data layouts that make effective use of memory and caches. There’s also no closures and dynamically created functions, so your code and data mostly stay in predictable places in memory.

Then look at something like Ruby, where the language design encourages maximum dynamism. Everything tends to be behind a hashmap lookup, even calling a fixed unchanging method on an object (because someone might have replaced that method with another since you last called it). Anything can get moved or wrapped in yet another anonymous closure. This creates memory access patterns with a scattered “mosaic” of little objects scattered all over the place, and the code spends its time hunting each piece of the puzzle from memory which then points to the next one.

In short, C encourages very predictable memory layouts, while Ruby encourages very unpredictable memory layouts. An optimizing compiler can’t do much to fix this.”

I had to agree. Which led me to articulate my point thusly: Programmers who do not understand where syntax stops and libraries begin are doomed to write programs whose execution they do not really understand.

My belief is that it is more difficult to write a truly awful C program because (if it runs at all) it would be too much work to manually reproduce the memory chaos that Ruby so casually produces.

We then had an interesting chat about how a certain large tech company created a “cleaned-up PHP++”. He has some interesting things to say, maybe he will write an article about that.

Thank you for your help Pauli.

So an implicit part of my contention is that modern programming languages lower the bar so that many programmers do not think about about basic computer science (memory structures and computational complexity), and therefore have no basis upon which to understand how their programs will execute.

The other part of my contention is that any Turing complete language could run about as quickly as any other when considered from a pure syntax perspective. For example I believe that it would absolutely possible to create a high performance implementation of Ruby on let’s say the JVM. I readily acknowledge that most current Ruby code would break on that system, but that is as a result of the programming choices made in the standard libraries, not as a fundamental constraint of the language (syntax) itself.

I have say, (as a self-admitted cargo cult programmer) that it is definitely possible that I just don’t understand Ruby syntax and/or the Church–Turing thesis.

CPUs (or VMs) speak only one language

Ruby and its shotgun approach to memory management notwithstanding; any programming language that would have as many hours invested in optimizing its compilation as Java or C, would run just as fast as Java or C and this is because CPUs (and VMs) speak only one language: machine language. No matter what language you write in, sooner or later it gets compiled to machine language and the things that affect performance are how fundamental computer science principles are implemented in the standard libraries, and how effective the compilation is, not when it happens.

The moral of the story is that programmers should spend less time crushing out on languages and more time understanding how they work under the hood.

I will finish with a quote from the great Sussmann:

“…computers are never large enough or fast enough. Each breakthrough in hardware technology leads to more massive programming enterprises, new organizational principles, and an enrichment of abstract models. Every reader should ask himself periodically ‘‘Toward what end, toward what end?’’ — but do not ask it too often lest you pass up the fun of programming for the constipation of bittersweet philosophy.”

* The original Sun JVM was written in C. The original IBM JVM was written in SmallTalk. Other JVMs have been written in C++.
The Java API (class libraries without which most Java programmers would unable to make even a simple application) are written in Java itself for the most part.

** It is worth noting that this happens again when the machine language of the VM “hits” the hardware CPU, which immediately takes a group of instructions, breaks them apart, looks for patterns it can optimize, and then rebuilds it all so it can pipeline microcode.

The invisible art of programming

No art form is less visible than that of the programmer for whom “humble labour and profound obscurity” are preconditions of the medium.

But first a word on art: Programming is is like writing music. There is no one true way to write a melody, but there are plenty of wrong ones. There are many combinations of notes that will be absolutely displeasing to all who listen to them, but amongst the many combinations are a few that will please almost anyone.

Similarly, there is not one true way to write a program.

A programmer’s coding style is unique and identifiable. Programmers are able to recognize the author of a piece of code once they know their style.

Code can be funny. I have spotted more than one joke in reading other people’s code, and I have left a few of my own when writing my code. But I will never know if anyone saw them and laughed.

Authors of poetry and prose can hope that once published they will be treated to some kind of feedback. They might even achieve fame and recognition. When I write these articles, I get comments (and thank you for that). Yet code by its nature is invisible. Code running on a server is inaccessible by design. Complied code even more so. Open source code is available for all to see, but disconnected from an operating environment, detached, intangible, inscrutable to most people, even other programmers.

No programmer has ever been awarded a medal or interviewed on television because of the elegance and expressiveness of their code, but this does not mean that code cannot be elegant or expressive.

Code is so expressive that it creates whole worlds, whole universes. Self-contained universes, Hermetic universes, whose prisca theologia is the Church-Turing thesis.
Code is Logos, the Word that brings order out of chaos.

It is precisely this world-creating power that makes programming an art that is so difficult to master.

Lesser programmers might create a portion of the world, but the master programmer must first bring the world into being with its own rules, logic, physics, and all other manner of internal consistency. These days we tend to call these worlds frameworks, but languages, compilers, and even some programs (here’s looking at you Stallman) are all examples of universes created by master programmers.

Heavy is the head that wears the crown. It is hard work being the demiurge, making the rules.

Stacked one inside of the other, these worlds are like Russian dolls, each layer revealing an ever deep level. Peel away the framework, and you find the platform class libraries. Step through the class library inheritances and eventually you arrive at the language. Underneath the language you find the language, in all its glory, which covers up the machine instructions. And the machine instructions depend upon the particular micro-processor instructions, which are millions of tiny transistors with connections to each other just a few atoms wide.

We even have computers which glean information from the spin of a quark.

No medium is so pliable, so infinite in its possibilities as that of the programmer. Stone cracks and cannot be made whole. Paint is limited by its physical properties too, water-colours are unforgiving, oil darkens with age. Photography limits what can be put in frame and how much dynamic range can be captured. No such limits with code.

“The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds his castles in the air, from air, creating by exertion of the imagination. Few media of creation are so flexible, so easy to polish and rework, so readily capable of realizing grand conceptual structures” — Frederick P . Brooks Jr.

Sic mundus creatus est was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.

I have been working with an Agile development shop of late, and there is something about it that I definitely do not like. I do not enjoy being “educated” by well-meaning but nonetheless smug Scrum Masters and Product Owners who assume that anyone in corporate IT has not the slightest clue. These are not ad hoc lectures, they are formal presentations; Powerpoint decks approved by the development shop’s executives. The implication is that the company assumes we have never even heard of Agile.

The reality is that many people in corporate IT are as familiar with Scrum (because that is the only kind of Agility anyone seems aware of these days) as they are. And some, like myself, have significantly more experience, having worked with other Agile practices such as XP, CRC cards, Crystal, et al. I started Agile practices (XP in 1996) before they were officially “Agile”, and for that matter before some of these Scrum Masters were born.

This attitude of smugness is nothing new, I have been more or less continually confronted by it since late 2004 when Ken Schwaber published his paper on the “SCRUM Development Process” which triggered a new round of holy wars in the field of software development.

Or did it…?

This holy war was very one one-sided. I haven’t seen any significant opposition to the basic ideas of Scrum. What I have seen is two things:

1. Annoyance with the holy fervor and maddeningly preachy attitude of the Scrum faithful, and
2. The requirement that anyone coding this way still conform to PPM and Stage-Gate governance.

It is this last point that seems to be radically misunderstood by the Scrum contingent. They seem to believe that CFOs, Business Sponsors, and Project Managers actually give a rat’s ass about how they write code.

Believe me: nothing could be farther from the truth. I can easily name a dozen common practices that would immediately cease if corporate IT management knew the least thing about (or truly cared about) how the sausage gets made.

But what is Agile, really?

I consider the concerns of Agile to be orthogonal to the concerns of governance strategies such as PPM and Stage-Gate, not in direct conflict to it.

I have been careful to use the word “practices” rather than “methodology”. As several of the original signatories of the Manifesto for Agile Software Development (Dave Thomas, or Bob Martin or Martin Fowler or Ron Jeffries) are at great pains to point out: the original impetus for Agile practices were concerns about the craftsmanship or practice of software development, not the process of software development. The manifesto itself says: “Individuals and interactions over processes and tools”.

Yet most people “doing Agile” today have this idea that they are the sworn enemies of the oh-so-evil waterfall process. It stands to reason that they also believe that Agile is a process.

And what does Waterfall really mean?

What most Agile developers think of as “waterfall” is actually a combination of two very widely adopted models of governance, the first being Project Portfolio Management and the second being the Stage-Gate Model.

The first and most important thing to understand about these two models is their reason for existing. They exist to protect corporate executives. They do this by demonstrating good-faith best-efforts to comply with two important Securities and Exchange Commission (SEC) regulatory requirements.

The first of these is the American Institute of CPA’s Statement of Position (SOP) 98-1, which was passed as a result of CFOs seeking SEC approval to account for software development costs as capital expense (CapEx) because that would allow them to report higher earnings and higher company valuations.

The second is the Sarbanes–Oxley Act of 2002 (SOX), which had been passed in reaction to a number of major corporate and accounting scandals, notably Enron and WorldCom (who manipulated earnings, underreporting line costs by capitalizing rather than expensing).

To make a long story short, corporations are allowed to count software development as CapEx, but the officers of the company will be criminally liable if audits discover that this has been done improperly.

PPM and Stage-Gate were promoted by Gartner and the Project Management Institute respectively, as governance that would protect corporate officers from criminal liability as regards the capitalization of software development costs.

So what?

So… the concerns of Agile and the concerns of “waterfall” are quite separate, there is no reason why they would not co-exist peacefully. All that is required for a Scrum team to comply with almost any modern IP project governance (PPM and Stage-Gate) are six artifacts:

1. a Project Charter which is basically just a scope document and is roughly equal to the Product Backlog.
2. a WBS (work breakdown structure) which can usually be satisfied with the results of Epic-level Planning Poker combined with a rough Sprint Planning
3. weekly Status/Progress Report is the weekly burndown report or any other form of progress report
4. weekly Open Risks which may not be a standard Scrum artifact but there certainly is nothing about this that is problematic in Scrum
5. weekly Open Issues which are a direct result of Sprint retros and therefore standard in Scrum
6. Lessons Learned which are also a direct result of Sprint retros and therefore standard in Scrum

There is nothing here that is particularly antithetical to Scrum.

Traditional Stage-Gate separates “detailed design” from “development” but it really is no big deal from the perspective of the main concern: both are admissible as CapEx, so it is materially* inconsequential whether or not they are performed sequentially, concurrently or iteratively.

* I use the the word “material” in its accounting sense

Conclusion

I would urge Agilites to stop positioning yourselves as the “sworn enemy” of waterfall, you will not win. Corporate executives do not want to go to jail, and there is nothing in Agile or Scrum to protect them from that. PPM and Stage-Gate are not going away, learn to live with them.

Consider dialling back on the pre-project Scrum evangelization and focus on making things as easy and as transparent as possible for your enterprise customers, particularly the Project Management Office (PMO).

The most evangelical thing you could possibly do is simply deliver a functioning software product on-time and under-budget without creating any waves in the PMO. Just do that, and your customer will gladly be all Agile, all the time.

Actions speak louder than words.

To read more of me, get my book The Chaos Factory which tells the inside story of corporate IT failure, and provides the real recipe to fix it.

Agile: the greatest struggle that never existed was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.

The man in the back without a jacket is Mel Kaye, about whom Ed Nather wrote his timeless classic of programmer lore, TheStory of Mel.

The year is 1960 and although a wealth of information exists about the hardware he programmed on, nothing remains of his programming other than Ed Nather’s delightful tale.

Hackers and hobbyists

For the first twenty years of programming history, most programmers in the field, the ones writing the programs that got used in real life, were self-taught. There were no schools for programmers, no formal practices, nobody of knowledge, no discipline to speak of. Programmers either learned from a mentor or pursued their own individual instincts and intuitions.

The profession of electrical engineering was well organized, well documented, and acknowledged as being very significant. Computer hardware design was a recognized science and a respectable career for electrical engineers.

Meanwhile programming was a sideline that people with real careers (such as math or physics) engaged in out of personal interest. This eventually changed with the emergence of the discipline of computer science, which was focused on programming and software as the engine of innovation. Software would eventually become more important than hardware, but during these first critical years of computing history, only hardware engineering was respected, and history reflects this.

Lest the reader think that I am overstating the case of how little people thought of pro­gram­ming in those days, allow me to provide the following anecdote¹: when Edsger Dijkstra married Maria C. Debets in 1957, he was required as a part of the marriage rites to state his profession. His declaration that he was a programmer was not accepted by the authorities, because there was no such profession at that time in The Netherlands.

The typical programmer was self-taught and highly internally motivated. Apart from a small number of young prodigies, a programmer was more likely than not already highly educated, often with a Masters or PhD which was how they came to be anywhere near a computer to begin with. They tended to be a verysmart bunch.

There were no schools or courses for programmers, and when computers cost millions of dollars there were no casual programmers. They were some of the brightest lights of humanity simply by virtue of how difficult it was to become a programmer at that time.

Yet the fruit of those minds is gone forever, they labored in obscurity.

For every Hopper or Dijkstra or Knuth or Wirth that we know a little about, there were a thousand Mel Kayes, creating works of pure genius, of elegance and rare beauty. All these works have been lost forever as the magnetic tapes and punch cards that preserved the deepest thoughts of this hidden generation became obsolete and were binned, unceremoniously and out of sight.

[1]E. W. Dijkstra Archive. & James, Mike (1 May 2013). “Edsger Dijkstra — The Poetry of Programming”.

Hidden Figurers was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you want 3 or 4 very different opinions on this, just ask 1 or 2 programmers. On this particular subject, some of my colleagues feel so strongly that they might even be able to flame themselves.

And the meanings of words….

The dictionary

If we were to stick to dictionary definitions there is no question. Art is defined by the OED as “The expression or application of human creative skill and imagination…”, and Webster’s has this to say: “skill acquired by experience, study, or observation”.

Programming is clearly a human creative skill acquired by experience, so that is that. Bam!

But lets take a more nuanced approach.

Programming vs. software application development or production

The two are not the same thing. All software applications involve some programming (surprisingly little usually), but not all programming is for the production of a software applications.

Definitions of what a “software application” is vary a little, but generally agree that is a special kind of program; a program that is designed for the end user. This means that it has certain defining characteristics: a user interface, it usually allows the user to enter data, it often has logic, and it almost always provides the user some way to interact and trigger the execution of functions.

Not all programs look like this. There are about a million lines of code in a car that do not do any of these things. And more code that makes the traffic lights in your neighbourhood work, and some more in your flat panel TV set that does quite a bit of image processing — without any interaction from you.

Application development…

…involves very little actual programming.

Alexander Stepanov, the primary designer and implementer of the C++ Standard Template Library, relays to us that Scott Byer, the architect of Adobe Photoshop, estimates that 90% of developer efforts are dedicated to “glue” and housekeeping tasks such as memory management, scripting, UI management, file I/O, and color management. 10% is spent on “substance”. He provides the following estimates of how much time is spent on what he deems “substance” for a variety of application types:

• Word processing — 3%
• Presentation App — 1%
• Databases — 10%
• Technical/CAD — 30%
• Operating System — 1%
• Enterprise Application Software — 1%

From personal experience, these numbers sound just about right to me.

Now about programming

Programming is like writing music; there is no one true way to write a melody, but there are plenty of wrong ones. There are many combinations of notes that will be absolutely displeasing to almost all who listen to them.

Similarly, there is not one true way to write a program.

Programming is like literature; nobody would confuse even a single paragraph of Umberto Eco with a paragraph of Richard Brautigan.

Similarly, a programmer’s coding style is as unique and identifiable as the styles of any author in any medium. Programmers can easily recognize each other’s work once they know a particular programmer’s style.

Programming is like painting; what you don’t paint is as important as what you do. Negative space defines form, adds depth.

Similarly, more lines of code is not better. Top notch programmers measure the success of refactoring by how many lines of code are removed, not by how many lines were added.

Programming is like dance; anyone can dance, but only a very few people can dance well enough to be paid for it.

To write good code a person has to have a certain kind of mind, a mind that can analyze messy human processes and practices, and then boil it down to formal logic.

A good programmer needs abundant patience. Far more patience than the average person has, because :

“Computers are like Old Testament gods; lots of rules and no mercy.”

~ Joseph Campbell

A good programmer has to be able to mentally process things that are quite simply impossible for most people, such as recursion, or concurrency, or callbacks or closures.

…and that brings us to the second question

Can we really teach every child to program?

I cannot even fathom that anyone takes this question seriously.

We might as well ask if we could teach every child to perform appendectomies. Doctors are in short supply, it would make a lot of sense to increase the supply of doctors and surgeons by teaching all children to become medically literate, no?

Or maybe we could teach every child to become a teacher, then we wouldn’t need schools because when this children grew up, all adults would be teachers for the following generation.

I suppose we could teach every child to program, in the same way that anyone can dance. But it is no more possible to teach every child to program well than it woud be to teach every child to dance en pointe, or entrechat.

In conclusion

Programming is art by anyone’s definition, but we should not confuse the art of programming with the repetitive integration and fitting together of code libraries that most of the “developers” working in enterprise IT departments are doing. The same way we should not confuse the engineer who designs the car with the assembly line worker that assembles the car.

“Application development” is not programming and will inevitably become more and more automated and accessible to more and more people with less and less skill.

Meanwhile “real programmers” will continue to practice their art, the same way that cars are still modeled in clay.

Is programming art… was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.

I am an over-eager programmer. I always have been.

When a friend or colleague comes to me in distress and asks me if I can help by writing a “quick” program, in spite of myself I always say yes.

What happens next follows three very predictable stages:

Stage 1: Gratitude

At first I am a hero. So unlike all those other stick-in-the mud prgrammers that hem and haw, the hedge and dodge and evade. So what if there are rough edges? I had warned them there would be, and speed trumps perfection.

Stage 2: Presumption

After the initial delight has worn off, we quickly move to a phase of rapid-fire change requests. Can you do this? Can you do that? Can you make this a little different. Can you make that completely different?

Forgotten are my warnings that each change brings us closer to the day when the only way to get a stable bug-free application will be to re-write it. Of course each change request is more urgent than the last, so “just do it” is the order of the day. No time and no money to take a step back and learn from what was essentially a working prototype.

Stage 3: Resentment

In the last stage, the rough edges, that were there from the beginning now become unbearable, unacceptable bugs. I am a loser, a hack. All this time, all this money, and what we have to show for it is a Big. Pile. Of. Crap.

I have been doing this a long time. I have been through this with programming I have done myself, and with programming done by teams working for me. I know how the story plays out, and yet I can never bring myself to play the game that so many of my colleagues do: “oh, that’s very complicated, I am not even sure it’s possible, it could take a lot of time. Give me two weeks to estimate it.”

So ultimately I am to blame. I should know better.

Most of my colleagues are hyper-defensive, refusing to give estimates (that will automatically become deadlines), refusing to commit to anything. So Business owners and project managers, and enterprise architects all dream up arbitrary estimates for them, and these become budgets and deadline that are almost never, ever, met.

This is how the salt mines of IT work these days. Driven by the worst of human behaviour on all sides. I guess I am a dreamer, I just refuse to accept it.

To read more of me, get my book The Chaos Factory which tells the inside story of corporate IT failure, and provides the real recipe to fix it.