A new health record management system is going live in Australia soon. It’s called My Health Record. It will affect every Australian (and potentially temporary and permanent residents as well) as it will store some of their most intimate information — their health records!

But do the risks of such a system outweigh the advantages?

Starting development around 2009, the system has cost at least AUD 2 billion and is touted as being:

A secure online portal, designed to provide a summary of your health information to trusted medical professionals.

However, according to the ABC:

It’s not a comprehensive picture of your health — it will only contain what you and your doctors choose to upload, and will depend on the quality of those records.

Yet, despite the comprehensive nature of the project little information exists about it; whether on billboards around at airports, or via advertisements on television, social media, or in the traditional newspaper press.

Given that, it must be quite a surprise when you find out about the existence of the system, and that unless they opt-out a record will automatically be created for you.

What’s In a Health Record?

To be as specific as possible, the record:

can include details of your medical conditions and treatments, medicine details, allergies, and test or scan results, all in one place.

Each record is created from up to 2 years of health information from your Medicare records, which can include:

• Medicare and Pharmaceutical Benefits Scheme (PBS) information held by the Department of Human Services
• Medicare and Repatriation Schedule of Pharmaceutical Benefits (RPBS) information stored by the Department of Veterans’ Affairs (DVA)
• Organ donation decisions
• Immunisations that are included in the Australian Immunisation Register, including childhood immunisations and other immunisations received

Once a record is created:

Healthcare providers such as GPs, specialists and pharmacists can add clinical documents about your health to your record, including: — An overview of your health uploaded by your doctor, called a shared health summary. This is a useful reference for new doctors or other healthcare providers you visit — Hospital discharge summaries — Reports from test and scans, like blood tests — Medications that your doctor has prescribed to you — Referral letters from your doctor(s).

Regarding how Healthcare providers can update your record, the My Health Record twitter account says that:

Healthcare providers access My Health Record through their own integrated software system and the timeout rules are subject to the specific clinical system.

I’ve yet to find out what the “integrated software” is, but I’ll update this post when I do. Then there’s one other point to remember. The OAIC (Office of the Australian Information Commissioner) says:

A healthcare provider organisation can upload information to a patient’s My Health Record even if the patient has set access controls. A healthcare provider organisation should not upload information to a record if a consumer requests them not to.

How Long Does a Record Last For?

Once created, the records persist until either:

1. At least 30 years after an individual’s death; or
2. 130 years after an individual’s birth — if it is not known when they died.

Sounds OK, No?

What Are The Top Concerns?

Being a security researcher, warning bells — which rapidly changed to alarm bells — went off in my mind. You might not appreciate why I was becoming so concerned. You might see the system in a completely different light, as something benign and beneficial.

On the surface of it, the system sounds like a valuable and time-saving system. Given the current system, where records may need to be emailed or faxed, duplicated between a range of different health care providers and specialists, that’s understandable.

To be fair, if handled well this could be a massive win for the Australian population. Think about it; all your medical records in one place, able to be easily located and accessed. What’s more, if you change doctor you don’t have to tell the next one your medical history, as they can see it online for themselves.

If you need emergency care or have to visit several specialists to identify a medical issue, the benefits of the system only increase further. Conceptually, that’s something to work towards. However, that’s where it all breaks down for me. Here’s why.

The Records are Centrally Located

If the records are hacked or lost, consider the implications of such intimate and personal information being available for anyone to see.

Once that information is lost, what might it mean if it was later uncovered by anyone other than the health authorities?

• Could an insurer decline to accept you as a customer, decline to renew your subscription, or inflate your premiums?
• Could you be discriminated against by an employer based on your sexuality, such as being a member of the LGBTIQ community?
• Could you be discriminated against by an organisation based on preconceptions around why you either sought out or were prescribed specific medication?
• Could you be declined service at a government department based on some aspect of your medical history?

And these are just some of the potential implications.

Perhaps you don’t think that centrally located health information is a genuine concern. However, it is. Centrally-located information is far easier to hack than data that is distributed.

This happened in the UK twice. The first was when the NHS (National Health System) lost more than half a million pieces of confidential medical correspondence. In that case, up to 708,000 pieces were lost.

The second, also in the UK, was the result of “an IT glitch”. As a result:

GPs will be forced to review the records of over 44,000 records across more than 2,500 GP practices in England following the error, which led to the loss of vaccination information and pathology results.

As the information is centrally located, it’s reasonable to expect that the security measures will be exceptionally well thought out, implemented, reviewed, and improved on a regular basis. However, are they? It’s unclear.

The most detailed information that I’ve been able to find is this:

We use a range of technology to protect the sensitive personal and health information held in the My Health Record system, including:

• Firewalls to block unauthorised access
• Audit logs to track access to records
• Initial and regular anti-virus scanning of documents uploaded to records, and
• System monitoring to detect suspicious activity.

Let’s consider each of these points.

I’m assuming that they’re referring to a standard firewall here, not a Web Application Firewall (WAF). To the layperson, using a term such as “firewalls” might sound impressive, but as I’ve covered previously, firewalls are a pretty blunt and broad tool.

It makes more sense to use a WAF. If you’re not familiar with a WAF, it’s a special-purpose firewall for HTTP/S-based applications. Given that, that’s what My Health Record is, then it’s logical to use one over a traditional firewall.

However, even these have shortcomings, which include:

• They can generate false positives and false negatives
• They can be easily bypassed
• They provide no protection against 0-Day exploits
• Their rules have to be maintained as the application changes

Audit Logs To Track Access To Records

Logging is a must in any application. Developers and systems administrators need this information so that they can ascertain what went wrong, when, where, and why. However, from the title above, it’s not clear as to whether the information is user-facing or only for support staff. If it’s user-facing, what details does it contain?

Will a user know:

• Whose credentials were used to access their information?
• The location from which the record was accessed, including country and town?
• The IP address of the access location?

In addition to this information, will accesses from anyone other than the owning user require a reason to be provided? If the information is internal, for systems administrators and support staff:

• Is it stored internally in the same data centre as the My Health Record application?
• Is it stored externally, using an external provider such as Loggly, Splunk, Rollbar, or Gralog?
• What information is contained in a log entry?

It’s essential to know this because while it’s stated that the health records must be stored in Australia, it’s not clear that that applies to log information. Secondly, it’s quite common for log data to contain sensitive information accidentally, such as user credentials, information that can be used to hack the system.

Initial and Regular Anti-virus Scanning of Documents Uploaded to Records

How regular is “regular”? Hourly? Daily? Weekly? Monthly? Does “Initial” mean on file upload? While it’s excellent that anti-virus scanning of (I’m assuming all) documents occurs as a regular part of the service, there are a range of other security best practices to observe. For example:

• Are there restrictions on uploadable file types?
• Are there restrictions on the maximum uploadable file size?
• Where are the files stored after upload?
• How are the files stored after upload?
• Have attacks such as path traversal and path manipulation been accounted for?

System Monitoring to Detect Suspicious Activity

What defines suspicious activity? I asked this of @MyHealthRec recently. If you missed the tweet thread, here are some of the questions that I asked:

What happens when unauthorised access occurs, such as my doctor’s credentials being used to access my records from two different locations simultaneously — regardless of whether they were in the same town or a different country?

Will the access be automatically detected and blocked? If so, will I then be notified within a short period after that by phone or SMS? Alternatively, will I be notified via a slower and more indirect means, such as email? At some point, access credentials will slip out, so it’s essential to know how this will be handled.

Alternatively, will I have to contact support when I see suspicious activity? This is concerning, as the onus for checking access appears to be placed on the individual. I say this because I found this on a report by the ABC:

You can also set up an email or SMS alert for when a healthcare organisation accesses your record for the first time. The privacy commissioner recommends regularly checking for unexpected or unauthorised access. You can call the ADHA on 1800 723 471 if you think something’s gone wrong.

While a good start, this concerns me for two reasons. Firstly, it says that an email or SMS alert is sent when organisations access your records the first time. However,

• What about any time after that?
• What about if fraudulent activity was suspected?

Then, “The privacy commissioner recommends checking regularly for unexpected or unauthorised access.”. Human habits being what they are, what percentage of people are going to do this? It’s fair to say that it won’t be the majority.

Given these two things, it appears that if you find out about suspicious activity on your record or access, you don’t consent to, it’s only going to be after the fact.

How Can You Control Who Access Your Records, When, and Why?

A post on the ABC raises many concerning questions about who can access your records, along with when, how, and why. It seems that the limitations are far from clear. What is known is that your records are:

reserved for people who work for a registered healthcare provider and who are authorised to provide you with care

If you want to limit access to your data, you can do three things:

1. Set a Record Access Code (RAC): This controls which healthcare provider organisations can see your record.
2. Set a Limited Document Access Code (LDAC): This controls healthcare providers organisations’ access to specific documents
3. Set a Personal Access Code (PAC): This allows your nominated representative(s) to access your My Health Record.

That all sounds pretty good, and I agree. However, if you read the information more carefully, you’ll find the following:

Please note that when emergency access is granted, any access controls previously set are overridden. This means that any restricted information can be accessed in an emergency.

Emergency access is defined as:

If you have set an access code for your My Health Record and there is a serious threat to your life, health or safety, emergency access to your record may be provided. If you were unconscious, for example, hospital staff may be granted access to your record if there is serious threat to your health or safety.

Now that’s perfectly understandable. If you’re not able to act on your behalf, then it’s essential that someone else can. However, that’s not the end of the story. There’s another reason for providing emergency access:

Emergency access may also be granted to lessen or prevent a serious threat to public health or safety.

Again, on the surface of it, this sounds reasonable, yet it’s still very broad. What’s more, there’s a “secondary use” provision.

Here’s the brief version of what it means:

My Health Record data may be used to provide insight into Australia’s health system and the services being provided to improve health outcomes for patients. Find out how My Health Record data can improve health outcomes for all Australians. If you are happy for your data to be used for secondary purposes such as research, you don’t need to do anything.

Note how that last sentence ends: “you don’t need to do anything”. This says that, unless you opt out — assuming that you knew you were opted in — your records are included in information used to provide said insight.

That said, from what I’ve ascertained, the information is only available in an aggregated and anonymised form; meaning that it should be impossible to link information to an individual. Intentional emphasis added on “should”. Research has already shown that anonymised information can be de-anonymised.

This concerns me for several reasons, primarily because…

Users can see who has looked at their records. This information includes:

• When their record was accessed.
• Which organisation accessed it; and
• How the record changed.

However, they won’t be able to know, for sure, precisely who accessed their information. In part this makes sense, and in part, it doesn’t. Either way, it’s concerning.

What’s more concerning, however, is the legislation that governs access, the My Health Records Act 2012. This was pointed out recently by Dr Karyn Phelps, former AMA (the Austrlian Medical Association) head.

After reading through the relevant sections of the act, it appears that access may be available to all of the following:

• The Police
• The Australian Taxation Office
• The Courts
• The Veterans Affairs Department
• The Defense Department
• The Department of Human Services
• Medicare
• Any established regulatory body
• A body established by the Commonwealth
• Any Australian Public Service employee in the above departments
• Ministerial Council for data purposes
• Computer programs as designated by the “System Operator.”
• Any other person with the consent of the Minister

As Dr Phelps shared in her article:

These provisions break the bond between clinician and patient. The idea that police and security agencies, and the ATO and other agencies could trawl these databases at will is abhorrent. To its great shame, the Parliament passed these provisions.

And according to The Guardian, the Queensland Police Union (QPU), after taking legal advice, says the following about record access requirements:

Access to My Health Record will not be limited to police, as the list of enforcement bodies who may access records includes the immigration department, anti-corruption commissions, financial regulators and any other agencies that impose fines or are tasked with the “protection of the public revenue”.

The Guardian’s report goes on further to say:

[the QPU has] legal advice that there is nothing in the legislation that requires any enforcement body to obtain a warrant to access My Health Record.

This clearly suggests that a — now removed — article from the Parliamentary Library was correct when it said:

It represents a significant reduction in the legal threshold for the release of private medical information to law enforcement. As legislation would normally take precedence over an agency’s ‘operating policy’, this means that unless the ADHA has deemed a request unreasonable, it cannot routinely require a law-enforcement body to get a warrant, and its operating policy can be ignored or changed at any time.

And here’s something concerning that ZDNet wrote recently, to round out this point:

Those types of clauses in other legislation has previously allowed the likes of Bankstown City Council, Victorian Taxi Services, the RSPCA of Victoria, and Australia Post to get their hands on telecommunications data in the past

Can you see why having such broad access is so concerning?

UPDATE

According to a report by the Guardian, new privacy protections will be added to the controlling legislation to ensure that the service can only be used for medical purposes.

And according to a Media Release by Health Minister Greg Hunt there are two excellent changes to the legislation.

Firstly:

This policy requires a court order to release any My Health Record information without consent. The amendment will ensure no record can be released to police or government agencies, for any purpose, without a court order.

Secondly:

In addition, the Government will also amend Labor’s 2012 legislation to ensure if someone wishes to cancel their record they will be able to do so permanently, with their record deleted from the system.

The changes are yet to be enacted. But it’s heartening to hear that concerns have been listened to.

According to section 70 of the My Health Records Act:

The ADHA is authorised, by law, to disclose someone’s health information if the ADHA “reasonably believes” it’s necessary for preventing or investigating crimes and “protecting the public revenue”.

On the whole, that sounds reasonable. If law enforcement reasonably believes that by accessing someone’s information they can prevent a crime, then it’s arguable that they should have access. You’d like to think that they could do their job and protect the public.

However, what does “protecting the public revenue” mean? The closest definition that I could find is this one from the Office of The Queensland Information Commissioner:

The public revenue includes levies, taxes, rates and royalties charged on a regular basis. It does not include occasional charges, such as fines, or the recovery of the occasional overpayment by an agency. Protection of the public revenue includes the activities of agencies and bodies intended to ensure that lawful obligations are met by those subject to the charges, such as routine collection, audits, investigatory and debt recovery actions. Prosecution for failure to pay the charge would fall under the criminal law exception. Activities intended to identify and eliminate inefficient but lawful spending of public money will not fall within these IPPs.

How does that apply to a health record management system?

Apps Can Access Your Data

According to Tim Kelsey, the person in charge of the process, initially, apps which you consent to give access to, can “show” people their records, but won’t be able to store them.

According to the ADHA (the Australian Digital Health Agency), any apps that seek to access your data undergo a “strict assessment” and need to abide by a Portal Operator Registration Agreement. The agreement includes that:

They do not download or store My Health Record information on their own system, or pass that data on to a third party.

I see several problems with this situation. Firstly, will users always know — for sure — that they have consented to give apps access to their health records?

We need only consider how Facebook presents the choice for users to opt-out of facial recognition on their platform, which you can see in the image below.

While you can toggle a Yes/No select box, the text above it, and in several of the steps required to get to this point, make it appear as though this is not the choice that you want to make and that you’ll be left out of many positive things if you do opt-out.

Will apps you allow access to your My Health Record data act in the same, or a similar way?

If you’re interested, the currently known list of apps that can access your My Health Record are:

• Telstra
• HealthEngine
• Tyde; and
• Healthi

Do We (Really) Understand What We’re Agreeing To?

Next, consider privacy policies and terms and conditions that we’ve all agreed to before we can install or use an app or service. How many of us read them, or read them in their entirety.? Of those that do, how many of us fully understand the implications? For an excellent example, have a read of the license terms for Microsoft Windows.

Alternatively, will the information provided when consent is requested, be too broad in nature? Take the example above of when the HealthEngine app requests consent to access My Health Record account. Is that sufficiently detailed, or provide access to sufficiently detailed information to understand the level of access you’re consenting to?

In time, I’m confident that countless users will say that the either never knew, or never fully appreciated, what the app would do, or was asking for.

Now let’s consider the condition that apps can only “‘show’ people their records, but won’t be able to store or share them.”. Will the government review the source code of every app to know that once the downloaded data is not stored — it has to be downloaded before it can be viewed?

I’d suggest that they won’t be given access to the source code, as it would contain intellectual property which the developer(s) don’t want to be viewed — for obvious reasons.

Alternatively, will the government test each app before it’s allowed to access the platform to check if it stores information locally? Depending on how the application’s developed, it might be impossible to know if this happens or not.

And to reiterate, the information must be downloaded so that the user can see it.

And here’s a third potential problem: as yet unknown architectural flaws. What if the user installs an app that, seemingly, had nothing to do with My Health Record? And what if that app was able to conduct a side-channel attack and extract and share information downloaded by a legitimate app?

It might seem a bit far-flung, but recent attacks such as Spectre and Meltdown have indicated that these types of attacks are not only possible but that they can happen.

Conclusion

For The Time Time Being — Opt-Out.

I know that My Health Record has the potential to be a significant improvement upon existing practices, such as faxing or emailing records between hospitals, organisations, and relevant government departments. And I’m sure that, if developed properly, it would allow medical professionals to do their work more effectively and efficiently.

However, given the security concerns, lack of documentation, and grey areas that I’ve covered in this post, along with a significant number of other concerns that I’ve read through, I’m far too concerned about the downside to opt-in for the potential upsides.

Perhaps you see it differently. Perhaps you feel that Australia’s existing privacy legislation provides sufficient protections. If so, then make the choice that you’re most comfortable with.

For my own part, I’m opted out until I see these concerns answered and the documentation made much more specific.

Tagged with: My Health Record / Australian Government / Online Security / Cyber Security

Originally published at matthewsetter.com.

This was originally published at matthewsetter.com, on May 1st, 2018.

Have you ever stopped to wonder about just how trusting you are with your online privacy, and how much information you freely give away?

Have you ever thought how savvy you are in the modern era, but how naive your parents and grandparents were? I believe we could be living in a much more naive time now. Here’s why.

As I was researching last week’s post, on whether we should still trust Facebook with our online privacy and personal data, I was talking to my Mum. She said that she really liked the article (go Mum!) and that it reminded her of her Mum and her Mum’s attitude to sharing personal information.

She shared the following quote with me — that floored me! Have a read and see what you think.

My mum lived in the day when a person connected phones through to the number you called and in small towns notoriously that person knew, and spread all the town news and gossip. Until the day she died, mum was very careful about what she discussed over the phone for fear of “ anyone could be listening in”. I fear she might have been right.

I fear she may have been too, Mum. And to be honest, it makes me cringe more than just a little when I look back over how open and transparent I’ve been over the years.

I’m not saying that sharing facts about ourselves online is necessarily a bad thing, and that we shouldn’t share anything about ourselves. After all, sharing is a fundamental part of building and maintaining relationships with others. It helps makes us human. It helps makes us who we are!

However, when you share too much without stopping to consider the ramifications of doing so, it just may come back to haunt you at some point in the future. I hope for my sake that none of my previous sharing does. :-)

What about you?

Next time you’re asked, such as in a Facebook game, what your favourite comic character is, or the first street you lived in was, will you consider not sharing that information?

CC Image Courtesy of Royce Bair on Flickr

This post is dedicated to my Mum, and my two Grandmothers, who I still love very deeply.

Categorised under: privacy | Tagged with: sharing, online privacy, Facebook.

This was originally published at matthewsetter.com.

In light of the recent Cambridge Analytica / Facebook scandal Mark Zuckerberg testified before the US Congress. Did you tune in to hear what he said about what they knew? Were you keen to know more and about how they might be planning to protect your privacy in the future?

I was certainly keen to hear what he had to say. With over 2 billion users, Facebook is the largest social media platform in the world; the actions that they take are important, and have wide-ranging consequences.

I have to be honest though, while I wasn’t surprised by what I heard, I still came away feeling somewhat disappointed. And as I listened to the testimony, I became a little concerned.

Mark hardly gave a straight answer to any of the questions, and his few conclusive responses were lost, I felt, in many half answers, and broad, somewhat innocuous statements.

However, Let’s Put It In Greater Context

To be fair to Mark — despite his power and influence within Facebook, the tech community, and the wider business world — the rules are completely different when you’re facing the US congress. If the rules are anything like they are in Australia, you don’t have much power.

I’m not entirely familiar with the legal situation in the United States. However, some colleagues have advised me on a recent post on my Facebook page, that he does have to be quite careful with what he says, as it may be used against him far into the future.

Whereas for the politicians, they have quite a lot of liberty to make accusations and assertions, even some that are incredibly specious at best. Compounding that problem, I noted several others. The first is that a number of them didn’t seem all that apprised of how Facebook or social media platforms work.

The next is that some didn’t come across as all that tech savvy to begin with. Moreover, the third is that each only had about 4 minutes to ask questions. That’s barely enough time to form a proper question, while also providing sufficient background context, along with additional meaning if required.

So the entire environment seems imperfect all round. Regardless, that was the state of play. Given all of these, and no doubt many more factors, it’s entirely understandable that Mark Zuckerberg would be careful about what he divulged at the hearing.

Add to that the amount of heat that’s on the company at the moment, and I’d suggest that it’s only logical to expect that this outing is more about saving the company’s reputation and limiting the fall out more than anything else.

When you put all of that together, what we heard seemed to be only a set of well thought out and rehearsed talking points, ones designed to create the impression that the company:

1. Is genuinely sorry for what has happened and for the mistakes that they made
2. That they will do all that they can to do better in the future — ideally avoiding it happening again.

And let’s be honest, most companies would likely operate similarly, if they were in the same position. So, similar to the banking royal commission in Australia, it doesn’t make sense to either pre-empt anything or to attempt to rush change prematurely.

As I’ve never worked at Facebook, nor at the upper levels of the company, nor know anyone who does or has, there’s no way that I can know anything about the discussions and goings on there regarding the platform, their attitude, and their data protection and privacy policies.

Despite all this, I’ve been feeling uneasy about what I heard. Yet feelings are notoriously unreliable things, so I decided to do a little bit of digging, to see what else I could find out about Facebook, and how it works.

Hopefully, but understanding the company in a more significant context, it could answer some of the lingering questions that I had.

In this post, I want to share some of that research with you and discuss several questions that I have. I’d love to hear your thoughts about them in the comments below.

Why Didn’t They Notify 87 Million Users That Their Personal Data Was Breached?

According to Politico, Facebook first learned about the breach of up to 87 million users’ data around 2015. In addition to that, Facebook made a conscious decision not to inform users about the breach.

Zuckerberg also said he didn’t remember when Facebook made the decision not to inform users that their data had been breached and that he didn’t remember being involved in the conversation where the decision was made.

What’s more, when the scandal publicly broke, instead of providing a public update they said nothing for a further week. Now let’s consider another point. According to Politico, when Facebook learned of the breach, they approached Cambridge Analytica and asked them to remove the breached data.

What they did not do was an audit to ensure that the information was removed. Apparently, they accepted Cambridge Analytica’s word that it was addressed.

Now let’s consider Mark’s response when asked about the situation. During his testimony to the US Congress, he said:

We didn’t take a broad enough view of our responsibility, and that was a big mistake,” he added. “It was my mistake, and I’m sorry. I started Facebook, I run it, and I’m responsible for what happens here. It will take some time to work through all of the changes we need to make,” he added, “but I’m committed to getting it right.

Sounds good, doesn’t it, but words aren’t the same thing as actions. Stop for a moment and consider all of these points individually, and then collectively. Consider how you’d feel — and what you’d do — if your bank, credit reporting agency, or insurance company had such a large proportion of their accounts breached and said nothing.

How would you feel if they both knew about it, but made a conscious decision not to inform anyone, then, when it all came to light they went quiet for another week. What would that do to your perception of them?

The information that banks, credit reporting agencies, and insurance companies, among others, is different to what Facebook holds. This is still private data, entrusted by users to Facebook. So while it may appear that the comparison’s ill-suited, there’s a measure of comparison which can be afforded.

I don’t want to single out Facebook’s handling of the breach, nor their delay in acknowledging it as being unique. Of all the high profile cases of recent years, we only have to look at Yahoo!

While they publicly acknowledged in 2016 that up to 1 billion accounts had been compromised, that breach dated back as far as 2012 or 2013.

We later found out that not only had up to 1 billion accounts been breached but that all Yahoo! accounts had. On top of that, we could look at a variety of other companies, including Ashley Madison, AdultFriendFinder, Telegram, and BitFinex.

Facebook’s hardly unique here. I want to think that, with the relationship that they’ve worked to build up over the years, that they’d have handled the situation better, but they’re not alone.

Why Fight Regulation Designed to Improve Users’ Privacy?

According to The Guardian, Facebook recently shifted the responsibility of more than 1.5 billion users (all Facebook users outside of the US and Canada) to their Californian offices.

That’s significant, because the new EU privacy regulation framework, the GDPR, is set to come into effect on May 25, 2018.

So they’ve moved more than 1.5 billion users from a jurisdiction where privacy regulations are set to become much stronger, to one where they’re weaker? If you were really committed to user privacy and “getting it right”, why make that move?

Let’s consider some potential reasons.

• Given the scale and reach of their platform (over 2 billion users around the world), are they not ready yet — despite having since 2014 to begin preparing for it?
• Are they not ready for fines of up to 4% of global turnover (or around $1.6bn), if they’re found to be in violation of the GDPR?
• Is the cost of moving the users less than the cost of one or several violations?
• Is it easier and less costly to comply with one regulatory framework, rather than several simultaneously?
• Is it easier to consolidate all their users under one regulatory framework, until a newer, foreign framework has been fully established and is more widely understood? Until the GDPR is in effect and better understood — and tested — it’s likely a sane choice to consolidate, wait, and see.

Any one of these or several of them are rational, logical, and pragmatic.So while the timing could be justifiably seen as conspicuous, it needn’t be.

However, according to Fortune, on April 19, 2018, the move may have to do with Facebook’s re-introduction of facial recognition for European users and the fact that it may be illegal under the GDPR. According to the article:

The Irish DPC has jurisdiction over Facebook because its international services all run out of its Irish headquarters — all users outside North America currently agree to the terms of services coming out of that office.

Now consider that in tandem with the fact that:

The Irish DPC is querying the technology around facial recognition and whether Facebook needs to scan all faces (i.e., those without consent as well) to use the facial recognition technology. The issue of compliance of this feature with GDPR is therefore not settled at this point.

It would seem, given the timing of the move and the outstanding nature of the technology concerning the GDPR, that the decision to move the users away from the Irish DPC’s jurisdiction has more to do with being able to re-introduce facial recognition in Europe than any of the other potential reasons.

What do you think?

Why Fight the Strongest Privacy Legislation in the US?

This legislation is called the Biometric Information Privacy Act (BIPA), which requires explicit consent before companies can collect biometric data like fingerprints or facial recognition profiles. If they were genuine in wanting to do better would it not be logical to:

• Take advantage of the introduction of the legislation to show just how genuinely committed they are
• Ensure that they were prepared for it as soon as possible and that new user controls were as transparent and straightforward as possible?

Unfortunately, according to The Verge, they’ve been quietly lobbying for an amendment to the proposed legislation which:

Allows companies to collect biometric data without notice or consent as long as it’s handled with the same protections as other sensitive data. Companies could also be exempted if they do not sell or otherwise profit from the data, or if it is used only for employment purposes.

To be more specific, according to the post, Facebook isn’t actively or directly lobbying for the watering down of the law. However, it notes that they’re an active member of the Illinois Chamber of Commerce’s Tech Council, which is.

Whereas the move of over 1.5 billion users could have any number of pragmatic reasons for being carried out, this one seems to have less room for doubt.

However, let’s look deeper. Facial recognition is a reasonably new area of modern technology. Given that, the implications of being able to scan, store, and track people is far less understood, than something such as fingerprints or metadata.

Moreover, like anything new, it’s been used by corporations and organisations long before people are fully aware of it happening, the consequences and implications of it have become modestly known, and any legal framework has had a chance to keep pace.

As a result, it’s logical to expect three things:

However, on this matter, I agree with a statement by a group of privacy advocates, including the Electronic Frontier Foundation (EFF) and the American Civil Liberties Union (ACLU), which The Daily Dot reported on back in 2015:

At a bare minimum, people should be able to walk down a public street without fear that companies they’ve never heard of are tracking their every movement — and identifying them by name — using facial recognition technology. Unfortunately, we have been unable to obtain agreement even with that basic, specific premise.

Now some of the referenced articles are a few years old, and deliberately so. I’ve done that to consider a before and after perspective. Now let’s look at what has changed at Facebook in recent years.

Recently, they re-introduced facial recognition for European users, after it was removed because of privacy concerns 5–6 years ago. This might sound concerning if you’re against it, but according to Facebook spokesperson Rochelle Nadhiri, the feature is off by default.

However, Wired reports that it’s not quite that cut and dry:

The new setting respects people’s existing choices, so if you’ve already turned off tag suggestions, then your new face recognition setting will be off by default. If your tag suggestions setting was set to ‘friends’, then your face recognition setting will be set to on.

So, it’s not a clear opt-in or a yes or no situation. If you’re not aware of the implications of tagging, then it’s reasonable to expect that you’d be confused to find that facial recognition was already enabled when you’d not explicitly chosen to opt-in.

Moreover, with a feature as pervasive as this could become, or already may be, this needs to have a clear-cut choice to opt in or out.

Now, while this may seem concerning, at least Facebook, while not completely clear with their notifications about the introduction of facial recognition, are evident that they’re rolling it out again.

This stands in stark contrast to Westfield shopping centres (Australia). According to a report in October, 2017:

Every time you walk into a Westfield shopping centre you are being tracked. Sophisticated facial detection software, working in conjunction with the devices you are carrying on you, are monitoring your mood and tracking your every move. The company uses small cameras fixed atop advertising screens, that detect individual faces in order to record the age, gender and mood of shoppers. Westfield also tracks shoppers’ movements by pinging their Wi-Fi enabled devices with routers littered across its centres.

The report went on further to say that specific consent isn’t required as “the Privacy Act 1988 only regulated the collection of personal and sensitive information”.

At least Facebook let’s you know what they’re doing, even if you have to do some digging and be somewhat savvy to appreciate the full ramifications of the choices that you’re provided with.

Besides, if you want to opt out at any time, a little googling is all it takes to find out how — even if the process is somewhat convoluted.

Why Give $200,000 to Help Fight Against Increased Privacy Controls?

Now let’s consider a third political lobbying effort. According to NBC News, back in February of this year, Facebook gave $200,000 to The Committee to Protect California Jobs. Why’s this significant? Because the committee is:

A business-backed political action group dedicated to defeating a state ballot initiative that would expand Californians’ privacy controls.

If Facebook were keen to do better, why actively support a group that’s actively lobbying to avoid improving privacy controls for users? Would you not put support behind already existing groups that were working to improve privacy?

To be fair, the article goes on to report that Facebook has since pulled support from the committee, because:

We took this step in order to focus our efforts on supporting reasonable privacy measures in California.

However, they still gave the money earlier this year, not years ago. As a nod to them though, they have stated that they would adopt the recommendations of the Honest Ads Act, which seeks to regulate online political advertising.

Next, I want you to think about two points that an article in Der Spiegel (German) shared recently:

1. We put our online data there ourselves.
2. We love to vicariously enjoy the intimate details of the lives of others (e.g., royals, celebrities, and politicians, etc.), but we don’t like it when the details of our own lives are shared.

While Facebook does use, and has the power to abuse our data, we do, willingly, hand a lot of it to them in the first place. Now let’s dive down the rabbit hole just a little bit further. Let’s say you willingly give Facebook (or any other social network or organisation) your data.

Then, let’s say you give “friends of friends” access to your data so that your friends’ friends can see and comment on your data, or your friends can share your thoughts and posts with their friends. And when all that’s in place, let’s say that you have a friend who then grants an advertiser access to that data.

How should Facebook handle it? Consider that you have expressly given that friend access to that data and then that friend has explicitly given access to that data to the advertiser. You could say that, technically, Facebook is just doing what you have expressly allowed them to do with your data.

This is what bothers me about how social media platforms work and how our lack of understanding about them can lead to situations that we genuinely don’t expect or appreciate.

Given the change of events that I just described, it’s reasonable for Facebook to assert that you allowed them to do what they did. At the same time, it’s reasonable to expect that the average user may not fully appreciate what they’re consenting to.

Chew over that for a moment or two.

That’s a Wrap

There’s a lot to consider around the use of our personal data, how social media platforms (and other companies) collect, analyse, and user our data, and what that means for our privacy in the modern age.

And this article, even if it were twice or three times as long, couldn’t hope to discuss it, nor Facebook’s motivations or actions, in sufficient detail.

However, I hope that by pulling more threads together, you are in a more informed position, and able to make a more informed choice about how you approach using Facebook and other sites in the future.

Are you going to close your Facebook account? Are you going to continue as before? Alternatively, are you going to take a more considered view as to how you use it, what you share, like, and comment on?

Have you reviewed your privacy settings, and locked them down as much as you can? If you haven’t, I strongly urge you to do so!

Do you ever wonder just how much information Google is storing and sharing about you, all so that they can give you those search results?

Ever felt like you were being tracked online, all so that you could do some simple searches? Well, it’s time to consider alternatives to Google, ones that give you great search results, but that don’t invade your privacy.

Why You Should Ditch Google

We’re all familiar with Google. It’s not only a household name, it’s also a modern verb! We no longer search online anymore. We google for things.

And that’s entirely understandable. Because depending which report you read, Google accounts for as much as 87% of all internet searches. It’s also referenced in popular culture.

Moreover, despite efforts by alternative search engines — most notably Microsoft Bing, which only has 5.2% market share — no one’s ever managed to gain any amount of meaningful traction on Google’s dominance. I don’t know of anyone who bings anything. Do you?

For the longest time, this setup seemed all fine and dandy. You opened up your browser, which defaulted to Google’s home page, did some quick searching, found what you wanted to learn, do, or buy, and did it. Happy days! Right?

However, in recent years that’s begun to change. People have been feeling that they’re being tracked online. Perhaps you have. As a result, we’ve started learning about how Google is able to provide such high-quality search results Moreover, what we’ve found is more than a bit concerning.

Here’s an example; you search for something in Google, perhaps a new vacuum cleaner. Perhaps you buy it; perhaps not.

Then, as you visit other sites, you notice that the advertisements being displayed on those sites have an uncanny relationship with your recent vacuum cleaner search history.

Now some would argue that there isn’t necessarily anything sinister nor big brother going on here. The reason why Google’s able to provide such excellent results — and why other sites know what to suggest to you — is because you’ve willingly given away enough information already!

However, that’s not the whole story. And there’s more than an element of truth in the fact that:

1. Google does store significant amounts of data about you; and
2. That information is used to enhance their search results and those of their partners.

Have you been feeling the same way? If so, then it’s time to do something about it! It’s time to find an alternate search engine, one that still gives you excellent search results yet also respects your privacy.

You might be forgiven for believing that you can have one, but you can’t have both, but you’d be wrong. In recent years, enough people have been feeling the same way that you have, so much so that decent alternatives not only exist, but they’re getting even better.

So in this two-part series, I’m going to introduce you to two of them that I’ve been experimenting with recently; these are DuckDuckGo and Qwant. In this, the first part, we’ll start off with DuckDuckGo. And next week, we’ll finish up with Qwant.

Yes, there are others, but I don’t have sufficient experience with them to speak confidently or authoritatively about them, to recommend them. However, if you can, please do so in the comments.

What is DuckDuckGo?

Despite the name, it’s a genuine service, one I’ve been happily using for the last four weeks. To quote their website:

DuckDuckGo is an Internet privacy company that empowers you to seamlessly take control of your personal information online, without any trade-offs. With our roots as the search engine that doesn’t track you, we’ve expanded what we do to protect you no matter where the Internet takes you.

What’s more:

1. They donate to several privacy-focused organisations, such as the Electronic Frontier Foundation (EFF), Riseup, and F-Droid, as well as to many notable open source projects, including NGINX, FreeBSD, and the Tor Project.
2. They have a mobile app and a browser extension.
3. They provide privacy education.

I loved it when I first heard about it. In short, it’s a search engine that doesn’t track you and helps educate you about online privacy. What a concept!

How do they do that, you may be asking? Firstly, according to their official documentation, they don’t store:

• Your search history
• Your IP address; or
• Any details about the user agent that you used (usually your web browser).

Secondly, when you click on a search result, they don’t send the linked site the search term(s) you used, nor your IP address, nor any details of your user agent.

This is important because, without this information it’s much harder to create a browser fingerprint. If you’re not sure what that is, a browser fingerprint is analogous to your physical fingerprint; it’s something that uniquely identifies you online.

If someone knows your IP address, then they have a pretty fair idea of where you were when you searched. If you extract details about your browser, it’s even more telling. This can include details such as:

• Which one it is (Firefox, Safari, Google Chrome).
• Its version number.
• Your device’s screen resolution.
• Whether you’re using a desktop or mobile device, what it is, and its version number.
• The plugins you’re using and whether they’re active or not, etc.

From all of this information, and more, you can be pretty sure that:

1. You’ve got the same user
2. You can watch where they go, and from that information, you can get a pretty clear idea of where they go on a regular basis.

I don’t want to freak you out though, but it’s something that really frustrates me, and I hope you too. Safe to say, without this information your time online is a lot more secure.

With all that said, let’s consider a further three reasons why you’d use it instead of Google.

One: You Know Your Privacy Level

Whether you install the app, the browser extension (Firefox and Safari), they show you your privacy grade. This shows you how much a site can be trusted, both before and after DuckDuckGo’s Privacy Protection service is applied.

Two: It Forces Sites to Use Encrypted Connections

It forces sites to use encrypted connections (HTTPS) when they’re available. By using encrypted connections, you protect yourself from attack vectors, such as MitM (Man in the Middle) attacks, along with, seemingly innocent alterations to your browsing sessions.

Have a look at this one, reported by Troy Hunt some time ago. Have a look at the URL, and see what shouldn’t be there.

The DuckDuckGo tracker blocker stops advertisers from tracking you on the sites you visit. Take the example below, where it blocked Google, Amazon, Nielsen, comScore, Krux, and several others, during a quick visit to The Guardian.

There is a catch!

Now, these are solid reasons to use it. However, there’s another side to the story. The privacy that DuckDuckGo affords you only extends to online tracking and advertisers.

If, however, an organisation such as the NSA were to demand access to DuckDuckGo’s servers and get the SSL certificate, then they could then un-encrypt all of the communication with their servers. However, that’s a conversation that I’m going to leave for another time.

For now, we’re going to stay focused on helping you browse more privately online, so that it’s much harder for advertisers to track you around the internet, and potentially manipulate you or to target their advertising to you too specifically.

What do you think?

Like what you’re hearing? Want to give it a try? Well, no matter if you’re using Safari on macOS or iOS, or Firefox on Windows, Linux, or Android, in a few short minutes, you can.

Change Your Browser’s Default Search Engine

If you’re short on time, watch the short video below.

Otherwise, here’s a step-by-step guide to show you how to change your browser’s default search engine.

Firefox

In Firefox open “Preferences”, then click the “Search” tab. Then, under “Default Search Engine”, choose “DuckDuckGo” from the drop-down list.

Google Chrome

In Chrome, yes, Google Chrome, under settings, scroll down till you come to “Search Engine”, then click “DuckDuckGo” from the list. With these changes made, any searches that you go through the address bar are sent to DuckDuckGo instead of Google.

Safari

In Safari’s settings, click the “Search” tab, then in the Search Engine drop-down, click “DuckDuckGo”. It’s been there since late 2014.

Install the Browser Extension

Now let’s go further, and install the DuckDuckGo Privacy Essentials add-on for both Firefox and Safari. We’ll start off with Firefox. Open the configuration menu, and choose “Add-ons”, then click “Extensions” in the left-hand side navigation menu.

Once there, add “DuckDuckGo Privacy Essentials” in the search bar, and click return (or enter). When it appears in the search results, click “Install”.

Now for Safari; open https://safari-extensions.apple.com in Safari, and search for (again) “DuckDuckGo Privacy Essentials”. You’ll see one result appear. Click the “Install now” link to install the extension.

Let’s Test it Out

When you’ve installed the extension in both browsers, you’ll see a new toolbar icon appear, showing the DuckDuckGo logo. As it’s ready, it’s time to test it out. And what better way, than to eat your own dog food, and test it out on my freelance website, which you can see that I’ve done in the screenshot below.

When I did this test, I was a little surprised, to be honest; not entirely, but a little. I wouldn’t have thought I’d get a ‘D’ rating. However, at least it’s upgraded it to a ‘B’.

Let’s dig in and find out why my website got such a low rating, and what was changed to improve it. The encrypted connection was already there, as I use an SSL certificate from Let’s Encrypt to serve the site only over HTTPS.

Now let’s look at the blocked trackers. If you open that section, you can see that the top offenders are Twitter, Google, AddThis, and Disqus.

That’s hardly surprising, as I try and make it easy to share content on Twitter and Google (I’m not entirely against these sites), and Disqus is the comment engine, which I love most of all the comment engines available at the moment.

What surprised me, however, are that these aren’t the only ones. Trackers from VigLink, BrightTag, BlueKai, eXelate, and AppNexus were also blocked. I’ll be honest, I’ve never heard of any of these, and have no idea what they are.

I can only assume that they were bundled in as part of the plugins that I’ve used — plugins that I’ll be eagerly reviewing by the time you read this.

Now for the third and final part, Privacy Practices. It’s given me a rating of “Unknown Privacy Practices”. Interesting. What exactly does that mean?

Let’s find out. At the bottom of that section, you can click a link that takes you to https://tosdr.org/, which is “Terms of Service, Didn’t Read”. Having a quick read through their About page, here’s what the site is about:

“Terms of Service; Didn’t Read” (short: ToS;DR) is a young project started in June 2012 to help fix the “biggest lie on the web”: almost no one reads the terms of service we agree to all the time.

Makes sense to me. Though I’ve tried to be more diligent about actually reading Terms of Service, given all the apps and services that I use on a regular basis, it’s nigh-on impossible.

Install the App

Okay, we’ve changed your search engine, and we’ve installed the browser extension. Now, let’s go the final step and install the app. It lets you browse with complete privacy confidence. Using an iOS device? Download it here. Using an Android device? Download it here.

Is Google Really Evil?

Before we finish up, I want to be clear that I’m not saying that Google is evil or that you should never use it again. What I am saying, however, is that it’s not the only game in town and that you should strongly consider whether it’s the right one for you.

If, after reading through this post and the linked articles, you’re still comfortable using it, knowing what you know now, at least you’ve made an informed choice. However, if you want to change, then options await you.

In Conclusion

I have to be honest, while I love DuckDuckGo, it’s not quite as feature-rich as Google. However, then, if they don’t have the resources that Google has, nor are as old (DuckDuckGo was founded back in 2008), then that’s understandable. Despite that, however, it’s still a competent search engine.

What do you think? Are you willing to try it out? Have you made the switch already? What’s your experience been like? Please share your thoughts in the comments below.

Coming up next?

Now that you’ve learned about DuckDuckGo, in next week’s post, I’m going to step you through another alternative, perhaps a better one — Qwant. See you then.

Originally published at www.matthewsetter.com.

If you use Sculpin, the excellent PHP-based static site generator, it may not be obvious as to how to configure pagination. In this article, I’ll step through how.

By default, Sculpin’s global paginator will render up to ten items per page, if there are at least ten items available to be rendered. If that suits your needs, then you won’t need to change the setting. However, if, like me, you need to change the setting, then it’s good to know how to do it.

It’s something I’ve meant to do for some time now but never got around to finding out how to do. If you look at the image below, you’ll see why.

There are ten records rendered with the tenth trailing off at the end of the page. I’ve always been one for aesthetics and a professional appearance, so have always felt that having an even set of rows renders better, as in the screenshot below. Much better, don’t you think?

So if you’re in a similar position, how do you do it? It turns out that it’s not that hard at all. All you need to do is add a setting in app/config/sculpin_kernel.yml, which I've included a sample of below.

sculpin_pagination: max_per_page: 12

As you can see, there’s not much to it. You first define a new configuration section, sculpin_pagination, and then in it set a value for max_per_page. After saving the file, if you run either the generate or publish script, your site will then render up to twelve items per page when paginating records (assuming you've set it to 12).

Configuring Pagination For Individual Content Types

That’s how to do it on a global level, what about for individual content types? Good question. Gladly it’s straightforward to do.

In the template that you use to render a specific content type, in the YAML front-matter, you can add a configuration which is quite similar to the configuration which we’ve just seen.

The only difference is that instead of using the category sculpin_pagination, use pagination, as in the following example. Then, in it, set max_per_page as we did previously.

pagination: max_per_page: 3

Again, after saving the configuration and running either the generate or publish script, that content type will render a new maximum number of records. All other content types will be unaffected.

And That’s a Wrap

If you’re thinking of migrating away from your current blog or website backend to a static site generator and you want to know anything more about Sculpin, then check out its documentation. If you are already using Sculpin and have any tips or suggestions to share, please do so in the comments.

Originally published at www.matthewsetter.com.

How the $170 million Ethereum bug could have been prevented

Disclosure: Codacy, the automated code reviews platform, has previously sponsored Hacker Noon. For Hacker Noon readers, they’re offering 15% off using this code: HACKERNOON.

There have been a spate of high profile security breaches in recent years. These include Ashley Madison, Telegram, and BitFinex, but most significantly Equifax and Yahoo!.

Together, these breaches exposed the records of well over 4 billion private individuals. As you can imagine, in their wake security has been getting a significant level of increased attention.

And so it should. The internet, and the services which run atop it have become so essential to the functioning of modern life. When things go wrong there, they can have disastrous consequences for so many people.

So as software developers we appreciate — more than ever before — that if we create an application which is accessible via the internet, then it is a potential target.

That said, application security isn’t always easy to implement — especially when working for pointy-eared-bosses who place tight time and budget constraints upon us.

But just like testing, security’s something that we have to accept responsibility for, because we’re the ones who write the code.

Luckily for us though, as with using testing tools to help us thoroughly test our code, there are tools available to help us find potential security vulnerabilities. They’re called static code analysis tools.

What is Static Code Analysis?

If you’ve not heard the term before, according to OWASP, static code analysis:

Commonly refers to the running of Static Code Analysis tools that attempt to highlight possible vulnerabilities within ‘static’ (non-running) source code by using techniques such as Taint Analysis and Data Flow Analysis.

These tools, as the description indicates, don’t just find security vulnerabilities. They can also scan for a wide range of other problems, including coding standard violations, unused code, performance problems, and error-prone code.

Regardless of the software language(s) that you use, there are a number of static analysis packages available for it. And in addition to these packages there are a range of online services, often able to scan multiple languages and frameworks.

Here’s a short list for PHP (as it’s the language with which I’m most intimately familiar):

• OWASP WAP: Web Application Protection
• Side Channel Analyzer: Static Analysis for Detecting Side-Channel Vulnerabilities in PHP applications
• psecio/parse: A static scanning tool to review your PHP code for potential security-related issues
• PHP-malware-finder: Attempts to detect obfuscated/dodgy code as well as files using PHP functions often used in malwares/webshells
• progpilot: A static analyzer for security purposes
• Codacy

With all these libraries and services, you’d be forgiven for thinking that static code analysis would be a panacea for your security problems? For example;

1. Write your code
2. Run an analyzer
3. Correct what it finds
4. You’re done!

And in some ways this process is just such a panacea. But it is not without limitations as well; limitations you need to be aware of.

These include:

• Identifying false positives and negatives during software scans. It can be hard to analyze code in the context of a larger codebase. For example, how do you validate all calls to a function and what parameters are passed to those calls? How do you programmatically validate a function which is part of a larger call-chain?
• Difficulty detecting hard to find vulnerabilities, such as access control errors and misconfigurations. It’s common for aspects of an application to be configuration-based. A prime example is configuring access control lists. If the application being scanned is built this way, and a static code analyzer is only checking code, it may report the application as containing no known vulnerabilities, but miss an ACL misconfiguration. What’s more, how can a static analyzer fully understand ACL lists?

These are just two drawbacks to bear in mind when working with static code analysis; there are others. That said, static code analysis is an excellent way to automate the process of identifying an extensive range of security issues, well before your clients and users encounter them.

While not perfect (but what is) it goes a long way to finding issues with your code, that would not normally be detected consistently and predictably.

Static Code Analysis Has Other Benefits

Getting back to the positives, in addition to finding security problems, static analysis tools can also be used to educate developers about security issues and about how to write more secure code.

Well designed ones will not only find valid problems, they will also explain why the problem was identified.

Secondly, they can help the team grow a sense of trust in the security-readiness of the software that they’re producing. They’re not a panacea, but they can still help to know that quality is improving.

Thirdly, they’re automatable, as they can be integrated with your team’s development tools. These include IDEs such as PhpStorm, and development pipelines such as Phing, Ant, Make, Codeship, and Capistrano.

For example, you could integrate WAP as a run configuration in PhpStorm, and run it intermittently. You could also integrate it as part of a Git commit hook.

Integrating them with existing tools and workflows will help ensure that they’re used consistently. As we all know, if something’s too hard to setup or to use, it’s likely going to be avoided — even if it’s the best tool in the world!

One Thing To Remember!

By now, you may be thinking that the essential thing to scan is your application’s code. And you’d not be wrong. But do we write code in isolation? Do we ever write code that doesn’t have external dependencies?

I’d suggest that unless it’s a very small script, then it will always have at least one dependency, if not several. Looking specifically to the language which I spend most of my time with, PHP, I’d say that, more often than not, they’re going to have at least one very obvious dependency — application frameworks.

If you take a look at some of PHP’s top frameworks (Laravel, Zend Framework, and Symfony), you can see that each of them has at least a handful of vulnerabilities, which can lead to exploits. If you’re using Ruby, what about Ruby on Rails? If you’re using Python, what about Django?

This isn’t meant as a hack job on any one of these frameworks. I mention them just to point out that you have to be holistic in your security thinking. Make sure that you check your external dependencies and frameworks to see if they have vulnerabilities which need patching.

A Step-through Example

Now that we’ve had a good introduction to static code analysis and started considering what to scan, let’s work through how to use static code analysis to find and resolve security issues in our code.

In this example we’ll use Codacy to scan the Laravel framework codebase, PHP’s most popular application development framework.

We’re not going to go too deep into Codacy. We’ll just stick to the essentials of what we need to know so that you get a broad overview of how the service works and how scans work. If you’d like to explore further however, feel free to do so.

First: Import the project into Codacy

To get your project into Codacy, the first thing that you need to do is to import it. This is quite trivial if your Codacy account is linked to either a GitHub or Bitbucket account.

If so, as you can see in the image above, scroll through the list of projects, and check one or more of them that you want to import.

After that, the project’s code will be reviewed, cloned, and analyzed. Depending on the size of the project, this may take a bit of time.

For what it’s worth, I imported a large project yesterday, which took about 15 minutes to prepare. However, I’ve never found the import process to take more than about 3–5 minutes normally.

Second: View The Security Vulnerabilities

Once the project’s completed, you’ll be redirected to the project’s dashboard, which you can see above. As Codacy performs a wide-range of static code analysis, not just for security problems, you’ll see project analysis for a number of those areas.

However, you can see a birds-eye-view security score on the lower left-hand side of the dashboard, just above “Open Pull Requests”.

To view, the security vulnerabilities click “Security” in the left-hand navigation menu. There, you’ll see the 15 categories which Codacy uses to classify security problems; including Auth, CSRF, DoS, SQL Injection, and XSS.

Only categories marked with a red cross contain security problems.

One thing to bear in mind; not all of the security patterns are enabled by default. This can be a good thing, depending on your project’s age. In the example I’m using, which is Laravel, this is not an issue.

You can see that it performs quite well, as you’d expect. However, it does show problems in File Access, Input validation, Other, and Regex. Looking in “Other”, you can see that one issue has been flagged in one file. The other categories only refer to the same file.

On a side note; as a PHP enthusiast, I’m proud that no PHP-issues were found.

Three: View An Issue

Looking at the issue, you can see that it’s highlighted “Generic Object Injection Sink” as the problem in public/js/app.js. Expanding it out further, you can see quite a detailed description of why the issue has been flagged and what it may mean.

It’s worth noting here, again, that highlighted issues may be false positives or false negatives. So you have to be discerning when reviewing issues and determine if they are actually an issue or not.

Four: Fix an Issue

When you’ve found a genuine issue, something which needs fixing, you can begin doing so from the dashboard. Codacy helps you in two ways. Firstly, the dashboard gives you an estimate of the time it would take to fix. In this bug’s case, it’s anticipated that it should take no longer than 15 minutes to fix.

Secondly, you can create an issue in either GitHub or Bitbucket directly from the issue. By clicking “Settings” (the cog icon next to “View File”) ->“GitHub” -> “Create Issue” a popup window appears, which you can see below.

The window contains a default issue title, pre-filled with the identified issue, and a message with some default information, including the message, commit hash, file name, and line number.

From there, you can then resolve the issue with a future pull request, and then rescan the codebase to ensure that the issue is no longer reported.

Wind Up

And that’s been a rapid introduction to static code analysis, how it can be integrated into one or more places in your development tools and workflow, as well as a working example of how to use it to identify (and stay abreast) of security vulnerabilities within your application.

If this is your first time hearing about the concept and the tools, I strongly encourage you to have a look through the supplied list of tools, along with the links in the further reading section.

That way, you’ll progressively expand your knowledge and awareness of how to develop more secure, more robust software. If you’d like to know more about Codacy, feel free to sign up for a 14-day trial, and begin analyzing your applications.

Read more articles on The Codacy Blog.

How Static Code Analysis Helps You Rapidly Fix Software Vulnerabilities was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.

How the $170 million Ethereum bug could have been prevented

Disclosure: Codacy, the automated code reviews platform, has previously sponsored Hacker Noon. For Hacker Noon readers, they’re offering 15% off using this code: HACKERNOON.

Kotlin is a language I thought about using for some time, yet it wasn’t until recently that I decided to give it a try. And I’m so excited that I have!

If you’ve not heard of Kotlin, it’s a language that, while newer than most, has gained significant amounts of traction in recent years. And why not? It has a series of advantages, especially over Java, that makes it quite compelling.

Here’s a short selection:

• It’s developed by an experienced software company, JetBrains.
• It’s a first-class language for Android.
• It’s influenced by Java, Scala, Groovy, C#, Gosu, JavaScript, and Swift.
• It’s interchangeable with Java.
• High Order functions. These are functions that accept functions as parameters.
• Null safety. Variables need to be nullable to able to be set to null.
• Extension Functions. These allow you to extend an existing class, without inheriting from it.
• Type inference for variable and property types
• Data classes: These automatically have equals(), hashCode(), toString(), and copy() functions.

And these are just a small selection. If you want more, definitely check out this list from Android Authority.

However, if you’re still sitting on the fence regarding learning Kotlin, I thought I’d share a series of tips and resources with you, to get you off the fence and learning.

Resources

The Official Kotlin Site

Like most languages, there is an official site for the language, at kotlinlang.org. There, you’ll find a wealth of information and helpful resources, whether you’re a beginner, intermediate, or more advanced Kotlin developer.

There’s Kotlin’s reference documentation, which steps you through from the absolute basics, including basic syntax, idioms, and coding convention, through the entire feature set of the language, such as classes, objects, generics, nested classes, and lambdas.

There’s a range of tutorials that

• Get you started using both IDEs and the command line
• Show how to work with build tools
• Show how to use Kotlin in a range of different environments, such as with Android, Java, JavaScript, and Web Development.

Then there is a list of books, including Kotlin in action and Kotlin for Android Developers, and a host of other resources, including frameworks, libraries, editor and IDE plugins.

After having spent much time on KotlinLang, and performing numerous searches, I’ve yet to find a site that so offers such well organized and comprehensive information on Kotlin as this site does. Other sites, such as those created by individual developers, are great, but nothing tops the depth of information that you’ll find here.

My tip is to either pick one of the getting started guides in the tutorials section or work through the Overview and Getting Started sections in the Reference section. Then, build your knowledge progressively from there.

The Kotlin Slack Channel

The next best resource I’ve found is the Kotlin Slack channel. I begrudgingly accept that IRC doesn’t have the popularity or mindshare that Slack has achieved. And if I could, it’d be IRC all the way.

However, the people have spoken and the Slack channel it is. In the channel, you’ll find channels for discussions for:

• Working with Android.
• Getting started.
• Server and JavaScript development.

Like all good communities, you’ll find helpful advice to questions that people have, links to exciting tidbits, recent happenings, libraries and packages, and a heap more.

The experience I’ve had there has been excellent. Even though I’m relatively new, the community’s been welcoming, knowledgeable, and responsive. I can’t say that for every community I’ve been a part of, or which I’ve attempted to interact.

A Twitter Search for Kotlin

While Twitter doesn’t have quite the reach and popularity that it once did, it can still rapidly yield significant results for technical topics. Just searching on “Kotlin” returns links to a plethora of information, which includes:

• Kotlinconf17
• Code style discussions
• Tooling support, such as Spring, Gradle, and IntelliJ IDEA
• Platform-specific discussions, such as for Android and iOS
• Discussions about language features, such as coroutines
• Accounts of leading Kotlin developers and influencers, such as Andrey Breslav, Kotlin’s lead language designer.

Tips

Now that we’ve worked through a series of top resources let’s look at a set of tips to help you get your Kotlin on.

Grab a (Free) Copy of IntelliJ IDEA Community

While there are a range of IDEs that support Kotlin, such as Eclipse, none has as through support as IntelliJ IDEA, by Kotlin’s core development company, JetBrains.

I might seem biased, as I love using PhpStorm when writing PHP code, and in a way I am. However, even a small amount of googling will return results for IntelliJ than for any other editor or IDE.

There are two versions, Ultimate (commercial and not free) and Community (open-source, released under the Apache 2.0 license). Naturally, there are some differences between the two.

For just getting started, the community version has all the functionality that you will need, such as code analysis, insights, inspections, integrated version control, code completion, and an absolute plethora of plugins.

However, those differences aside, it provides rich functionality, including:

• Integrated tooling for rapidly bootstrapping new Kotlin projects using both Maven and Gradle, of which the latter I’m finding the preferable of the two.
• Handles template project directory creation
• Has integrated Gradle support, so that you can perform any task directly from the UI. You can drop to a terminal if you prefer that approach, as the projects provide gradlew and gradlew.bat, command-line utilities for running any Gradle task.

Do the Koans

Let’s say that you’ve installed a copy of IntelliJ IDEA Community and created a basic project. Now what? Good question. Like any skill, to get better at it, you have to use it regularly, progressively learning every aspect of it.

You have to learn in an organized and focused manner. To help you do that, Kotlin offers Koans. Koans are:

A phrase, dialogue, or saying that a practitioner uses to break down the shell of his or her identity. The koan serves as a surgical tool used to cut into and then break through the mind of the practitioner.

They’re a way of breaking down and learning a skill, and they come in three forms:

1. Kotlin Online Koans
2. As plugins for IntelliJ and Android Studio
3. As a GitHub repository

The online version, which you can see above, offers a list of 42, covering the core language features, starting with the most basic, simple functions. Using them, you can learn in an organized fashion, progressively improving your skills.

That way, you have a clear path from beginner to advanced, you can see how much you’ve learned and how far you’ve come, and you don’t have to search for what to learn next endlessly.

Port an Existing Application

Assuming that you’ve worked through several — or all — of the Koans, and you’re feeling comfortable with your Kotlin knowledge (as well as of its idioms), it’s time to build a meaningful application; something other than the typical Hello World example.

The trouble is, everyone’s needs are different. So what kind of application should you create?
I offer advice from a programming mentor of mine, John Sonmez, when he quoted Douglas Hofstadter:

The only way that we can learn new information is that it’s an analogy to something else.

He suggests that when you port an existing application (preferably one that you’ve created), you learn so much quicker as:

• You don’t need to think about what to develop.
• You can learn the new language more rapidly, as you’re learning it through analogies to an existing one.
• You can relate the language to what you already know and build on that existing foundation

Ideally, I’d suggest that the application which you port is smaller rather than bigger. That way you can gain a sense of success and satisfaction quicker.

However, the application needs to be more than “hello world”, so that you can cover a sufficiently broad cross-section of the new language, and not just grab the low-hanging fruit.

Learn from Existing Projects and Other Developers

The last way that I suggest that you learn Kotlin, is something not specific to Kotlin, but applicable to any software language, it’s by reading through code of existing projects and by learning from other developers. I can’t recommend this approach highly enough!

If there’s one way that propelled my knowledge of a language, it was by working with developers (often far) more experienced than I was.

By seeing how they approached and solved problems, the way that they wrote code, the evolution of their code over time, and asking them questions, I learned far more rapidly than learning on my own.

I’m not going to go into significant depth here, suffice to say that these two approaches are veritable treasure troves of knowledge that you never want to pass up.

Links

Time for one last tip, a set of links to various blog posts that have helped me no end as I’ve been learning.

• Why Kotlin is my next programming language. This post provides a thorough and detailed analysis of why developer Mike Hearn made the move from Java to Kotlin. Despite being written back in mid-2015, the reasoning is still very valid. He discusses why Kotlin’s a good language, as it compiles to JVM bytecode and JavaScript, interchangeable with Java, can use existing Java frameworks and libraries, and has strong commercial support; if you’re keen to get a deep-dive introduction, read this first!
• Why You Should Totally Switch to Kotlin. Written by Magnus Vinther, provides seventeen convincing reasons for switching to Kotlin; including IDE support, better Lambdas, Ranges, Destructuring Declarations, operating overloading, and the When Expression. Afterward, he provides a series of links to other blog posts, some of which I’ve included here, that build on what he started.
• Why I Abandoned Java in Favour of Kotlin. Written by Jan Vladimir Mostert, this post was an amusing read as Jan openly states that he fell in love with Java, yet felt that he had to move on and that Kotlin was where he had to go. Given that strong affiliation with Java, he presents a compelling set of reasons for doing so.
• 10 Cool Things About Kotlin. As the name implies, this is a list post about ten excellent features of Kotlin. A lot of the points are covered in the other links provided. However, what’s nice about this post, is that it allows you to dip your toe in the water, without getting overwhelmed in too much detail. So if you’re short on time, or want to know what the fuss is about quickly, start here.
• Swift is Like Kotlin and From Java to Kotlin. These two sites provide a concise and well laid-out comparison of a range of Kotlin’s language features, and how they would be written in either Swift or Java.

In Closing

Those are my shortcuts for learning Kotlin. I’m not as experienced as other Kotlin developers, but I see that as an advantage, not a hindrance.

I hope that my relatively fresh eyes and enthusiasm have given you a substantial insight into why Kotlin must be on your list of languages to learn.

If you know of other reasons or want to share your experience coming from another language, I’d love to hear it in the comments and to have a discussion with you. So don’t be shy, share your knowledge!

Oh, one last thing, if you’re keen to review the quality of your Kotlin code, Codacy has your back, as they have Kotlin in their list of supported languages. Sign up, login, add your project, and get a handle on its quality.

Read more articles on The Codacy Blog.

Shortcuts to Learning Kotlin was originally published in HackerNoon.com on Medium, where people are continuing the conversation by highlighting and responding to this story.

There have been a spate of high profile security breaches in recent years. These include Ashley Madison, Telegram, and BitFinex, but most significantly Equifax and Yahoo!.

Together, these breaches exposed the records of well over 4 billion private individuals. As you can imagine, in their wake security has been getting a significant level of increased attention.

And so it should. The internet, and the services which run atop it have become so essential to the functioning of modern life. When things go wrong there, they can have disastrous consequences for so many people.

So as software developers we appreciate — more than ever before—that if we create an application which is accessible via the internet, then it is a potential target.

That said, application security isn’t always easy to implement — especially when working for pointy-eared-bosses who place tight time and budget constraints upon us.

But just like testing, security’s something that we have to accept responsibility for, because we’re the ones who write the code.

Luckily for us though, as with using testing tools to help us thoroughly test our code, there are tools available to help us find potential security vulnerabilities. They’re called static code analysis tools.

What is Static Code Analysis?

If you’ve not heard the term before, according to OWASP, static code analysis:

Commonly refers to the running of Static Code Analysis tools that attempt to highlight possible vulnerabilities within ‘static’ (non-running) source code by using techniques such as Taint Analysis and Data Flow Analysis.

These tools, as the description indicates, don’t just find security vulnerabilities. They can also scan for a wide range of other problems, including coding standard violations, unused code, performance problems, and error-prone code.

Regardless of the software language(s) that you use, there are a number of static analysis packages available for it. And in addition to these packages there are a range of online services, often able to scan multiple languages and frameworks.

Here’s a short list for PHP (as it’s the language with which I’m most intimately familiar):

• OWASP WAP: Web Application Protection
• Side Channel Analyzer: Static Analysis for Detecting Side-Channel Vulnerabilities in PHP applications
• psecio/parse: A static scanning tool to review your PHP code for potential security-related issues
• PHP-malware-finder: Attempts to detect obfuscated/dodgy code as well as files using PHP functions often used in malwares/webshells
• progpilot: A static analyzer for security purposes
• Codacy

With all these libraries and services, you’d be forgiven for thinking that static code analysis would be a panacea for your security problems? For example;

1. Write your code
2. Run an analyzer
3. Correct what it finds
4. You’re done!

And in some ways this process is just such a panacea. But it is not without limitations as well; limitations you need to be aware of.

These include:

• Identifying false positives and negatives during software scans. It can be hard to analyze code in the context of a larger codebase. For example, how do you validate all calls to a function and what parameters are passed to those calls? How do you programmatically validate a function which is part of a larger call-chain?
• Difficulty detecting hard to find vulnerabilities, such as access control errors and misconfigurations. It’s common for aspects of an application to be configuration-based. A prime example is configuring access control lists. If the application being scanned is built this way, and a static code analyzer is only checking code, it may report the application as containing no known vulnerabilities, but miss an ACL misconfiguration. What’s more, how can a static analyzer fully understand ACL lists?

These are just two drawbacks to bear in mind when working with static code analysis; there are others. That said, static code analysis is an excellent way to automate the process of identifying an extensive range of security issues, well before your clients and users encounter them.

While not perfect (but what is) it goes a long way to finding issues with your code, that would not normally be detected consistently and predictably.

Static Code Analysis Has Other Benefits

Getting back to the positives, in addition to finding security problems, static analysis tools can also be used to educate developers about security issues and about how to write more secure code.

Well designed ones will not only find valid problems, they will also explain why the problem was identified.

Secondly, they can help the team grow a sense of trust in the security-readiness of the software that they’re producing. They’re not a panacea, but they can still help to know that quality is improving.

Thirdly, they’re automatable, as they can be integrated with your team’s development tools. These include IDEs such as PhpStorm, and development pipelines such as Phing, Ant, Make, Codeship, and Capistrano.

For example, you could integrate WAP as a run configuration in PhpStorm, and run it intermittently. You could also integrate it as part of a Git commit hook.

Integrating them with existing tools and workflows will help ensure that they’re used consistently. As we all know, if something’s too hard to setup or to use, it’s likely going to be avoided—even if it’s the best tool in the world!

One Thing To Remember!

By now, you may be thinking that the essential thing to scan is your application’s code. And you’d not be wrong. But do we write code in isolation? Do we ever write code that doesn’t have external dependencies?

I’d suggest that unless it’s a very small script, then it will always have at least one dependency, if not several. Looking specifically to the language which I spend most of my time with, PHP, I’d say that, more often than not, they’re going to have at least one very obvious dependency — application frameworks.

If you take a look at some of PHP’s top frameworks (Laravel, Zend Framework, and Symfony), you can see that each of them has at least a handful of vulnerabilities, which can lead to exploits. If you’re using Ruby, what about Ruby on Rails? If you’re using Python, what about Django?

This isn’t meant as a hack job on any one of these frameworks. I mention them just to point out that you have to be holistic in your security thinking. Make sure that you check your external dependencies and frameworks to see if they have vulnerabilities which need patching.

A Step-through Example

Now that we’ve had a good introduction to static code analysis and started considering what to scan, let’s work through how to use static code analysis to find and resolve security issues in our code.

In this example we’ll use Codacy to scan the Laravel framework codebase, PHP’s most popular application development framework.

We’re not going to go too deep into Codacy. We’ll just stick to the essentials of what we need to know so that you get a broad overview of how the service works and how scans work. If you’d like to explore further however, feel free to do so.

First: Import the project into Codacy

To get your project into Codacy, the first thing that you need to do is to import it. This is quite trivial if your Codacy account is linked to either a GitHub or Bitbucket account.

If so, as you can see in the image above, scroll through the list of projects, and check one or more of them that you want to import.

After that, the project’s code will be reviewed, cloned, and analyzed. Depending on the size of the project, this may take a bit of time.

For what it’s worth, I imported a large project yesterday, which took about 15 minutes to prepare. However, I’ve never found the import process to take more than about 3–5 minutes normally.

Second: View The Security Vulnerabilities

Once the project’s completed, you’ll be redirected to the project’s dashboard, which you can see above. As Codacy performs a wide-range of static code analysis, not just for security problems, you’ll see project analysis for a number of those areas.

However, you can see a birds-eye-view security score on the lower left-hand side of the dashboard, just above “Open Pull Requests”.

To view, the security vulnerabilities click “Security” in the left-hand navigation menu. There, you’ll see the 15 categories which Codacy uses to classify security problems; including Auth, CSRF, DoS, SQL Injection, and XSS.

Only categories marked with a red cross contain security problems.

One thing to bear in mind; not all of the security patterns are enabled by default. This can be a good thing, depending on your project’s age. In the example I’m using, which is Laravel, this is not an issue.

You can see that it performs quite well, as you’d expect. However, it does show problems in File Access, Input validation, Other, and Regex. Looking in “Other”, you can see that one issue has been flagged in one file. The other categories only refer to the same file.

On a side note; as a PHP enthusiast, I’m proud that no PHP-issues were found.

Three: View An Issue

Looking at the issue, you can see that it’s highlighted “Generic Object Injection Sink” as the problem in public/js/app.js. Expanding it out further, you can see quite a detailed description of why the issue has been flagged and what it may mean.

It’s worth noting here, again, that highlighted issues may be false positives or false negatives. So you have to be discerning when reviewing issues and determine if they are actually an issue or not.

Four: Fix an Issue

When you’ve found a genuine issue, something which needs fixing, you can begin doing so from the dashboard. Codacy helps you in two ways. Firstly, the dashboard gives you an estimate of the time it would take to fix. In this bug’s case, it’s anticipated that it should take no longer than 15 minutes to fix.

Secondly, you can create an issue in either GitHub or Bitbucket directly from the issue. By clicking “Settings” (the cog icon next to “View File”) -> “GitHub” -> “Create Issue” a popup window appears, which you can see below.

The window contains a default issue title, pre-filled with the identified issue, and a message with some default information, including the message, commit hash, file name, and line number.

From there, you can then resolve the issue with a future pull request, and then rescan the codebase to ensure that the issue is no longer reported.

Wind Up

And that’s been a rapid introduction to static code analysis, how it can be integrated into one or more places in your development tools and workflow, as well as a working example of how to use it to identify (and stay abreast) of security vulnerabilities within your application.

If this is your first time hearing about the concept and the tools, I strongly encourage you to have a look through the supplied list of tools, along with the links in the further reading section.

That way, you’ll progressively expand your knowledge and awareness of how to develop more secure, more robust software. If you’d like to know more about Codacy, feel free to sign up for a 30-day trial, and begin analyzing your applications.

How Static Code Analysis Helps You Rapidly Fix Software Vulnerabilities was originally published in Codacy on Medium, where people are continuing the conversation by highlighting and responding to this story.

Docker is a pretty awesome tool, I hope you’ll agree. With it, we can create environments for development, staging, production, and testing — or anything else — with next to no effort.

What’s more, using Docker makes building environments far quicker than it ever was before. Given that, plus a range of other benefits, it’s no wonder Docker’s gained such significant traction and popularity in recent years.

However Docker, like any technology, isn’t perfect. Moreover, one glaring way in which it isn’t is the size that Docker images can quickly grow to become. This point was driven home to me a month or so ago when I stopped to wonder where a significant chunk of my Mac’s drive space had gone.

Matthew Setter on Twitter

Do you have unused #Docker images lying around? Use „docker system prune" to clean them up. I just reclaimed 41GB.

Matthew Setter on Twitter

@jasper_jorna I ended up reclaiming 100GB.

After a quick bit of searching, I found that over 100 GB — that’s right 100 GB! — was being used by Docker images. To be fair, I have been doing quite a lot of experimenting with Docker while I’ve been learning about it. So it should come as no surprise that I’ve built a lot of images which consume quite a bit of space. However, that they took up this much blew my mind!

That’s Why You Should Slim Down Docker Images?

On a local development machine, it might be okay to consume significant hard drive space. After all, if it takes a while you can always grab another coffee or tea, take a break from the screen to rest your eyes, grab some lunch, or discuss something with a colleague.

However, what about when deploying to testing, staging, and production? Or what if you’re pulling down large images to form the base of your Docker images or clusters in your CI/CD pipeline?

While bandwidth and disk space is cheap, time isn’t. If you’re not careful, you’ll be wasting precious (read: expensive) time waiting for images to download as you build and re-build your images and push your deployments.

So today I’m going to step you through five ways in which you can slim down your Docker images, so that you’re spending as little time waiting, and on bandwidth and disc space, as possible.

How Do You Slim Down Docker Images?

That’s a good question. Gladly, the answer isn’t too complicated. I watched a talk from Codeship a little while ago, which advocates several steps, which form the basis of the following five steps. Let’s step through them.

One: Think Carefully About Your Application’s Needs

Do you genuinely think about the application that you’re deploying? Do you fully consider the language, framework, extensions, tools, and third-party packages which it needs? Or do you install everything, such as development tools and Linux headers, before installing your application?

Bear in mind that everything that you install increases an image’s size, as you’d expect. While it may be easier to work this way (some may argue that it is quick and efficient) is it useful — over the long term?

Alternatively, if you stop and consider your application’s needs in-depth, you may be surprised as to just how little it may need. Sure, you don’t get the thrill of jumping in straight-away and building something. Moreover, you might even see planning as somewhat dull and tedious.

However, remember the old saying: “Proper Planning Prevents Poor Performance”. It’s an adage well worth remembering!

Two: Use a Small Base Image

What base image are you using? For quite some time, the default Docker image used Ubuntu, which made sense for several reasons. Let’s consider two of the biggest.

Firstly, Ubuntu’s based on Debian, which uses the Apt package manager. Apt is one of the most well thought out and most feature-rich package managers available. What’s more, its interface is about as intuitive as you can get.

Secondly, Ubuntu is one of the most popular Linux distributions, so its conventions are well understood by a large percentage of developers and systems administrators.

That said, when using Ubuntu as a foundation, your base image won’t be particularly small. Weighing in at around 188 MB, Ubuntu comes packed with most of the tools that you’re likely to need, even if you don’t need them. It’s an excellent example of size being sacrificed for ease of use.

There’s nothing, necessarily, wrong with that, but if you want as small an image as possible, you’re going to have to use a different base image. Luckily, one is readily available and is the new default for Docker images; it’s called Alpine Linux.

Alpine Linux is:

a small Linux distribution based on musl and BusyBox, primarily designed for “power users who appreciate security, simplicity and resource efficiency.

As a result, an image based on Alpine Linux is typically around 5 MB in size. That’s a size saving of around 97%!

Compare that to Ubuntu’s size and you can see how making this one change alone is a significant saving. To make it easier, here’s an image size comparison, from Brian Christner:

See what I mean?

However, changing the default base image isn’t without some cost. The first is that Alpine Linux doesn’t provide much functionality out of the box. As a result, you’re going to have to better understand what your image needs and the dependencies upon which it relies.

The second is that Alpine uses APK instead of Apt. APK, while a little different from Apt, follows similar conventions to other Linux package managers, such as Yum, ZYpp, and Portage. While I’ve been using Apt for over ten years, transitioning to APK wasn’t that difficult, as its commands are pretty intuitive.

That said, if you have scripts for building images, you’ll have to refactor them to use APK commands and package names instead, which will take a little bit of time. However, it should be a, relatively, straight-forward process.

Three: Use as Few Layers As Possible

How many layers compose your Docker image? Layers?, you may be asking. If you’re not familiar, every Docker image is composed of several layers, one for each command in a Dockerfile (and those in its base image).

FROM php:7.0-apache

COPY ./ /var/www/html/

RUN chown -R www-data:www-data /var/www/html \
 && chown www-data:www-data /var/www
RUN apt-get -y update
RUN apt-get install -y libmcrypt-dev libzip-dev libpng12-dev libicu-dev libxml2-dev wget build-essential
RUN apt-get install -y git vim
RUN apt-get install -y npm nodejs-legacy

In the example Dockerfile above, the resulting Docker image will be composed of 7 layers, in addition to any layers in the base image. Building Docker images in this way allows Docker to be as efficient as possible.

If a change is made in one layer, then only that layer needs to be changed. Every other layer can be fetched from cache.
That way, build times are as quick as possible.

To know how many layers your image is composed of, and how large they are, run the docker history command, as in the following example:

docker history ze-php7.1.11-alpine
IMAGE CREATED CREATED BY SIZE COMMENT
5e0644e052e4 7 days ago /bin/sh -c #(nop) ENTRYPOINT [“/bootstrap… 0B
6791d155bd2e 7 days ago /bin/sh -c #(nop) EXPOSE 80/tcp 0B
6db0e80c9902 7 days ago /bin/sh -c #(nop) COPY dir:67f62ca0a16622f… 12MB
b53ae8739f99 7 days ago /bin/sh -c chmod +x /bootstrap/start.sh 10.1kB
9fe7fa43babf 7 days ago /bin/sh -c #(nop) ADD file:9fef33a371bdaef… 10.1kB
70949f1e3e70 7 days ago /bin/sh -c mkdir /app && mkdir /app/public… 0B
969f39e455b7 7 days ago /bin/sh -c mkdir /run/apache2 && sed -… 17.8kB
6cfc7b7d4a34 7 days ago /bin/sh -c apk — no-cache update && apk … 70.6MB
946e205f9ff5 7 days ago /bin/sh -c echo “http://dl-cdn.alpinelinux… 149B
7bb295030e89 7 days ago /bin/sh -c #(nop) LABEL maintainer=Paul S… 0B
72dfe9e82e16 3 weeks ago /bin/sh -c #(nop) CMD [“/bin/sh”] 0B
<missing> 3 weeks ago /bin/sh -c #(nop) ADD file:682bfba4bfda1ed… 4.15MB

There, you can see that it’s composed of twelve layers. For each layer you can see:

• The layer’s unique hash.
• When that layer was created.
• The command that created it.
• The layer’s size.

If you want to see an image’s combined size, use the docker images command, as in this example:

docker images ze-php7.1.11-alpine
REPOSITORY TAG IMAGE ID CREATED SIZE
ze-php7.1.11-alpine latest 5e0644e052e4 7 days ago 86.8MB

Then there’s the next aspect of how layers work. Referring again to the Dockerfile example above, notice the last three RUN commands. I’ve deliberately split up three calls to apt-get install to highlight this point.

It first installs the required development packages, which subsequent packages depend on. It then installs several system binaries. Finally, it installs the application’s software packages.

Working this way might make sense during development, as the Dockerfile’s easy to read and is ordered logically. However, while it might be logical, it will result in a larger image than if those steps were combined.

Moreover, gets even worse. Do you install packages in one RUN command only to remove them after a following RUN command? You may wonder what the problem is, as the packages are ultimately deleted. Well, no they’re not.

They’re deleted in a later layer — but they still exist in the one where they were installed! You see, anything that exists in one layer isn’t removed in a later one, it’s only covered over, so that following layers can’t see it, and that it appears as though it doesn’t exist.

It gets back to the point about caching that I spoke of earlier.
That’s why:

1. It’s better not to install a package in the first place if you don’t need it.
2. Don’t install a package in the first place if you can avoid it.

How we could do this? Well, here are four ways:

1. Develop the Dockerfile in logically separated blocks, but compact it in the final version.
2. Don’t create files if you don’t have to — use streams and pipes as much as possible.
3. Remove unnecessary files, such as cache files, or don’t use the cache in the first place.
4. If you have to install files, use the smallest one possible.

Let’s see some of those in practice:

# Add basics first
RUN apk — no-cache update \
 && apk upgrade \
 && apk — no-cache add apache2 ca-certificates openntpd php7 php7-apache2 php7-sqlite3 php7-tokenizer \
 && cp /usr/bin/php7 /usr/bin/php \
 && rm -f /var/log/apache/*

Here, you can see that I’ve combined the cache update and upgrade commands with the add command, by doing so, it’s not downloaded any extra files to the filesystem. Then, those commands were combined with the cp and rm command, which took care of some housekeeping and cleared out any unnecessary files.

Do some experimenting and see how you can compact your Dockerfile configurations.

Four: Use .dockerignore files

Referring back to the earlier Dockerfile example again, remember that it copied everything in its context into the images /var/www/html directory, in the second command? Are all those files necessary? Could some of them have been avoided?

If so, and it’s likely the case that some could have been, then do so by using [a .dockerignore file. These files, as their name implies, instruct Docker that specific files can be ignored.

If you’re familiar with .gitignore files, then you’ll know what I mean. If not, have a look at the example below.

vendor
.idea
data
mysql

Using this configuration Docker would not copy any files in the vendor, .idea, data, or mysql directories of my project, saving both time and space in the process. .dockerignore files are convenient, and I strongly encourage you to become a whiz with them.

Five: Squash Docker Images

The last tip I want to offer is on squashing Docker images.
The idea here is that after your image is created, you then flatten it as much as possible, using a tool such as docker-squash.

Docker-squash is a utility to squash multiple docker layers into one to create an image with fewer and smaller layers. It retains Dockerfile commands such as PORT, ENV, etc.. so that squashed images work the same as they were originally built. In addition, deleted files in later layers are actually purged from the image when squashed.

The idea is that your image is as small as possible so that when it’s transferred, it’s as quick as is possible. However, while the concept is quite appealing, I was never able to use it effectively.

Perhaps it’s something to do with running it on macOS versus on a Linux distribution, but despite leaving it running for over an hour on a small image, there seemed no end in sight as to when the process would complete.

Given that, while using docker-squash might drastically reduce my Docker image sizes, saving precious time pulling down and pushing up images, this time saving seems outweighed by the time to squash the image in the first place.

However, your mileage may vary, and I’m entirely new to the tool. So if you’ve found success with it, or a related tool, please share your thoughts in the comments.

In Conclusion

Those were five steps that you can take to reduce the size of your Docker images. One or a combination of them will reduce the size of your Docker images.

Some of them will reduce image sizes more than others, so always compare the size before and after using each one.

Also, the size reductions of each step is heavily dependent on some of the others and what your image needs to do. So please keep that in mind.

With that said, I encourage you to foster a spirit of healthy competition in seeing just how small, just how compact your Docker images can become.

I’d love to see us return to earlier times when we used as few resources as possible before disc space and bandwidth became so cheap, and some of us became quite lax and stopped caring.

Perhaps, together, we can encourage others to follow our lead.
I hope so. That said, try out each of the five tips, and share your progress in the comments.

If you have others which you use on a regular basis, please share them. I’d love to learn from you as well.

Five Ways to Slim Your Docker Images was originally published in Codacy on Medium, where people are continuing the conversation by highlighting and responding to this story.

The reason that I ask that I’ve been tearing through recently. As a result, I’ve gained a new lease on life as a technical writer — and as a developer.

This isn’t to say that I didn’t have a strong professional work ethic or solid experience prior to reading the book. It’s that since I’ve begun reading it, I’ve felt so much more confident in how I undertake the role than I have been up until now.

Specifically, I only document and approve PR’s about topics that I’ve personally tested. If I’m not sure they work, they don’t get my approval.

This might sound strange

However, like unit testing or security best practices in software, sometimes you can feel under pressure from deadlines to get things done quicker than you should, effectively rushing things through without being 100% sure that they work.

So it was recently when I was going through the manual installation section of the ownCloud administration documentation. I started a quick run through of the steps outlined in response to a new issue only to find that:

1. Some details were missing.
2. Some were wrong (likely outdated).

These omissions and errors meant there was room for doubt and error — especially for newer ownCloud users.

What’s more, while code samples are very helpful (you don’t have to figure out what you have to type) they still leave so much work for the user, as they have to copy and paste the code examples into their terminal manually.

Let’s Make The User’s Life Easier with a Script

As we’re attempting to help them save time and effort, why not provide a script that they can use as part of a build process, one able to be scheduled via Cron?

So despite (or in spite of) the factual errors and omissions, I became excited at the prospect of revising the documentation. Now I could have raised an issue with the core development team to create the script, detailing what it should do when it was ready.

However, I’m not only a technical writer, I’m also a software developer. And what do developers love to do more than most else? Design and write code!

So it was that I started designing a shell script that would automate the process of installing all of ownCloud’s dependencies.

The Script Outline

The first thing to do was to assess what it needed to do and the environment in which it would run. As I already knew what it had to do, using the documentation as my guide, I moved on to assessing the environment requirements.

Currently, ownCloud officially supports several Linux distributions. These are:

• Ubuntu 16.04
• Debian 7 and 8
• SUSE Linux Enterprise Server 12 and 12 SP1
• Red Hat Enterprise Linux/Centos 6.5 and 7

So when the script was finished, it had to achieve the same outcome, regardless of which distribution it was run on.

A bash shell script seemed to make the most sense. I could have written it in Ruby, Python, or PHP. However, I always associated shell scripts with SysAdmin & DevOps work. What’s more, I had an itch to scratch!

Here’s an admission: I’ve been a bash script hacker since 1999. But I’ve never actually developed my proficiency past a certain point.

So I saw this as an opportunity to grow my skills and learn more about bash while indulging one of my oldest, technical, passions: Linux. On top of that, I’ve long been curious about how the different distributions organise themselves.

And so the decision was made. Then came the next question:

How would I make it portable, yet not spend more time than was necessary provisioning the different Linux distributions?

VirtualBox, Vagrant, and a provisioner such as Ansible seemed out of the question. If I went down that path, I’d likely spend more time writing the provisioning scripts than writing the actually setup script — which was my main focus!

So I chose to go with Docker instead, as it demands only a limited amount of time and effort to get an environment up and going.

Given that, I created a custom Docker setup, [based largely on an existing project](), which [you can find on GitHub](). In it, you can see that it uses Docker-Compose to create a two-container setup.

There’s a web service that provides Apache 2 and PHP 7. And there’s a MySQL container that, surprise, provides MySQL.

The web service uses any one of three Dockerfiles, which are based on a different base image. There’s one for Ubuntu, openSUSE Leaf 42.3, and CentOS 7.

Each of them installs a set of packages, sets up a user & group, and sets up some permissions on a required directory so that that user can access it.

While they don’t do a lot, they’s still essential. By using them, I was able to write a shell script that achieved the same outcome across each distribution. The script doesn’t actually do a lot. But, here’s a quick summary:

1. Check that the user isn’t root.
2. Make sure the directory is readable and writable by the web user.
3. Install the required packages.
4. Run the installer.

Doing so taught me:

• What could stay the same
• What had to be changed
• Other changes in the environment

Interestingly, the key step was determining which distribution was being used, so that the script knew which package manager to use to install the required packages, and what the packages were called.

This was done with two functions: which_distro and install_required_packages.

Function 1: which_distro

function which_distro()
{
  case “$( grep -Eoi ‘Debian|SUSE|Ubuntu’ /etc/issue )” in
    “SUSE”)
    echo “SUSE”
  ;;
 
  “Ubuntu”)
    echo “Ubuntu”
  ;;
  
  esac

  redhat_release_file=/etc/redhat-release

  # Need to do a bit more work to detect RedHat-based distributions
  if [ -e “$redhat_release_file” ]; then
    case “$( grep -Eoi ‘CentOS’ $redhat_release_file )” in
 “CentOS”)
    echo “CentOS”
  ;;
  esac
 fi
}

The function first greps /etc/issue for one of Debian, SUSE, or Ubuntu. If it contains one of the three strings, then we know its that distribution. I did this because I’ve found that these distributions consistently show themselves there.

Determining if the distribution was RedHat or CentOS was a bit harder, as these two don’t always store the identifying information consistently in /etc/issue.

They can store it there. But they can also store it in /proc/version as well as /etc/redhat-release. /etc/redhat-release seems to be the most consistent approach.

Given that, if /etc/redhat-release is available, then we know that one of the two distributions is being used. From there, the script greps for which of them it is, similar to the previous approach.

Function 2: install_required_packages

function install_required_packages()
{
  case 

    "$( which_distro )" in "SUSE") 
      echo "Installing required packages on SUSE"
      install_required_suse_packages
    ;;
 
    "Ubuntu"|"Debian") 
      echo "Installing required packages on Ubuntu/Debian"
      install_required_ubuntu_debian_packages
    ;;

    "CentOS") 
      echo "Installing required packages on Centos"
      install_required_centos_packages
    ;;

  esac
}

The second script just uses which_distro to determine which distribution is being used, and then:

1. Provide visual confirmation to the user about which distribution it’s going to install the packages for.
2. Calls a distribution-specific function to install the required packages.

Now for the more interesting part, the distribution-specific installers.

Ubuntu/Debian

I started off with Ubuntu/Debian. I’ve got the most experience with them as I’ve been using them since about 2003. What’s heartening is that it was the distribution that required the least amount of effort.

I know a number of the idioms and quirks, so perhaps I’m biased. But it took the least amount of effort.

function install_required_ubuntu_debian_packages()
{
 sudo apt-get -y -q update && \
 apt-get install -y -q wget make npm \
 nodejs nodejs-legacy unzip git
}

openSUSE

I then refactored the script to work with openSUSE. To be honest, while I live in Nuremberg, the hometown of SUSE I think, I’ve barely used it.

What’s more, it took the most amount of effort to code. While the changes largely only reflect using Zypper instead of Apt, it took some experimenting to both get the base environment working and to find the combination of packages and dependencies.

function install_required_suse_packages()
{
 sudo zypper — quiet — non-interactive install \
 wget make nodejs6 nodejs-common \
 unzip git npm6 phantomjs
}

Centos

Finally, I added an installer for CentOS. While it looks quite large by lines of code, it took less effort than openSUSE. It took time to figure out how to get PhantomJS up and going to be fair, but not all that much.

function install_required_centos_packages()
{
  sudo yum update -q -y
  sudo yum — enablerepo=cr -q -y install \ 
    wget make nodejs unzip git npm bzip2 file

  # Install PhantomJS — see https://www.bonusbits.com/wiki/HowTo:Install_PhantomJS_on_CentOS
 
  # It’s not in the official repos, so needs to be installed independently.
 
  sudo yum install fontconfig freetype \ 
    freetype-devel fontconfig-devel libstdc++
 
  sudo mkdir -p /opt/phantomjs
 wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-linux-x86_64.tar.bz2
 
  sudo tar -jxvf phantomjs-2.1.1-linux-x86_64.tar.bz2 \
    — strip-components 1 — directory /opt/phantomjs/

  sudo ln -s /opt/phantomjs/bin/phantomjs /usr/bin/phantomjs
}

What About RHEL?

At this stage, I’ve not completed refactoring the script to work with RHEL.
I expect to have that done later in the week.

What Was Learned

It’s been an interesting journey building a shell script that works across multiple Linux distributions. I learned some things, including:

• How distributions don’t necessarily do the same things in the same way.
• How some packages are readily available across each distribution, while others you have to find and install all on your own.
• How by using Docker, I wasted almost no time building each distribution. This cemented for me why it’s my preferred approach to setting up and deploying software.

While it’s frustrating that each distribution doesn’t provide the packages that each other does, it makes sense. They’re created by different people, to serve different audiences and needs.

It makes no sense for them to be identical. And it’s something to keep in mind when you’re writing shell scripts. It may save you a lot of confusion and frustration.

And that’s been a whirlwind run through of both how Docker’s helped me create a shell script that works across multiple Linux distributions, as well as a bit of a step-through of the relevant sections of the script.

If you’re a bash expert I’d love to know how you would improve or change the script. I’d love to know if there are better ways, quicker ways, easier ways to do it.

Please leave your feedback in the comments or the discussion of the script on GitHub.