An old friend messaged last week with a great question: “What is spatial data science (as opposed to regular data science) for us uninitiated? And why is it significant?”

What is spatial data science?

Spatial data science is the practice of distilling insight from spatial data using algorithms and analytical techniques.

Spatial data is any data where the relative positions of observations are described by the data, and can be used as a dimension in the analysis. Put another way, spatial data has information about where each individual datum is — and therefore, where the observations are in relation to each other.

The most intuitive example is geospatial data, which carries information about where things happen on Earth. Geospatial data can describe natural or human subjects like topography, political boundaries, urban systems, weather and climate patterns, road networks, distributions of species, consumption habits, shipping and logistics systems, demographics and so on.

The spatial dimensions usually are measurements of latitude (the y-coordinate) and longitude (the x-coordinate), and sometimes altitude (the z-coordinate), which can place a point precisely on, above or below the Earth’s surface. GIS skills and tools (geographic information systems / science) are often used by spatial data scientists to manipulate, analyze and visualize geospatial data.

It’s worth noting that many spatial analysis techniques are actually agnostic to the scale — so we could apply the same algorithms to a map of the Earth as we could to a map of a cell, or a map of the universe. And spatial data science techniques can be applied to more abstract problems with a spatial element — analyzing how closely associated words are, for example, by working out how often they are found together.

Raster and Vector

Spatial data typically falls into two categories: raster and vector. Both are ways to describe space and represent features, but they work quite differently.

Raster Data

A raster is a “grid of regularly sized pixels”. By assigning each cell in the grid a value — or a few values — images can be described numerically, as multidimensional arrays.

For example, take a 3x3 grid that looked like this:

If 1 means black and 0 means white, we could represent it numerically like this:

img = [[ 1, 0, 1 ],
       [ 0, 1, 0 ], 
       [ 1, 0, 1 ]]

The numbers in raster cells can mean lots of things — the altitude of the land or depth of the sea at that specific point, the amount of ice or snow on that point, the number of people living within that pixel, and so on. Further, just about any color in the visible spectrum can be described by a combination of three numbers representing the intensity of Red, Green and Blue (RGB) — satellite images are raster data structures. GeoTiff, jpg, png and bitmap files contain raster data.

Vector Data

Vector data is a bit more abstract. In a vector dataset, features are individual units in the dataset, and each feature typically represents a point, line or polygon. These features are represented mathematically, usually by numbers that signify either the coordinates of the point, or the vertices (corners) of the geometry.

Points, Lines, Polygons

As a quick example, here’s a bare-bones numerical representation of each of these types of features:

point =   [ 45.841616, 6.212074 ]

line =    [[ -0.131838, 51.52241 ],
           [ -3.142085, 51.50190 ],
           [ -3.175046, 55.96150 ]]

polygon = [[ -43.06640, 17.47643 ],
           [ -46.40625, 10.83330 ],
           [ -37.26562, 11.52308 ],
           [ -43.06640, 17.47643 ]]
           // ^^ The first and last coordinate are the same

Vector features will often have some metadata included that describes the feature — the name of a road, say, or the population of a state. These extra, non-spatial metadata of a feature are usually called “attributes”, and are often represented in an “attribute table”. Very often spatial data scientists will combine the spatial dimensions (coordinates — for points, or coordinate arrays — for lines and polygons) with non-spatial dimensions in their analysis. GeoJSON and .shp files commonly contain vector data.

Why is this different from regular data science?

The short answer is that it isn’t: spatial data science is a discipline within data science. However, spatial data have some characteristics that make them need special treatment. This is true in the way the data is stored and handled from a programming / database perspective, and how it is analyzed from an algorithmic perspective. That means spatial data scientists have to learn some concepts — mainly from geometry, like representing 3D shapes on flat surfaces — that other data scientists might never have to deal with.

Tools for Spatial Data Science

Spatial data scientists try to make sense of these spatial datasets to better understand the system or phenomenon they’re studying. Some incredible (and often free) software tools make this possible. Most programming languages like Python, R and Javascript have amazing spatial analysis libraries like geopandas and turf.js, and desktop programs like QGIS make visualizing and analyzing spatial data accessible to less technical people. There are also powerful online tools like Mapbox, Carto and Google BigQuery to help with these analysis and visualization challenges. JavaScript libraries like Leaflet and Mapbox GL JS enable web developers to create interactive maps in the browser.

A few examples

Spatial data scientists might have the task of analyzing spatial distribution — to see if points cluster together, are spread out, or are randomly placed — to, say, work out the best place to build a new airport or retail center, or understand patterns of violence or crime.

It could entail analyzing trends through time — by seeing how a certain district’s voting outcomes evolved, or how sentiment toward some issue changed in different parts of a country.

Maybe analysts are analyzing satellite imagery to map unmapped areas to help deliver emergency services more efficiently, or work out how shady or sunny a new potential building site or bike route is. It might mean calculating the most efficient route to get from A to B given current traffic conditions. For as niche as the field is, spatial data science has a breadth of applications across just about every sector and field.

By analyzing spatial data with this specialized statistical toolkit, spatial data scientists are able to better understand spatial relationships, and, possibly, work out why things happen where, and predict where things will happen next.

Spatial data science for the uninitiated was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Of course, this is an enormous challenge, and extends beyond the scope of what is possible in a single masters dissertation. My advisor’s suggestion was to identify key steps to understanding such a system, and choose the highest impact (or most interesting) one to focus on for these next few months. The first one is technical: what is possible, and how? At the moment I’ve been diving down the rabbit hold of understanding the decentralized public key infrastructure, a concept put forward by Allen et al in 2015 addressed many of the problems inherent in the centralized trust systems the internet relies on today (certificate authorities, DNS registrars etc.). This is challenging some of my assumptions about the nature of authority, and how we can trust others in an environment where we might not know who they are, and where there are so many that it is infeasible to know them all. It seems as though a sort of Copernican revolution is imminent, in which we realize the ground we are standing on re: trusted authority may not be the absolute ground, but simply a local one … I’m also digging into Elinor Olstrom’s work on governing common resource pools (as well as Allen’s extensions of these ideas), and trying to understand how the informational commons differs from the physical one. For example, use of a physical resource diminishes the availability of that resource to another; in the informational space, this is not necessarily true. This is becoming my core interest: governing the informational commons in the context of decentralization.

Coming back to the technical mechanism, questions that need answering include: Is a secure decentralized data management system possible? How can devices share data in the context of decentralization? Can privacy be respected (device identity / ownership) if the public key infrastructure is implemented on a public blockchain? Can the system ensure the security of the information handled—i.e. end-to-end encryption? How will data producers and requestors authenticate themselves? Can all of this work in the IoT context, which is to say, on devices that are resource-constrained? How might the system scale?

I have not yet filled in the details here, but my approach will be to explore how access control to connected sensor data can be managed. Each of the aspects here address one or more of the requirements or challenges implicit in an informational governance system lacking a centralized authority.

Self-sovereign identities, including machine autonomy — i.e. private keys unique and private to the edge device. This relies on trusted hardware technology, which enables devices to sign transactions — vote / submit data to DAOs, and hold funds etc, and decrypt data only accessible to it.

Proxy re-encryption technology, to enable secure data distribution to many requesetors given resource limitations of edge devices (i.e. one that cannot re-encrypt data for each validated requestor). Possibly homomorphic cryptography for highly sensitive data.

Adherence to standardized data protocols, enabling interoperability of data captured by devices manufactured and managed by different firms / entities. This is crucial to unlocking the value of the data being captured, but questions remain around incentivizing participants to adhere to such a protocol, and to share their data.

Compliance with existing legal structures and regimes. Ricardian contracts (smart / legal contract pairs) offer the chance to create durable bindings between the legal and software systems, though the complexity of governing physical-informational entities, especially transnational ones like ships or shipping containers, is absolutely enormous. Legions of lawyers with technical competency to understand the computing concepts requisite to understanding the nuance of blockchain / distributed ledgers will be required to design compliant systems. And I’m realizing more and more the simultaneous difficulty and importance of crafting new international policy that optimizes between innovation and safety as we construct these informational infrastructures, AND that can adapt as technology inevitably evolves : adaptive governance (as discussed by Dietz, Olstrom and Stern, 2003).

Other questions remain, especially around incentivizing data creators to participate in such a network, understanding conditions for access and that each entity requesting access will be subject to different conditions, delegating access rights, scalability, and so on. The effort is worthwhile, though, if it might help us coordinate in building situational awareness, thereby enabling us to act as better stewards of the planet.

The Internet of Things holds the promise of improving our collective awareness of the world. Sensors are installed on web-connected devices that are being deployed or installed in places that have hardly been observed; even well-studied locations now are experiencing a greater density and frequency of empirical observation than ever before. But immense questions regarding privacy, security, the role of government and the rights of commercial entities and the public are being raised as we foray into the connected future.

I believe that unlocking the information being captured for the public benefit is a moral imperative. This is an effort to understand technical, ethical, economic and political considerations in creating such a system. My hope is that by being thoughtful about this now we can maximize the value captured for the public good, and minimize the risks to our freedom, privacy and security.

Informal in nature, this is my notebook for putting these ideas out, where someone might come across them and engage. These are not fully fleshed out; rather, they’re snippets, unfinished trains of thought, a drawing board. My goal is to bring them to light rather than to put out polished posts, so forgive me if the writing is disjointed or ideas incomplete.

Case Study: Smart Shipping Containers

Given the heterogeneity of the Internet of Things, this research effort will focus on exploring these considerations as applied to transport and logistics networks. The second half of the 20th century saw the rise of a standard unit of shipment: the twenty-foot equivalent unit (TEU). Shipping containers adhere to this standard, and compose a significant proportion of global goods transport. Designed for intermodal transport, shipping containers can hold goods as they move from origin — point of extraction or assembly — to destination — point of sale, including by sea, road, rail and, less frequently, air. (For more on this, see The Box by Marc Levinson).

Improved situational awareness for stakeholders in the global transport supply chain would create numerous opportunities to improve system efficiency, potentially resulting in commercial and environmental benefits (UN CEFACT Smart Containers White Paper 2019). A transparent and reliable mechanism to ensure the security of the smart container’s meta-information is a requirement if such a smart logistics network is to be implemented.

Capturing the value contained in sensor data is the primary incentive for participants in the logistics network ecosystem to invest in smart container technology. Unless regulated to do so, firms — compelled by a profit motive — will need to see the business case for such an investment. Improvements to operational efficiency may offer enough justification to forward-thinking or large actors in the space, but a clearer path to adoption would be to create new revenue opportunities, rather than increase profits by reducing costs. The clearest revenue opportunity created by investing in smart container technology — beyond possibly up-charging for carrying freight monitored by sensors — is to monetize the data collected by these sensors.

This opportunity is constrained, however, by technical and business realities. Edge sensors are highly limited by intermittent or low bandwidth, and by constraints to computing capacity due to storage / memory or energy consumption. It is infeasible to transmit all of the data collected by edge sensors to cloud servers, where they could be warehoused (at a cost, however small). Some blend of edge computing, aggregation and sampling will be necessary to retain the value of the information collected within the technical constraints of the system.

A standard data format

Machine learning tasks often are most successfully performed on structured datasets, with cross-comparable observations. This forms the foundation of the case for the establishment of and adherence to an industry-wide protocol for smart container data (as referenced in the UN CEFACT white paper). By ensuring that data collected — even from a single sensor at a single point in spacetime — is interoperable with data collected by competing entities will improve the global value of the data (from an informational perspective — perhaps not from an economic one). Larger players in the space will have much less incentive to do this, and to open their data, than smaller ones. Part of this enquiry will explore ways to incentivize participation, and / or spell out the case for regulating device owners to do so.

The motivation is our understanding that the emergence of a network of connected sensors is creating at once immense opportunity and risk to issues of human and environmental justice. Smart container data could provide information about environmental conditions around the world at unprecedented levels. Such data, combined with machine learning methods (supervised classification, perhaps) might be used to detect illicit actors utilizing licit commercial logistics networks to traffic contraband or humans. And, questions surrounding the systems for observing shipping containers also apply to those surveilling public and commercial spaces — cameras, satellites, biometric devices, monitors of internet traffic / social networks, etc. These technologies offer the opportunity to improve societies’ physical security through early detection of threats to public safety, but carry the risk of abuse by incompetent, unscrupulous or malevolent authorities. Our hope is to help maximize the former and minimize the latter.

This is an informal outlet for the process that lies behind this research effort. We invite any questions, discourse or critique to our thinking, code, etc. 🙏

I have been following along with the Brexit proceedings, from Colorado and, these past months, from here in London.

I was opposed to the idea of the United Kingdom leaving the European Union, and would have voted Remain if I had been entitled to vote. This is because I support openness and inclusiveness in the way we conduct our affairs, and would rather see greater connectivity and reciprocal reliance than less. In time I have come to understand a bit about the situation’s complexity, some of the many valid reasons people voted to leave. The truth is that our world is an incredibly elaborate intersection of interaction from moment to present moment. Many people in Britain do not feel better off than they were years ago, and for many of them the UK’s membership in the EU was among the causes. I can understand the impulse that I might be better off on my own. I certainly can agree with much of the underlying reasoning, that what we have does not appear to be working very well and has deep flaws. Changing it, improving it, is a noble aim.

On June 24, 2016 I woke up in Denver to learn — by the arrangement of pixels on the screen of my phone — that the people had voted to leave. I was surprised and concerned, but not long after I began to consider what opportunities might arise along with — because of — the disruption to our global system the UK’s action would entail, and how this act of isolation might still serve us as a global species.

Britain — and especially London — is at the core of the global system. Politically, sure, but more so of the global capitalist system, the world of business and finance, which defines so much of what we produce and consume. This is among the reasons that the UK has served as a creative, intellectual and commercial capital of the world through the past centuries into the present. As such, the UK is a hub of human flourishing, a place where some of the most brilliant and gifted instances of human aggregate, live and travel to learn and create and meet and enjoy and work. The UK a locus of power, a place where decisions are made and ideas conceived that affect billions of people and the world that we live in. By adopting this role, and especially by benefiting from it, Britain bears some responsibility for the effect it has globally.

The Challenge

In so many ways Brexit is a disaster for the UK, Europe and the world. It will carry enormous costs — economic ones, but also human, political, environmental — costs that may never be fully known, that will echo and reverberate in humanity’s story for a long time to come — centuries — as long as humanity exists. No person on Earth will be entirely unaffected. I say this because we now realize we live on one globe. The vast majority of people on earth interact in, or on the edges of, a global marketplace to survive. We are all connected by the air we breathe and the water that sustains us, the ground we stand on, the ways in which we communicate. This is undeniable.

I had hoped that the British political leadership could make the most of this disruption, communicate a clear vision in the uncertainty. Unfortunately the government has apparently been unable to do so, and Brexit Day has arrived without those elected to lead their country having found agreement. It still appears there is no coherent vision for how the country might move forward through these next months and years in a self-possessed, fair, just and inclusive way.

The Opportunity

I believe Brexit is the opportunity of the century for Great Britain and Northern Ireland. It is a chance for the country to define a new role for itself as the world order evolves. It is a chance to ensure relevance (and, as a result, wealth), a chance to set an example, for bold and forward-thinking policy makers to make the changes that their system needs to continue to serve its purpose. As the status quo is shattered it could be remolded to more capably assure that we protect and preserve the environment for future generations. Britain could lead by example in ensuring that all people alive are taken care of, and that the industrious and resourceful can create the opportunities they deserve. By sculpting the volatility inevitable in times of transition, British and EU political leaders might correct some of the profound structural injustices perpetuated by the global system in its current configuration. I don’t feel that British policymakers are personally responsible for these injustices, but they are in a position to address them. They have volunteered to be responsible. Inaction may be a form of complicity.

Make no mistake, I understand that this would require an inordinate collective effort, a sudden shift in the way the people involved engage with one another … and here perhaps my delusion is exposed … but I think they are capable of it. Leading sometimes means placing the welfare of the led above one’s own. British MPs were elected to lead the country — all people in the country. It should be apparent to them that if they don’t come together and find consensus their constituents will suffer — they will suffer. And — what an opportunity to make history, to define a legacy. My aspirational view of our capacity to adapt the way we work together extends beyond the political class: it is an effort I am undertaking in myself, and I see early indicators of across society.

A few ideas

I don’t have all the answers, though I do have a few ideas to offer. I should acknowledge that these have very little to do with the UK’s membership in the EU, and are not why most Leavers voted as they did. But emergent opportunities are opportunities are opportunities nonetheless. Here I offer three areas especially deserving of focus, major steps on the critical path to a new state of being. These are collective problems rooted in our history — in many ways no one alive is responsible for them, and at once we all are.

Illicit Capital

As part of being a key player in the cultural, political and (especially) commercial history of the world, London plays a role in the perpetuation of some of the greatest ongoing injustices in the world. By this I am primarily referring to the role played by financial institutions within British jurisdiction in providing haven for illicit capital. This money is illicit because we all agree it should not be legal to earn a profit through the practices these people engage in. It causes profound harm at scale, and is not fair to honest actors. As a young person in the world, I feel the need to raise my voice and ask: please, do whatever you can to stop this needless suffering. It is not tenable. We have enough for everyone to survive. Competition is incredibly valuable, a powerful tool for human motivation, but should never result in intentional human violence, certainly not so someone can make some money. Britain has never had a better chance to change the role it plays in enabling malevolent and unethical actors to scale their profits and the accompanying harm.

Drug Policy

This also is an opportunity for Britain to take a leading role in a conversation that we have needed to have for a long time, on the transition to a humane and evidence-based drug policy. The global prohibition of recreational drugs other than alcohol is, in my view, perhaps the most harmful policy ever instituted by men still in effect today. The Drug War causes layer upon layer of profound suffering every day. If deepens despair amongst those who feel the urge to use drugs, sharpening their loneliness and exposing them to the harmful additives dealers add to their product to increase their profits. It exposes people to the violence that results when profit seekers are forced to enforce contracts in an extrajudicial manner. And, it provides organized criminal networks billions in annual revenue to continue causing harm on an unconscionable scale. The longer it takes for us to break this entrenched aspect of our system, and finding something that serves us better, the more difficult it becomes to overcome the resultant violence, suffering and injustice.

Closely related, the UK government could use Brexit as an opportunity to reform many of the services it provides to its citizens and residents — especially mental health services. In six months living in London I have observed a major (and, evidently, rising) proportion of the population — especially young people — repressing their interior pain, turning to drink and drugs and sex and screens to avoid the difficult and lonely process of introspection, developing mindful and healthy life habits, admitting one’s flaws, identifying one’s orienting values. In some ways, I include myself in this population. I truly believe that we are in the early phase of a profound turning in human history, a waking up to our own potential. In realizing that our behavior affects the world and those in it, we are taking responsibility for our selves. Breaking social taboos and improving access to mental health services would do a great deal to speed up this necessary evolution in our collective consciousness, a step toward us adopting our role as stewards of this planet.

Armaments

Third, the British political leadership could use Brexit as a chance to set an example for the world in improving the efficacy of arms control regimes. We continue to produce weapons of war in enormous volumes every year. Arms are durable, and will be able to inflict grievous harm on people and property decades into the future. Again, I have been challenging my naive and idealistic attitude towards the armaments industry by trying to understand the positive role these technologies play in preserving peace, but I am convinced that the trends towards continued militarization on the global geopolitical stage are short-sighted and not sustainable. Again, I don’t have answers (though I am doing my bit …) — but perhaps Britain could stride ahead in this domain, as in so many others, in the lane opened by their exit from the EU. A senior MP acknowledged this opportunity in a recent Parliamentary Select Committee on Arms Export Control.

—

As we foray into this century we are realizing the scale of our challenges just as we are discovering — inventing — the tools to solve them. This is a chance for the UK government to reform how it operates. To innovate, and adopt a more agile rule of law, to break ground on building an antifragile system for administering public funds and making policy that can adapt with a world moving at the pace we are moving, accelerating as we are. I see promising and novel new techniques and technologies for managing incentives and distilling insight from informational assets in the digital era— computational modeling and machine learning, distributed ledgers and smart contracts, innovative governance models, connected sensors, crowd-sourced data mining. We are adapting. But we can adapt faster, and thereby mitigate much future human suffering.

I believe we can face down the global challenges we must address in these coming decades, for ourselves and our descendants. Many, many people are already adapting their behavior as they become aware that their actions impact the world around them, even devoting their lives to this cause — we have already come a long way. But reform from the top down offers a chance to make changes at a scale that can yield instant, significant impact. For this, I encourage the British political leadership to consider my perspective. I offer these ideas because it seems that culture might have a self-reflective quality to it, and if more positive perspectives are consumed, more might be projected.

My call to action? Parliament could go on a retreat — maybe to Scotland. Leave your phones, leave the city, and spend some time together. Reflect on your values. Engage in civil debate. Connect on a human level. Share meals. Remember you are all people doing your best. Rise above the din. Acknowledge that this an extraordinary moment in your history as a country, and rise to it. In this way, maybe, find a way forward, the path Britain and the world deserves.

My hope is that humanity will remember Britain at this moment for its brilliant blooming of human ingenuity, a new example of coordination and cooperation unique to our time.

John IV

MSc Candidate, University College London

London, March 2019

I’d go — I’d always go to practice, swim in the meets. I did the whole thing, I was physically present, participating … and I did alright, pretty well even. My races were the 50 and 100 freestyle, plus some relays and, my senior year, the 100 meter butterfly. I can’t put my finger on why I didn’t really care enough to push myself to be better. Maybe it was because I knew I would never be the best — what’s the difference between being the 1115th fastest in the state and 1116th? I can see the practical reasons to swim, but after a certain amount of progression, additional competence does not confer greater efficacy in an emergency (or at least diminishingly so). I didn’t push, and it was fine — I stayed in shape, had fun with my friends, had something to do after school, even got my first real job through team connections. (Casa Bonita, the summer after my senior year.)

My coach, Coach Scott, had a talk with my towards the beginning of my third season. He knew I showed up and worked but also knew that I could push harder, get faster quicker. At the time I was swimming 100 freestyle in maybe 56 seconds (yards), but he saw that in the fourth length of each race I swam my pace picked up noticeably. He correctly thought that I was deliberately going slower in the early stage of the race, saving my strength and endurance for later on.

The point, I guess — I’m not much of a story teller — was that he asked me what I was waiting for. In a longer race, sure, pacing is massively important. But not in the 100 free: it is a sprint. A long sprint, sure, but one where a reasonably well-conditioned high school boy should be able to swim at just about full power for the entire thing, less than 60 seconds of exertion. Unbroken mental focus and full physical effort is possible from the moment of departing to the moment of arrival. I’m still not sure why I didn’t figure out myself to sprint the entire race — to avoid the discomfort experienced when one pushes oneself to the edge, maybe, or a misjudgement of physical ability. Towards the end of high school I started to learn what was required, to be fully engaged through the duration of the race.

My sister calls the space space between departing and arriving the “liminal space”, an in-between time. It yields to many interpretations: aim- or placelessness, progress, transition (obviously), lost time, holy time (ritual). Sometimes it is a hassle to be endured, ignored — think a long flight — and sometimes it is among the most important moments of our life, a wedding or birth.

In the 12 or so years since my last swim meet my pattern of presenting myself without engaging has persisted, though not at a constant rate. The proportion of my time I spend aware of myself and my actions, considering the consequences of my decisions and behavior, has been increasing, as I think it has for almost everyone on Earth. (Happy to argue this point with anyone — [@robisoniv](https://twitter.com/robisoniv).) I realize now — we exist in the liminal space, always. The simple fact of existing means we abide in the space between when we didn’t exist and when we won’t. We departed at conception, give or take, and we arrive in death — beyond death, in many ways. Regardless, you and I and any one and any thing here is firmly planted in between those two moments in time: impermanent.

The reason I bring all this up is that I have realized that my habit of presence without engagement will not serve me if I want to do the things I want to do in this life. I am so privileged — not only do I have a functional body and my health, but I come from a loving, secure home, a life lived without exposure to true scarcity. I took this privilege for granted (which it was) as so many others in that situation do, and only recently have been realizing even the edge of the extent of my good fortune. I am humbled and formed by my lot in life, and the most humbling thing is that I am even here at all, that this whole production is even playing out. Absurd cannot capture even a figment of it — chlorophyll and transistors and Kanye West and Malala and Rick and Morty and the eurozone — and that’s just on Earth. What a laugh.

I don’t want to live my life like I swam the 100 free. I don’t want to switch on after the last turn, in the final quarter. I already spent the first half or so working out how this all makes sense, how it all can exist, the rules that govern reality and the patterns that arise and forms that manifest as time soldiers on, indifferent to the toing and froing of us perceiving agents in its midst.

I suppose I want to adopt the belief that the liminal space is holy. We’ve departed, we don’t know when we’ll arrive — so what do we do now?