The other week I visited Edinburgh for the latest instalment in Capgemini’s global roadshow series with Microsoft on Generative AI. I opened the proceedings setting the stage for an exciting day of discussion, debate and informative talks with Microsoft and our customers.

This post is a summary of my opening talk, where I unpacked five key trends related to Generative AI technology and its adoption within the enterprise that I foresee unfolding during 2024 and beyond.

1. Autonomous Agents & Digital Co-Workers

Generative AI will become a valuable work companion, boosting productivity and augmenting human capabilities, seamlessly managing tasks and schedules.

We are moving beyond simple API (Application Programming Interface) requests to generate content (akin to the chat interfaces we see today) to agent and multi-agent based approaches and architectures — where AI software and bots are able to seamlessly cooperate with each other to execute and complete a range of tasks against pre-defined or learned objectives and goals.

AI Town

A team from venture capital firm a16z, which invests in generative AI in games (amongst other start-ups and scale-ups), has recreated Smallville research paper as “AI Town” and released it as open source on GitHub.

Based on AI Town, you can build your own little AI civilisation, where OpenAI’s GPT-3.5 simulates AI agents in a small digital town based solely on prompts.

Autogen

Microsoft released Autogen as an open-source framework for building agent and multi-agent AI interactions. This technology is a stepping stone towards more autonomous AI. Where instead of giving it a specific request and it responds, we give it a goal or outcome and agents work together to complete various tasks.

Think of real world scenarios like travel agents or auctions and how they could be automated or augmented with this technology for a glimpse at how this could play out in the future.

OpenAI

OpenAI are exploring and will likely release updates to their platform and APIs in 2024 that positions them in the autonomous agent space. The assistants API from Open AI was a step in this direction and they are widely rumoured to be working on this functionality behind the scenes.

Langchain

The Langchain open-source Python and JavaScript framework today offers the ability to create agents with multiple tools at their disposal. This allows developers to construct agent like systems and platforms capable of performing an array of tasks based on request or intent. The array of third party services they can interact with are only increasing.

2. The Rise of Open Source Models

The open-source generative AI landscape is thriving, with constant innovations, growing communities, and exciting possibilities for the future. In 2024, open source pre-trained AI models will gain significant traction, empowering businesses to accelerate growth by combining these models with private or real-time data.

Open source’s low cost, flexibility and transparency make it attractive to organisations looking to fine-tune models for niche applications. Responsibly integrating open source generative AI into the enterprise requires rigorous evaluation of risks around security, ethics and technical capabilities.

Meta’s Llama 2 70B, Falcon 180B and Mistral AI’s Mixtral-8x7B became extremely popular in 2023, with comparable performance to proprietary models such as GPT 3.5, and Claude 2.

Mistral recently released their large model, their flagship model that is comparable with GPT-4.

Deploying Open Models in the Cloud

Given that 85% of enterprises will expand AI with open-source models, every major hyper-scale cloud provider has been racing to make available open-source models within private cloud enterprise environments.

The value that can be unlocked from specific open-source models, fine-tuned on datasets for enterprise or industry use cases is vast — and expect to see much more traction in this space in 2024.

Capgemini partners with Mistral AI

Capgemini and Mistral AI will focus on accelerating the evolution towards more accessible, versatile, and cost-effective generative AI implementation at scale.

Capgemini aims to help its extensive portfolio of global clients derive greater long-term value and accelerate their generative AI use cases by embedding Mistral AI’s highly efficient foundational models into their broader generative AI architecture.

HuggingFace

Some text generation models are either closed-source or their license limits commercial use. Fortunately, open-source alternatives are rapidly appearing and being embraced by the community as building blocks for further development, fine-tuning, or integration with other projects.

HuggingFace, a leader in the open AI space, has built partnerships with every cloud hyper-scaler to allow customers to bring in models from the HuggingFace platform into enterprise private cloud.

They also make open and available numerous leaderboards that allow model builders and users to compare models against each other for performance. Check out their open LLM leaderboard as one example.

3. Small Models

Large language models are impressive, but access and costs can be prohibitive. There will be a boom in smaller, use-case-specific generative models catering to niche tasks and industries — offering greater accessibility and customisation. These small models can be run on limited resources, making them more energy efficient, and they can often be fine-tuned on just a single GPU making them easier to develop and customise for specific tasks in certain scenarios.

For example, a healthcare-specific SLM might outperform a general-purpose LLM in understanding medical terminology and making accurate diagnoses.

Phi-2

Phi-2 is part of Microsoft’s Phi project, an attempt by its research team to make small, yet powerful language models. The model is just 2.7 billion parameter in size but outperforms models by up to 25 times its size.

TinyLlama

With its compact architecture and promising performance, TinyLlama can enable end-user applications on mobile devices or on edge devices, and serve as a lightweight platform for testing a wide range of innovative ideas related to language models.

Apple

Apple have been working on language models for years with recent research in animation. With their monopoly through the iPhone it is only a matter of time before Apple make more announcements in this space with small models capable of running natively on mobile devices.

4. Multimodal Interactions

Recent advancements in generative AI have focused on multi-modal capabilities, combining text, images, and other modalities such as video and audio to generate diverse content opening the door for richer and more personalised creative experiences and applications.

For example, during a customer service call, AI could analyse a client’s spoken request, interpret their financial documents and assess their facial expressions in a video consultation.

By synthesising these data points (speech, text and visual cues), AI can provide more personalised financial advice and enhance creditworthiness assessments with precision.

Audio

Spotify Research recently introduced LLark, a flexible multimodal language model for music understanding. Their initial research integrated a pre-retrained generative model for music with a pre-trained language model with the capabilities of music understanding, captioning and reasoning.

There have been other advances in this space with Meta open-sourcing Musicgen and Stability releasing Stable Audio.

Perplexity, Brilliant Labs and Rabbit R1

Perplexity teams up with Brilliant labs to integrate real-time answers into Frame, their AI-enhanced glasses. You can ask about what you see, instantly — all in your line of sight.

Rabbit Launched their R1 device at CES — amid a fanfare of both hype and intense scrutiny.

With many left wondering what this hardware offers beyond a traditional smartphone — The device looks a bit like a walkie-talkie and operates as a handheld pocket AI assistant capable of performing specific tasks.

Image and Video

This space is advancing at lightning speed — with only in the last week Open AI announcing their text-to-video model Sora — which creates amazing cinematic experiences up to 60 seconds long from a simple prompt. How long will it be before AI is augmenting Hollywood?

Runway on Twitter: "Introducing Multi Motion Brush.Control multiple areas of your video generations with independent motion.Available now for Gen-2 at https://t.co/ekldoIshdw pic.twitter.com/ZZoBWczNkg / Twitter"

Introducing Multi Motion Brush.Control multiple areas of your video generations with independent motion.Available now for Gen-2 at https://t.co/ekldoIshdw pic.twitter.com/ZZoBWczNkg

Runway just introduced the Multi Motion Brush for Gen-2 so you can apply different animations to several elements in static images.

Google have also recently release Lumiere a Space-Time diffusion model for video generation and other start-ups are entering this very hot space such as Pika.

5. Responsible Deployment

As AI becomes more integrated into our lives, the focus on AI safety and ethics at scale intensifies. In 2024, we will see this advance beyond regulations and principles into frameworks, tools and systems with standardised safety protocols and best practices that are embedded into the development and deployment lifecycle.

With the high profile nature of customer-facing Generative AI deployments, nobody wants to be on the front page of the news for the wrong reasons.

OnlyFake

OnlyFake, an underground website, used AI to generate fake IDs for $15 that were able to bypass crypto exchange OKX’s KYC verification process.

Pak N Save

Pak ’n’ Save’s Savey Meal-bot cheerfully created unappealing recipes when customers experimented with non-grocery household items

DPD Virtual Assistant

DPD had to disable its AI online chatbot after a customer was able to make the bot swear and write a poem criticising the parcel delivery company

The pace of technological innovation in AI does not seem to be slowing down any in 2024. We’re excited to see what comes next — and to help our customers figure out where and how to get the best value out of the technology.

If you’re interested in learning more about what we’re exploring at Capgemini and how we can help you create value from AI then please feel free to reach out to me directly to start a conversation.

Five Trends Shaping Generative AI in 2024 was originally published in Applied Innovation Exchange on Medium, where people are continuing the conversation by highlighting and responding to this story.

What is RainbowKit?

Built by the folks behind Rainbow Wallet, RainbowKit is a React library that makes it easy to add a “connect with my wallet” flow to your decentralised application (dapp). It provides out-of-the-box wallet management for your app, supporting numerous different wallets (e.g. Coinbase, Metamask etc…) and connections to different blockchain networks (e.g. Ethereum, Polygon etc…) and much more!

Prerequisites

In order to get started with this example we’ll assume you’ve already got the following available and installed

• Node.js 12.22.0 or later
• Metamask, with an available ETH account / wallet (for testing purposes). Here’s a quick explainer on how to setup a Metamask wallet, if you don’t have one already

Step 1: Install dependencies 👷‍♀️

For the purposes of this example we’ll use create-next-app to scaffold out a bare NextJS app with React. If you’ve already got an existing app you can skip over this part. Next we’ll install the necessary dependencies for RainbowKit using yarn (or npm) as follows

We install the main RainbowKit package alongside wagmi. Wagmi is a super-helpful collection of React hooks, containing a lot of what you need to get started working and building apps in the Ethereum ecosystem. A must have for any dapp developer, wagmi contains 20+ hooks for working with wallets, ENS, contracts, transactions, signing, and more! Worth a look regardless of if you’re using RainbowKit or not.

Step 2: Setup a chain provider and wrap the app ⛓️

Next we’ll configure and wrap our app using wagmi and the RainbowKit providers, inside pages/_app.js.

There’s a bit going on there so let’s break it down.

First up, we’re using wagmi’s configureChains helper to setup and configure our enabled blockchains and providers. This means we don’t need to worry about defining RPC URLs and chain configuration — this is all managed internally by wagmi.

For this example, we’ve set up with Alchemy as our default provider, but you’re free to use Infura, Moralis or any other provider of your choice.

The public provider ensures our chains always have an RPC URL to fallback on (in case Alchemy does not support the chain).

We’re also going to support connecting to the Ethereum mainnet and Polygon. You can see a list of chains RainbowKit supports over here, which can subsequently be configured with wagmi like this.

Next up, we create the wagmi client. This manages the wallet connection state and various configuration options using the connectors available from RainbowKit.

Lastly we wrap our application in the RainbowKit provider and WagmiConfig with our desired client and chain configuration from above. To add an extra bit of pop we enable cool mode which will give us some snazzy animations on mouse click select of the wallets in the UI.

Step 3: Embed the connect button 🔵

That’s the hard bit done. Lastly, we’ll embed the “connect my web3 wallet button”. This is as simple as importing the ConnectButton component and dropping it into the app in a place of your choosing. For example purposes we’ll drop it right into the homepage of our example app.

Let’s see it in action 💥

We can run yarn dev in our terminal to spin up the example. Below you can see the entire authentication flow UX / UI in action.

There we have it. Signed in to my custom app using Ethereum wallet via Metamask, which subsequently displays my wallet address and current balance. While logged in, the out of the box UI also gives me the option to switch across chains.

Going forward from here ⏩

For more advanced usage, such as having users sign a message with their wallet during connection to prove they own the connected account, or for accessing session token server side for authentication and privileged access purposes, RainbowKit has first-class support for NextJS Auth.

If you’re interested in getting this up and running on Moralis, I found this handy guide over on their blog over here. If you’re interested in exploring more you can find a full Codesandbox of this example here.

Adding a “connect my web3 wallet” to your app in 3 steps with RainbowKit was originally published in Applied Innovation Exchange on Medium, where people are continuing the conversation by highlighting and responding to this story.

As part of designing and building this experience, we’ve partnered with one of the leading start-ups in this space, AiFi — to embed their hardware, software and platform into the fabric of our store.

Our CornerShop autonomous shopping partner: AiFi

AiFi offer solutions that allow retailers to create autonomous, checkout-free experiences at any location. Powered by on-site cameras and edge compute with artificial intelligence (AI) algorithms that track shopper activity and purchases in store — and already deployed at many leading retailers across the globe including ALDI, Zabka and Carrefour.

At the CornerShop we’re building a MACH (Microservices, API-first, Cloud Native, Headless) store Operating System (OS) to power all of our customer and employee experiences.

To integrate AiFi into our platform and channels, we developed a suite of micro-services that enable various elements of the customer experience. This included registering customers with the AiFi platform on sign-up, generating mobile QR codes to allow customers and employees to check into the store to enable autonomous shopping sessions and crafting backend services to process customer orders and payments based on autonomous shopping events.

Smoothing the developer experience for building AiFi integrations

To improve the developer experience (DX), and to ensure we developed our micro services with consistency and reliability we first decided to build a re-usable NodeJS Software Development Kit (SDK) wrapper around the AiFi REST APIs. This then enabled us to scale the build of our various AiFi integrations with ease and speed. And we’re open-sourcing this SDK today for other retailers and brands that may want to build their own checkout-free experiences on top of the AiFi autonomous shopping platform.

You can find our library over on Github here. It includes support for Typescript and wraps most of AiFi’s APIs in easy to use helpers. For example to create a customer, it’s as easy as…

We hope other builders find this useful. Please give us feedback or raise issues directly on the Github project if you run into any problems or have any suggested improvements.

In the future we’re looking forward to adding the following enhancements

• Automatically generated types based on OpenAPI specs
• More robust and specific HTTP handled responses for edge cases and errors
• Typescript support for web hook events

Your experience and store transformation partner

We’re expanding our services and support to help our retail customers plan, pilot and scale the transformation of their stores, through start-up ecosystem partnerships and the innovations we are delivering in the CornerShop in London.

If you’re interested in learning more, or working with us to create your own checkout-free experiences get in touch at the bottom of this page.

Announcing an AiFi NodeJS SDK, the autonomous checkout accelerator was originally published in Applied Innovation Exchange on Medium, where people are continuing the conversation by highlighting and responding to this story.

You only have to walk into most high street retail stores today, to see that the majority of brands still struggle with connecting the dots (and data) between channels to provide truly connected in-store experiences.

In the future, the winners will be those that are able to fuse online retail with the physical store. Providing customers simultaneously with the best of both worlds — emotional connectivity within the store and online convenience.

Towards a more connected omni-channel purchase experience

Over at the CornerShop, our store-of-tomorrow in London, we’re experimenting with connected cross-channel shopping experiences. In one such experiment, we explore a re-imagined purchase experience — one which allows you, a consumer to walk into a store and for the store to recognise who you are at certain signature moments on your shopping journey. The experience should be connected across different touch points physically and digitally seamlessly, maintaining context and data and providing a more relevant and tailored in-store experience to you.

Our vision: Connecting the shopping cart

For our start point, we anchored on creating an experience surrounding a consumers shopping cart. Our vision was for our store to be able to understand what each consumer had been shopping for online through our native app and transition that context into the store to connect the physical and digital shopping journeys together — whenever the customer pays a visit.

In store we had some large format digital displays, near the physical product shelves — which would serve as the perfect point to try and stitch the channel experiences together.

Meet Mei: Our Shopper

Mei’s user journey begins browsing the CornerShop product catalogue on her phone through our native app, which she previous installed on her last visit to the store.

Intent on grabbing some morning goodies on her way to work, A product catches her eye, which she adds to her cart through the app.

On her walk into the office, she stops by the CornerShop for her morning coffee and to grab her products.

Mei walks into the store and up to the digital endless aisle displays near the product shelves, which automatically recognise she is nearby and welcome her personally to the store.

Mei interacts with the large display, where her shopping cart context is fluidly transitioned across channels — maintaining the earlier products she added to her cart in app, on the large display. Mei browses the extended product catalogue on the display and is able to top up her cart in situ, which automatically synchronises the update across to the native app.

Mei walks away from the screen, receiving a notification in-app that she has 2 items in her cart and prompting her to checkout. She completes the transaction using a card payment in app, and her goods are handed over to her by an in store employee before she leaves on her way.

Powered by next-generation composable commerce engine ommercetools

At the heart of this experience is commerce engine commercetools. Built on modern MACH principles (Microservice-based, API-First, Cloud-native, Headless) and allowing us to create flexible modern commerce experiences.

We’re using their world class commerce APIs to connect our products to channel experiences both in store (endless aisle) and in app seamlessly. This allows us to tailor the frontend experience to different needs across the devices while still maintaining a unified product catalogue in the backend.

With an event-driven, real-time, cloud-native architecture

For the cloud native platform we are running Serverless on Google Cloud and Firebase. The ability to easily deploy, orchestrate and operate micro-services was crucial, with Cloud Firestore and the Firebase software development kit (SDK) additionally offering us real-time web socket subscriptions for the various channel experiences (web/mobile).

For in-store proximity / location awareness we are using Estimote BLE Beacons, which triangulate the users proximity to in-store displays through an SDK embedded within our native mobile app.

The in-store displays are running offline-capable full-stack web apps built using React framework and remotely deployed and managed through Google Chrome OS device manager.

Bringing it all together

When Mei adds a product to her cart, this kicks off an event-subscription notification in commercetools which sends the updated cart payload to a REST micro-service in Google Cloud Functions. The latest version of her cart is then stored or updated against her profile in Cloud Firestore.

On the client side channels (web/mobile) the Firebase SDK web-socket subscription subscribes to the record in Cloud Firestore for the current users cart version. When this is updated (in the cloud) this causes the client libraries to trigger a fetch request to get the latest cart data from commercetools. This is processed asynchronously and the users cart is subsequently updated in app or web in near real-time.

This means that Mei can update her cart from any given channel (on the screen, or in the app) and the updates will make their way to all relevant client subscribers, thus keeping her cart data synchronised and up to date across the different channels.

A world of possibilities with connected data

In our experiment we’ve only explored a fraction of what is possible if you can connect your consumers shopping cart information across channels. From targeted push notifications, to better in-store advertising to surfacing consumer cart data to store associates to provide an elevated and improved customer experience. There are numerous innovative possibilities that could help your brand better connect your data to provide a superior experience for your customers that drives retention and ultimately cart conversion.

We’re looking forward to expanding on this experiment and further exploring how we can leverage our customers cart data for more effective audience segmentation and marketing execution, powered by cloud-native recommendation and machine-learning systems in our Cornershop store in Shoreditch, London.

If you’re interested in learning more or working with us to innovate your omni-channel experiences, get in touch at the bottom of this page.

Omni-Channel shopping carts with commercetools and Google Cloud Platform was originally published in Applied Innovation Exchange on Medium, where people are continuing the conversation by highlighting and responding to this story.

Within our London Applied Innovation Team we’ve been discussing the latest new tech, partnerships, interesting innovations and the frankly bizarre. Here are the best bits from the last couple of weeks.

Samsung open up a virtual flagship store in Decentraland

Hot off the back of their announcements at this years Consumer Electronics Show (CES) conference, Samsung opened up a virtual replica of it’s New York City store with a few twists. Dubbed 837X, and open for a limited time only — the experience allows consumers to explore different aspects of Samsung’s new strategy and vision. The virtual world created in the Decentraland gives users the chance to relax in an auditorium and enjoy keynote talks, go on a treasure hunt to collect Non-Fungible Token (NFT) related badges and explore the Samsung ‘Forest’ — a virtual and mythical forest of over two million trees — that highlights Samsung’s partnership with blockchain based global restoration player Veritree. In addition, Samsung announced it’s also introducing an NFT platform into it’s TVs — allowing users to browse, purchase and display digital collectibles.

Australian Open swings into the Metaverse

Much of the news headlines the past couple of weeks have centred around “No-vax” Djokovic and the controversy surrounding his vaccination status and entry to Australia to participate in the Australian Open tennis tournament.

Slipping under the radar somewhat, the Australian Open has been developing a couple of exciting innovations which are launching imminently related to Metaverse and NFTs.

First up, they are recreating Melbourne Park and the Rod Laver Arena in Decentraland — a fully virtual immersive Metaverse. Secondly, they are copping their own unique selection of 6776 “Art Ball” dynamic NFTs — digital, customisable, generative-artwork tennis balls. Interestingly, each Art Ball NFT will also be mapped to a 19x19cm plot of the tennis court surface. If a winning shot from any match lands on this plot the NFT will be updated in real-time to highlight the match information. Additionally, in an epic move which bridges NFTs with the physical world — if any championship point lands on a plot the corresponding NFT owner can claim and receive the championship point tennis ball from the match — delivered to them in a hand-crafted display case with a certificate of authenticity.

This is certainly one of the most interesting uses of NFTs we’ve seen to date — and we anticipate that we’ll see an explosion of similar ‘dynamic’ NFTs in 2022. NFTs with adaptable/upgradable traits that integrate data from the real world, and produce physical as well as digital moments for consumers.

Is the Metaverse off-limits for Apple?

Rumours are rife that Apple plans to release its own mixed reality headset this year, but the company has been quick to distance themselves from the word ‘Metaverse’ and some of the surrounding hype — suggesting that an Apple based fully virtual world where users can escape to is ‘off limits’.

Whether we will see an Apple iVerse or if Apple will attempt to make a foundational platform play for the Metaverse, just as it has done with the App Store for mobile, remains to be seen. If anyone can accelerate mainstream augmented/immersive technology adoption it could be them — just as they did previously with the iPhone.

The debate and controversy continues around web3

Much of the promise and hype around web3 paints a vision of a more transparent, decentralised and equitable web — breaking free of some of the constraints of web2 and the capitalist control of ‘big-tech’.

A recent technical breakdown analysis by a respected cryptographer, highlights some areas where the technology and platforms are not necessarily living up to this promise, particularly when it comes to NFTs. He suggests that companies stealing a march in the space (such as OpenSea) are not bad, but simply seizing a business opportunity that exists within the limits of blockchain today.

Meanwhile, the Financial Times suggest that the web3 bubble will pop. Still, the most valuable developments we will see in this space could be the physical and industrial innovations in the real world that emerge off the back of this wave. These innovations will ultimately be the most lasting and impactful — not what we end up creating in a fully virtual Metaverse.

Best-selling author and business professor Scott Galloway weighs in with a scathing breakdown of web3, shining a light on some of the inequalities of the current movement stating “Every member of Forbes’s 2021 cryptobillionaires list is a man. A third of them attended Stanford or Harvard. Out of the 12 listed, only one isn’t white. The web3 narrative feels like a Ted-X talk given to a survivalist group.”

Finally Web3isgoinggreat offers a slightly sarcastic (but true) timeline of insights into some of the scams, data breaches and controversial happenings in the web3 space.

Even cows are joining the ‘Mootaverse’

We’ve heard of Virtual Reality (VR) being used in healthcare for chronic pain management and as a platform to offer patients escapism for dealing with chronic conditions. In a slightly bizarre twist, one budding farmer from Turkey, Izzet Kocak has used VR headsets on his cows claiming they ‘reduce anxiety and improve overall emotional mood in the herd’, leading to an increased milk production.

We’re looking forward to joining the real-life / Metaverse mash-up of Farmville soon.

Other interesting tools and tech we’re playing with and getting excited about

• Pounx — Create and collaborate on 3D designs in real-time in the browser. Looks like Figma but aimed at 3D design
• Ceramic — A open-source web3 platform that aims to help people build web3 apps backed by stateful data storage — leveraging decentralised mutable (and shareable) data streams
• Pinata — A platform built on top of IPFS that makes storing and managing NFT media easier
• Hardhat — Local Ethereum development environment, aimed at speeding up smart contract programming in Solidity. Deploy contracts, run tests and debug easily — and it comes with an extensible plugin ecosystem to boot

What’s happening in the London Applied Innovation Exchange

Over the past couple of weeks our London team has been

• Exploring the start-up / tech ecosystem in relation to the Metaverse (see above for a snapshot on some interesting players we’re tinkering with)
• Implementing Augmented Reality (AR) virtual try-on within the CornerShop app with our partner WANNA — watch out for this launching soon
• Hatching plans for how we integrate NFTs and Metaverse worlds into our retail store and app at the CornerShop
• Creating and collaborating as a team in 3D worlds in Spatial.io

Stay tuned for more content, developments and launches from us in this space in the very near future. If you’re a brand or partner who is interested in working with us on a project related to the Metaverse, don’t hesitate to get in touch.

Applied Innovation x Metaverse round-up was originally published in Applied Innovation Exchange on Medium, where people are continuing the conversation by highlighting and responding to this story.

Legislation driving change

Just a few days ago, the UK government announced that the petrol and diesel ban will be brought forward 5 years to 2035 rather than 2040. To add to this, the UK government announced that to accelerate the shift to zero emission cars, all company cars will pay no company car tax in 2020–2021.

This is advancing the energy transition and when it comes to mobility, driving consumers towards more sustainable electric vehicles. Every one in 10 new cars sold in the UK in December were either electric or plug-in hybrid.

The need for ecosystems

A recent report we’ve released around the Sustainable Business Revolution underlines the need for why ecosystems are important to support a more sustainable planet in the future. This allows players to drive more sustainable business models and accelerate the speed to applied innovation in the market through partnerships.

During a recent innovation project with a client in the energy and utilities space we embarked on an ecosystem scan into the mobility and connected car space. Off the back of this, we worked with one of our designers at the Applied Innovation Exchange — selecting one of the vendor’s identified through the scan, and dedicating 1 day to prototype a new concept in the electric vehicle space.

Our ambition was to ideate around the following challenge statement: how might we promote sustainable driver and electric vehicle charging behaviour.

Taking a deeper look at ecosystem player Smartcar

Smartcar provides a single API for mobility applications. It currently supports over 15 car brands through a unified interface including Audi, Volkswagen and Tesla to name a few.

The great thing about Smartcar is that it’s a pure software solution. No need to fit your vehicle with any additional hardware like some applications in the wild today. On top of this, as a developer there’s no need to develop individual integration points for every different OEM, saving you time.

Smartcar allows a developer to onboard vehicle(s) — which can be done through any responsive web, mobile or native app via Smartcar connect. I liken this to OAuth (or sign-in with Google) but for cars.

After onboarding a vehicle, your application receives a token, which can then be used to call the Smartcar APIs to ingest live vehicle data such as battery levels, charge status, odometer readings and much more. Smartcar cloud talks directly to the OEM cloud to retrieve the live vehicle information at point of the API call — taking away the hassle of integrating with lots of different manufacturers for you as a developer.

Similar players in the market today include Otonomo, or High Mobility, which is aimed at a slightly different (and smaller) group of OEMs.

I was surprised at how easy the Smartcar APIs were to use, in a couple of hours I was able to knock-up a quick journey in a mobile app that onboards a vehicle and calls through Smartcar to the OEM cloud to retrieve live vehicle data through the NodeJs SDK. The API documentation is great too!

The Concept: Shockz — Super-charged EV rewards and experiences

The concept we came up was a loyalty app that rewards consumers for sustainable electric vehicle and driver behaviour. Earn kudos against your friends or others in your community and unlock unique badges and experiences that promote green behaviour.

How it could work

A native app that allows you to connect your electric vehicle. We then use the data from your car to engage and personalise the experience for you. How rewards could work:

• Environmentally friendly driving: By comparing the distance you’ve travelled between two charging periods we calculate a driver score for you and place you in a leader board amongst your community / friends. See how you fare over time and improve your driving to be more sustainable — getting more out of your vehicle.
• Charging off-peak: We reward you for charging your EV during hours of low electricity consumption.
• Charging at specific destinations: Collect badges (a la Foursquare) for charging at work or other specific locations (e.g. airport car parks) that allow energy to be stored and recycled for other business uses

This is similar to how consumers interact with the Nest Leaf system and how Nest uses this to personalise engagement with you to promote energy-saving behaviour in your home.

Looking into the mobility ecosystem again there are a few other players developing similar rewards-based solutions such as Juicepoints and Fleetcarma.

How this could scale

Over-time and by continually collecting data around vehicle and driver behaviour machine learning algorithms can be developed to forecast and predict future energy consumption as well as used to help optimise or pro-actively nudge consumers to change their behaviour. Additionally, AI could provide automated or smart charging schedules based on individual’s behaviour or help provide energy demand-response programs.

Looking at fleet usage — a solution like this could be scaled to handle multiple vehicles and machine learning algorithms could optimise the time that vehicles are charged and for what duration in a fleet based on battery usage across multiple vehicles spread geographically.

Checkout the onboarding journey through Smartcar to retrieve the Tesla battery status and vehicle information below.

Wrapping up

As the sustainable business revolution takes over, it’s clear there’s still a huge amount of disruption and innovation in this space to come, impacting our cities (to make them even smarter), our energy ecosystems and changing the way people move.

Companies need to shift from platforms to spin a web of ecosystem partnerships helping them to accelerate the application of sustainable innovation. We need to translate the words we see in the world into action and start to create sustainable business solutions for the future that address our environmental impact.

This was a short experiment into the world of smart mobility that we had fun with. If you’ve got a bias for action and are looking to build new products, services and systems for the sustainable mobility ecosystem that drive meaningful change in the world — get in touch — there’s lots to do and we can make a bigger impact if we join forces!

Experiments in sustainable mobility was originally published in Applied Innovation Exchange on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Applied Innovation Exchange team in London recently looked at implementing an MVP (Minimum Viable Product) against a challenge for a U.K high street retailer.

The challenge

Improve the in store experience for consumers buying shoes or trainers

If you have ever bought a pair of shoes or trainers from any high street retailer you will be familiar with the process:

• Walk in to a store
• Find a trainer/shoe on display that you like
• Try to locate an available friendly assistant to tell them the size you need
• Assistant goes off to find your size and if they have it, brings a pair back for you to try on
• The shoes fit and you like them, so you purchase at the till

Analysing this process, we identified some issues and areas for improvement.

Customers may abandon their journey if:

1. They cannot locate an assistant quick enough
2. The store is too busy

In addition to these factors, if the assistant goes to the stock room and comes back and the customers shoe size is out of stock, they are likely to be unhappy and leave empty handed, meaning the brand will have missed an opportunity to sell. The fact the customer also has to wait to realise they don’t have the correct size adds to this tension.

The retailer hasn’t actually connected with the customer in any way during the above process. They don’t ‘know’ them better. Sure, if the customer already has a retail loyalty card — then they might have made that connection at the Point Of Sale (POS) — otherwise the customer is still anonymous to the brand.

We have two potential bottlenecks in the store:

1. Finding trainers
2. At the POS

At busy times the retailer would have to scale the number of assistants (and potentially the number of POS) to meet the customer demand in store.

For our MVP we decided to focus on the first bottleneck — finding the right trainers/shoes.

We had some ideas around how we could improve wait times at the POS as well but decided to park those for the meantime.

Our process

For the project we knew we only had around 4 weeks to turn around an MVP. First and foremost we wanted to address the main customers need:

A faster and optimised journey with less hassle of finding and matching trainers/shoes in their size.

We also wanted to showcase how we could connect the journey better for in in-store assistant so the retailer starts to learn more about their customers wants and needs — which could influence future marketing or communications to them.

We started off constructing a storyboard of our new potential customer journey and comparing it to the as-is.

Our ‘to be’ journey looked something like this:

• Walk in to store
• Find a trainer/shoe on display you like
• Walk up to smart digital display, hold the trainer up to camera. Trainer is recognised, then you select the size(s) you need (display only shows sizes which are in stock). Optionally, the customer can enter their phone number and receive a notification when their shoes are ready to be collected
• A message is sent to back of store to an assistant with a tablet to pick the shoe and bring it to the front
• The assistant can tap a ‘Ready to collect’ button on their tablet and notification will be sent to the customer (if they entered their phone number). Otherwise a number will appear on a digital display letting the customer know their trainers are ready to collect and try on
• As the customer is trying on the trainers, the in-store assistant has information at their fingertips (on the tablet) for other recommended products, similar shoes and trainers, loyalty offers and the ability to capture the customer information there and then should they need to.

We knew we were introducing a new potential bottleneck around the smart digital displays, but in future we envisaged the application being available through the Apple or Google Play store. This would allow a customer to scan a trainer from their own mobile phone camera.

This also opens up the possibility for visual search for consumers who are out on the street or at home and see trainers they like.

For the purposes of the MVP we wanted to control the environment in which the pictures were taken.

Our solution

The solution was broken down into 3 parts:

1. The consumer application which leverages the camera on the smart digital display

2. The backend image recognition and visual search capability (powered by AI)

3. The employee tablet application to track and fulfill incoming orders for shoes and gather customer data

For the consumer application we developed this using a combination of React, Firebase and Electron. This allowed us to create a hybrid web and desktop application so that we could easily test and deploy across multiple platforms. It was also super simple to package the application as an installer (through Electron) to be installed on the in store screens. We stored incoming orders for shoes in Firebase database and used Firebase storage to host the trainer and shoe images.

The backend image recognition was powered by IBM Watson Visual Recognition and surfaced through a NodeJS REST API for the frontend app.

We trained the image classifier on a cut down dataset of various shoes and trainers. We also used IBM Watson’s visual search capabilities to return recommended similar shoes and trainers from our dataset when the customer scanned a trainer. The speed we were able to train and deploy our custom image recognition model with Watson was really key. In the space of a couple of days we had a working model and out of the box REST API ready to integrate to our frontend application. If we had to engineer this ourselves from scratch — i’m not sure we would achieved it in the timeframe.

On the backend we integrated Twilio to handle sending SMS messages to the customer when they placed an order, and again when their shoes had arrived at the counter for collection.

Finally the employee tablet app was a pure web application built in React JS. We leveraged the same Firebase database to pickup the incoming orders in real time. As we were utilising Firebase’s realtime database, no need for the employee to refresh the screen for the web application — it updates instantly whenever a new order comes in.

Lastly — when the employee creates or updates the user profile information we integrate with Salesforce to send and store the data. This allows the retailer to use the CRM capabilities for future marketing and communication.

We had a lot of fun creating this application in a really short amount of time. It’s interesting to think of the future possibilities as we look at a connected retail experience and ‘store of the future’ concepts. I expect Artificial Intelligence (AI), Augmented Reality (AR), Virtual Reality (VR) and the intersection of customer data in all of the above to play a pivotal part in how retail companies look to exploit emerging technology to provide a personal and experiential customer journey that delights and allows us (as consumers) to optimise our time when in store.

Retail companies that are able to grasp this concept and experiment with these emerging technologies in a fast way, in my view, will start to win the hearts and minds of consumers.

Improving consumer buying experience with in store shoe recognition was originally published in Applied Innovation Exchange on Medium, where people are continuing the conversation by highlighting and responding to this story.