Stories by Gerard Sans on Medium

Building Advanced AI Voice Assistants Using Google Gemini 3.1 Live and Angular

Gerard Sans — Sun, 24 May 2026 07:34:32 GMT

Explore the Gemini Live API — a real-time, multimodal API for creating advanced, voice-first user experiences using the latest Gemini 3.1 Live models and the official Google GenAI SDK.

This article presents a quick-start project demonstrating Gemini 3.1 Live’s real-time, multimodal AI capabilities within an Angular web application.

What’s Gemini Live?

Gemini Live API is part of the Google Gemini model family, enabling a new generation of dynamic, multimodal, real-time AI experiences, as showcased in Pixel phones and the Gemini mobile app.

Gemini Live powers innovative applications across devices and platforms:

Hands-free AI Assistance: Users interact naturally through voice while cooking, driving, or multitasking.
Real-time Visual Understanding: Get instant AI responses as you show objects, documents, or scenes through your camera.
Smart Home Automation: Control your environment with natural voice commands — from adjusting lights to managing thermostats.
Seamless Shopping: Browse products, compare options, and complete purchases through conversation.
Live Problem Solving: Share your screen to get real-time guidance, troubleshooting, or explanations.
Integration with Google services: Leverage existing Google services like Search or Maps to enhance its capabilities.

Try Gemini Live in the Gemini app (for iOS and Android)

Gemini is now available on iPhone and Android, so you can enjoy free, natural conversations with Gemini Live and get the most out of Google’s personal AI assistant.

Try Gemini Live in Google AI Studio

Before diving into the code, explore Gemini Live’s capabilities — voice interactions, webcam, and screen sharing — by using Google AI Studio Live. This interactive playground will help you understand the available features and integration options before adding them to your projects.

First step: Obtaining your API key from Google AI Studio

Navigate to aistudio.google.com and generate an API key. You can check the global availability of the API here.

Cloning the Angular Application

Run the following command in a folder of your choice to create a local copy of the project:

$ git clone https://github.com/gsans/gemini-3-live-angular.git
> Cloning into `Spoon-Knife`...
> remote: Counting objects: 10, done.
> remote: Compressing objects: 100% (8/8), done.
> remove: Total 10 (delta 1), reused 10 (delta 1)
> Unpacking objects: 100% (10/10), done.
$ cd gemini-3-live-angular
$ npm install

These commands will download the complete repository and install its dependencies.

System Requirements

Node.js and npm (latest stable version)
Angular CLI (globally installed via npm install -g @angular/cli)

Setting up the project

Add a new environment by running the following command:

ng g environments

This creates the necessary files for development and production environments:

src/environments/environment.development.ts
src/environments/environment.ts

Modify the development file, replacing with your actual API Key:

// src/environments/environment.development.ts
export const environment = {
  API_KEY: "",
};

To execute the code, run the following command in the terminal and navigate to localhost:4200 in your browser:

ng serve

Usage Guide: Getting Started

Launch the application and click the Connect button under Connection Status.
This project requires a WebSocket connection, which will be used to send and receive real-time voice, images, and video from the Gemini Live API.
Monitor the browser’s Developer Tools Console for connection issues.
You can also use the Control Tray, located in the top right corner, to: toggle the microphone, share your screen or webcam, and use the play/pause button to start a conversation.

A few examples using Gemini’s Tools

You can try any of the prompts you already use with Gemini. The following examples explore Gemini’s Tools that access real-time data via Google Search, execute Python code, or execute actions through external APIs.

Gemini real-time access via Google Search

Gemini can use Google Search to provide real-time information beyond the training data cut-off date and grounding to minimize hallucinations.

Tell me the scores for the last 3 games of FC Barcelona.

Gemini using a Python sandbox via Code Execution

Gemini can access a Python sandbox to generate and execute code as part of its response. This sandbox has access to libraries such as altair, chess, cv2, matplotlib, mpmath, numpy, pandas, pdfminer, reportlab, seaborn, sklearn, statsmodels, striprtf, sympy, and tabulate. matplotlib can generate 2D/3D diagrams as images.

You can try this option in Google AI Studio. Enable _Code execution_ in the _Tools_ section of the _Run settings_ panel.

What’s the 50th prime number?

What’s the square root of 342.12?

Gemini running an external weather API (mock) via Function Calling

The function calling feature in Gemini allows you to provide users with access to tools: external APIs that provide real-time data or perform actions (e.g., making restaurant reservations or ordering food). These tools remain available throughout the conversation.

This example demonstrates this capability using a mock weather API. The model doesn’t directly call the function; instead, it generates structured output specifying the function name and suggested arguments. You’ll need to add code to handle the user request (as identified by Gemini) by calling the external API. Finally, provide the API’s output to Gemini, which formats the final answer for the user.

What’s the weather in London?

Configuration Options

The main configuration is handled in src/gemini/gemini-client.service.ts within the MultimodalLiveService class. You can customize the LiveConnectConfig settings, including response modalities (e.g. text vs. audio), proactivity features, and tools.

The application utilizes a dynamic model selection mechanism in getCustomConfig() based on the features enabled in the user interface Control Tray:

// src/gemini/gemini-client.service.ts
if (user.affectiveAudio || user.proactiveAudio) {
  // Both Affective Dialog and Proactive Audio features are unsupported in gemini-3.1-flash-live-preview.
  model = "gemini-2.5-flash-native-audio-latest";
} else {
  model = "gemini-3.1-flash-live-preview";
}

This ensures that advanced voice features like emotional tone sensing (Affective Dialog) and smart turn-taking (Proactive Audio) connect seamlessly using the specialized native-audio model, while normal sessions default to the fast, low-latency gemini-3.1-flash-live-preview model.

Usage Limits. Daily and session-based limits apply. Token count restrictions are in place to prevent abuse. If limits are exceeded, wait until the next day to resume.

With the upgrade to Gemini 3.1 and the GenAI SDK, the application now natively supports bidirectional audio transcription directly via the inputAudioTranscription and outputAudioTranscription session configuration parameters. The service buffers and logs the live transcripts, displaying them in the side panel without requiring any third-party speech-to-text APIs.

Technical Overview: Data Flow, Events and UI

Compared to a standard Gemini’s client, this project introduces significant complexity due to its real-time streaming capabilities for audio (microphone and Gemini’s voice) and video (screen and webcam). This requires two clients: one for Gemini and another for the WebSocket. The following diagram illustrates the methods, observables, and events involved in the data flow.

Managing the real-time connection (over WebSocket)

Contrary to a REST API that follows a request-response protocol, WebSockets follow a bidirectional flow, meaning that both sides of the WebSocket channel can initiate or interrupt the communication. This creates three stages: setup handshake (open connection), bidirectional message exchange (connection remains open), and termination (one side closes the connection).

Processing Audio (Web Audio API)

This section of the project manages both user microphone input (outgoing audio) and Gemini’s voice response (incoming audio) using the AudioRecorder and AudioStreamer classes, respectively.

Audio processing within the Web Audio API occurs in a separate thread from the main UI thread, ensuring smooth performance, even under heavy load.

The AudioRecorder captures audio from the user’s microphone and sends it to the Gemini’s client via the sendRealtimeInput() method. This is accomplished using two audio worklets: worklet.audio-processing.ts and worklet.audio-meter.ts. The first worklet, worklet.audio-processing.ts, handles the core audio capture from navigator.mediaDevices.getUserMedia({ audio: true }), performing buffering and converting the audio data to a suitable format for transmission. The second worklet, worklet.audio-meter.ts, provides volume level data, used by the audio-pulse control to visually represent the audio levels from both the AudioRecorder (microphone) and the AudioStreamer (Gemini’s voice).
The AudioStreamer receives Gemini’s voice data through the ws.audio event handler (indicating audio data received from the WebSocket). It manages buffering and scheduling of audio chunks to ensure smooth, gap-free playback, even with network fluctuations. The Gemini’s client calls the AudioStreamer's addPCM16() method to pass the received audio data for playback.

Processing Video (WebRTC)

This part of the project uses WebRTC to stream the screen or webcam to a video element. Using a separate process, we render a frame via a canvas element and send it over to the Gemini Live API using sendRealtimeInput().

Current FPS is set to 0.5, or a frame every 2 seconds.

User Interface: connection, chat window, and control tray

The user interface is split into three main blocks: the connection, the chat window, and the control tray. The chat window is followed by the video and canvas elements used by the control tray component. Below is a diagram showing the main methods behind each button, component inputs, and events.

Gemini Live API Setup and Tools

The Gemini Live API configuration is dynamically assembled in getCustomConfig() within src/gemini/gemini-client.service.ts based on active feature toggles (Affective Dialog, Proactive Audio) from the UI Control Tray component, falling back to the standard configurations on MultimodalLiveService. I recommend starting with a simpler project where you can learn each Gemini Tool separately. You can get started with this article below.

Conclusion

Congratulations! You have successfully accessed the Gemini Live API. Use the completed GitHub project as a reference to create your own features.

This tutorial covered the following:

Introduction to the Gemini Live API.
Project setup and getting started.
A few examples to showcase Gemini’s Tools (Code Execution, Google Search, and Function Calling).

This project shows how to create a Gemini Live Client using WebSockets to stream images, documents, audio (microphone and Gemini Live voices), and video (screen sharing and webcam) from an Angular project.

Building your own Voice-First Assistant Applications

Use this project to start familiarizing yourself with real-time voice user interactions and build your own. These are some ideas to get you started:

An AI booking assistant with access to your schedule, capable of handling dentist bookings and cancellations (via voice or chatbot interface).
An automated robot cafe that takes orders and relays them to a robot barista (using voice, text, or visual interactive controls).

Thanks for reading!

Do you have any questions? Feel free to leave your comments below or contact me on Twitter at @gerardsans.

Resources

Building Advanced AI Voice Assistants Using Google Gemini 3.1 Live and Angular was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Building Advanced AI Voice Assistants Using Google Gemini 2.0 and Angular

Gerard Sans — Sun, 09 Mar 2025 01:37:54 GMT

Explore the Gemini Live API (Experimental) — a real-time, multimodal API for creating advanced, voice-first user experiences.

This article presents a quick-start project demonstrating Gemini 2.0’s real-time, multimodal AI capabilities within an Angular web application.

This project originated as an Angular adaptation of the Multimodal Live API — Web console, which is presently only available in React. The Gemini Live API is in active development and may change; there is currently no official JavaScript client or SDK.

What’s Gemini Live?

Gemini Live API is part of Google Gemini 2.0, enabling a new generation of dynamic, multimodal, real-time AI experiences, as showcased in Pixel 9 phones and the Gemini app for Mobile (iOS and Android).

Gemini Live powers innovative applications across devices and platforms:

Hands-free AI Assistance: Users interact naturally through voice while cooking, driving, or multitasking.
Real-time Visual Understanding: Get instant AI responses as you show objects, documents, or scenes through your camera.
Smart Home Automation: Control your environment with natural voice commands — from adjusting lights to managing thermostats.
Seamless Shopping: Browse products, compare options, and complete purchases through conversation.
Live Problem Solving: Share your screen to get real-time guidance, troubleshooting, or explanations.
Integration with Google services: Leverage existing Google services like Search or Maps to enhance its capabilities.

Try Gemini Live in the Gemini app (for iOS and Android)

Gemini is now available on iPhone and Android, so you can enjoy free, natural conversations with Gemini Live and get the most out of Google’s personal AI assistant.

Try Gemini Live in Google AI Studio

First step: Obtaining your API key from Google AI Studio

Navigate to aistudio.google.com and generate an API key. You can check the global availability of the API here.

Cloning the Angular Application

Run the following command in a folder of your choice to create a local copy of the project:

$ git clone https://github.com/gsans/gemini-2-live-angular.git

> Cloning into `Spoon-Knife`...
> remote: Counting objects: 10, done.
> remote: Compressing objects: 100% (8/8), done.
> remove: Total 10 (delta 1), reused 10 (delta 1)
> Unpacking objects: 100% (10/10), done.

$ cd gemini-2-live-angular
$ npm install

These commands will download the complete repository and install its dependencies.

System Requirements

Node.js and npm (latest stable version)
Angular CLI (globally installed via npm install -g @angular/cli)

Setting up the project

Add a new environment by running the following command:

ng g environments

This creates the necessary files for development and production environments:

src/environments/environment.development.ts
src/environments/environment.ts

Modify the development file, replacing with your actual API Key:

// src/environments/environment.development.ts
export const environment = {
  API_KEY: "",
  WS_URL: "wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContent",
};

To execute the code, run the following command in the terminal and navigate to localhost:4200 in your browser:

ng serve

Usage Guide: Getting Started

Launch the application and click the Connect button under Connection Status.
This project requires a WebSocket connection, which will be used to send and receive real-time voice, images, and video from the Gemini Live API.
Monitor the browser’s Developer Tools Console for connection issues.
You can also use the Control Tray, located in the top right corner, to: toggle the microphone, share your screen or webcam, and use the play/pause button to start a conversation.

A few examples using Gemini’s Tools

Gemini real-time access via Google Search

Gemini can use Google Search to provide real-time information beyond the training data cut-off date and grounding to minimize hallucinations.

Tell me the scores for the last 3 games of FC Barcelona.

Gemini using a Python sandbox via Code Execution

You can try this option in Google AI Studio. Enable Code execution in the Tools section of the Run settings panel.

What’s the 50th prime number?

What’s the square root of 342.12?

Gemini running an external weather API (mock) via Function Calling

What’s the weather in London?

For a more complex example of Function Calling implementing CRUD, see GenList.

Configuration Options

The main configuration is handled in src/app.component. You can toggle between audio and text modalities:

let config: LiveConfig = {
  model: "models/gemini-2.0-flash-exp",
  generationConfig: {
    // For text responses in chat window
    responseModalities: "text",
    
    // For audio responses (uncomment to enable)
    // responseModalities: "audio",
    // speechConfig: {
    //   voiceConfig: { prebuiltVoiceConfig: { voiceName: "Aoede" } },
    // },
  },
}

Usage Limits. Daily and session-based limits apply. Token count restrictions are in place to prevent abuse. If limits are exceeded, wait until the next day to resume.

At the moment, the audio option is not sending over the transcript or corresponding text to your voice commands or Gemini’s voice responses. You can use an external speech-to-text API to achieve this, like Deepgram. This exercise is left to the reader to avoid adding further complexity to the current project. Let me know if you don’t know how to approach this task.

Technical Overview: Data Flow, Events and UI

Managing the real-time connection (over WebSocket)

Comparison of HTTP/REST and WebSocket communication protocols. WebSockets maintain an open connection for bidirectional communication.

Processing Audio (Web Audio API)

This section of the project manages both user microphone input (outgoing audio) and Gemini’s voice response (incoming audio) using the AudioRecorder and AudioStreamer classes, respectively.

Audio processing within the Web Audio API occurs in a separate thread from the main UI thread, ensuring smooth performance, even under heavy load.

The AudioRecorder captures audio from the user’s microphone and sends it to the Gemini’s client via the sendRealtimeInput() method. This is accomplished using two audio worklets: worklet.audio-processing.ts and worklet.audio-meter.ts. The first worklet, worklet.audio-processing.ts, handles the core audio capture from navigator.mediaDevices.getUserMedia({ audio: true }), performing buffering and converting the audio data to a suitable format for transmission. The second worklet, worklet.audio-meter.ts, provides volume level data, used by the audio-pulse control to visually represent the audio levels from both the AudioRecorder (microphone) and the AudioStreamer (Gemini’s voice).
The AudioStreamer receives Gemini’s voice data through the ws.audio event handler (indicating audio data received from the WebSocket). It manages buffering and scheduling of audio chunks to ensure smooth, gap-free playback, even with network fluctuations. The Gemini’s client calls the AudioStreamer's addPCM16() method to pass the received audio data for playback.

Processing Video (WebRTC)

Current FPS is set to 0.5, or a frame every 2 seconds.

User Interface: connection, chat window, and control tray

Overview of the user interface components and their interactions. The app-control-tray component manages user input for microphone, webcam, and screen sharing.

Gemini Live API Setup and Tools

The Gemini Live API configuration can be found in the connect method in src/app/app.component.ts and, if left empty, in the config property in the MultimodalLiveService. I recommend starting with a simpler project where you can learn each Gemini Tool separately. You can get started with this article below.

Building Advanced Chatbots with Angular and Google Gemini 2.0: Real-Time Data, Tools, and APIs

Conclusion

Congratulations! You have successfully accessed the Gemini Live API. Use the completed GitHub project as a reference to create your own features.

This tutorial covered the following:

Introduction to the Gemini Live API.
Project setup and getting started.
A few examples to showcase Gemini’s Tools (Code Execution, Google Search, and Function Calling).

Building your own Voice-First Assistant Applications

Use this project to start familiarizing yourself with real-time voice user interactions and build your own. These are some ideas to get you started:

An AI booking assistant with access to your schedule, capable of handling dentist bookings and cancellations (via voice or chatbot interface).
An automated robot cafe that takes orders and relays them to a robot barista (using voice, text, or visual interactive controls).

There are more examples available created by the Gemini team using React.

Thanks for reading!

Do you have any questions? Feel free to leave your comments below or contact me on Twitter at @gerardsans.

Resources

Final Warning

The official version of the Gemini Live API in the JavaScript SDK, once released, will likely have a different implementation and API design, with improvements in security and performance (possibly using WebAssembly). This project is at the forefront of innovation and may already be outdated by the time you read this. However, it is still a great way to get ahead and learn about the future of voice AI assistants today.

Build AI-Powered Angular Apps with Google Gemini

Gerard Sans — Mon, 26 Feb 2024 23:36:43 GMT

A step-by-step guide to integrating Google Gemini into your Angular applications.

In this post, you are going to learn how to access Gemini APIs to create the next generation of AI-enabled Applications using Angular.

We’ll build a simple application to test Gemini Pro and Gemini Pro Visual via the official client. As a bonus, we will also show how you can use Vertex AI via its REST API.

Here’s what we’ll cover:

Introduction to Google Gemini
Getting your API key from Google AI Studio
Creating the Angular Application
Generate text from text-only input (text)
Generate text from text-and-images input (multimodal)
Build multi-turn conversations (chat)
Generate content using streaming (stream)
Bonus: generate AI content with Vertex AI via the REST API
Running the code and verifying the response
Conclusions

Introduction to Google Gemini

Google Gemini is a family of large language models (LLMs) offering state-of-the-art AI capabilities created by Google AI. Gemini models include:

Gemini Ultra. The largest and most powerful model, excels in complex tasks like coding, logical reasoning, and creative collaboration. Available through Gemini Advanced (formerly Bard).
Gemini Pro. A mid-size model optimised for diverse tasks, offers performance comparable to Ultra. Available through Gemini Chatbot and in Google Workspace and Google Cloud. Gemini Pro 1.5 has improved performance, including a breakthrough in long-context understanding up to a million tokens including text, code, images, audio and video.
Gemini Nano. A lightweight model designed for on-device use, brings AI capabilities to mobile phones and small devices. Available on Pixel 8 and Samsung S24 series.
Gemma. Open-source models inspired by Gemini, offers state-of-the-art performance at smaller sizes and designed with responsible AI principles in mind.

Getting your API key from Google AI Studio

Go to aistudio.google.com and create an API key. If you are not based in the US, you can use Vertex AI which is available globally or use a VPN service.

Use the API Key together with Google AI JavaScript SDK or the REST API.

Creating the Angular Application

Use the Angular CLI to generate a new application:

ng new google-ai-gemini-angular

This scaffolds a new project with the latest Angular version.

Setting up the project

Run this command to add a new environment:

ng g environments

This will create the following files for development and production:

src/environments/environment.development.ts
src/environments/environment.ts

Edit the development file to include your API Key:

// src/environments/environment.development.ts
export const environment = {
  API_KEY: "",
};

Google AI JavaScript SDK

This is the official client to access Gemini models. We will use it to:

Generate text from text-only input (text)
Generate text from text-and-images input (multimodal)
Build multi-turn conversations (chat)
Generate content as is created using streaming (stream)

Add this package to your project:

npm install @google/generative-ai

Initialise your model

Before calling Gemini we need to go through the model initialisation. This includes the following steps:

Initialising GoogleGenerativeAI client with your API key.
Choosing a Gemini model: gemini-pro or gemini-pro-vision.
Setting up model parameters including safetySettings, temperature, top_p, top_k and maxOutputTokens.

import {
  GoogleGenerativeAI, HarmBlockThreshold, HarmCategory 
} from '@google/generative-ai';

...
const genAI = new GoogleGenerativeAI(environment.API_KEY);
const generationConfig = {
  safetySettings: [
    {
      category: HarmCategory.HARM_CATEGORY_HARASSMENT,
      threshold: HarmBlockThreshold.BLOCK_LOW_AND_ABOVE,
    },
  ],
  temperature: 0.9,
  top_p: 1,
  top_k: 32,
  maxOutputTokens: 100, // limit output
};
const model = genAI.getGenerativeModel({
  model: 'gemini-pro', // or 'gemini-pro-vision'
  ...generationConfig,
});
...

For Safety settings you can use the defaults (block medium or high) or adjust them to your needs. In the example, we increased the threshold for harassment to block outputs with low probability or above to be unsafe. You can find a more detailed explanation here.

These are all the models available and their default settings. Theres a limit of 60 request per minute. You can learn more about model parameters here.

Generate text from text-only input (text)

Below you can see a code snippet demonstrating Gemini Pro with a text-only input.

async TestGeminiPro() {
  // Model initialisation missing for brevity

  const prompt = 'What is the largest number with a name?';
  const result = await model.generateContent(prompt);
  const response = await result.response;
  console.log(response.text());
}

Generate text from text-and-images input (multimodal)

This example demonstrates how to use Gemini Pro Vision with text and images as input. We are using an image in src/assets for convenience.

async TestGeminiProVisionImages() {
  try {
    let imageBase64 = await this.fileConversionService.convertToBase64(
      'assets/baked_goods_2.jpeg'
    );
  
    // Check for successful conversion to Base64
    if (typeof imageBase64 !== 'string') {
      console.error('Image conversion to Base64 failed.');
      return;
    }
    // Model initialisation missing for brevity
    let prompt = [
      {
        inlineData: {
          mimeType: 'image/jpeg',
          data: imageBase64,
        },
      },
      {
        text: 'Provide a recipe.',
      },
    ];
    const result = await model.generateContent(prompt);
    const response = await result.response;
    console.log(response.text());
  } catch (error) {
    console.error('Error converting file to Base64', error);
  }
}

In order to convert the input image into Base64 you can use the FileConversionService below or an external library.

// file-conversion.service.ts
import { Injectable } from '@angular/core';
import { HttpClient } from '@angular/common/http';
import { firstValueFrom } from 'rxjs';

@Injectable({
  providedIn: 'root',
})
export class FileConversionService { 
  constructor(private http: HttpClient) {}
  async convertToBase64(filePath: string): Promise {
    const blob = await firstValueFrom(this.http.get(filePath, { responseType: 'blob' }));
    return new Promise((resolve, reject) => {
      const reader = new FileReader();
      reader.onloadend = () => {
        const base64data = reader.result as string; 
        resolve(base64data.substring(base64data.indexOf(',') + 1)); // Extract only the Base64 data
      };
      reader.onerror = error => {
        reject(error);
      };
      reader.readAsDataURL(blob);
    });
  }
}

Image requirements for Gemini:

Supported MIME types: image/png, image/jpeg, image/webp, image/heic and image/heif.
Maximum of 16 images.
Maximum of 4MB including images and text.
Large images are scaled down to fit 3072 x 3072 pixels while preserving their original aspect ratio.

Build multi-turn conversations (chat)

This example shows how to use Gemini Pro to build a multi-turn conversation.

async TestGeminiProChat() {
  // Model initialisation missing for brevity
  
  const chat = model.startChat({
    history: [
      {
        role: "user",
        parts: "Hi there!",
      },
      {
        role: "model",
        parts: "Great to meet you. What would you like to know?",
      },
    ],
    generationConfig: {
      maxOutputTokens: 100,
    },
  });
  const prompt = 'What is the largest number with a name? Brief answer.';
  const result = await chat.sendMessage(prompt);
  const response = await result.response;
  console.log(response.text());
  }

You can use the initial user message in the history as a system prompt. Just remember to include a model response acknowledging the instructions. Example: User: adopt the role and writing style of a pirate. Don’t lose character. Reply understood if you understand these instructions. Model: Understood.

Generate content as is created using streaming (stream)

This example demonstrates how to use Gemini Pro to generate content using streaming.

async TestGeminiProStreaming() {
  // Model initialisation missing for brevity
  
  const prompt = {
    contents: [
      {
        role: 'user',
        parts: [
          {
            text: 'Generate a poem.',
          },
        ],
      },
    ],
  };
  const streamingResp = await model.generateContentStream(prompt);
  for await (const item of streamingResp.stream) {
    console.log('stream chunk: ' + item.text());
  }
  console.log('aggregated response: ' + (await streamingResp.response).text());
}

As a result of generateContentStream you will receive and object where you can read each chunk stream as is generated and the final response.

Bonus: generate AI content with Vertex AI via the REST API

As an alternative to the official JavaScript client you can use the Gemini REST API provided by Vertex AI. Vertex AI is a full AI platform made available as a managed service in Google Cloud where you can train and deploy AI models including Gemini.

To secure your REST API access, you need to create an account and get the credentials for your application so only you can access it. Here are the steps:

Sign up for a Google Cloud account and enable billing — this gives you access to Vertex AI.
Create a new project in the Cloud Console. Make note of the project ID.
Enable the Vertex AI API for your project.
Install the gcloud CLI and run gcloud auth print-access-token. Save the printed access token - you’ll use this for authentication.

Once you have the project ID and access token, you’re ready to move on to the Angular app. To verify everything is setup correctly you can try these curl commands.

Edit the development file to include the project ID and access token:

// src/environments/environment.development.ts
export const environment = {
  API_KEY: "", // Google AI JavaScript SDK access
  PROJECT_ID: "", // Vertex AI access
  GCLOUD_AUTH_PRINT_ACCESS_TOKEN: "", // Vertex AI access
};

To make requests via the REST API, you need to include the HttpClient provider:

// app.config.ts
import { provideHttpClient } from "@angular/common/http";

export const appConfig: ApplicationConfig = {
  providers: [
    provideRouter(routes),
    provideHttpClient()
  ]
};

With this import, we can inject HttpClient into any component or service to make web requests.

// app.component.ts
import { HttpClient } from '@angular/common/http';

@Component({
  selector: 'app-root',
  standalone: true,
  ...
})
export class AppComponent implements OnInit {
  constructor(public http: HttpClient) {}
}

To access Vertex AI via REST API we need to do a bit more work as there’s no client available. Without any help, it takes a bit more effort to build the request and read the response.

async TestGeminiProWithVertexAIViaREST() {
  // Docs: https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/gemini#request_body
  const prompt = this.buildPrompt('What is the largest number with a name?');
  const endpoint = this.buildEndpointUrl(environment.PROJECT_ID);
  let headers = this.getAuthHeaders(
    environment.GCLOUD_AUTH_PRINT_ACCESS_TOKEN
  );

  this.http.post(endpoint, prompt, { headers }).subscribe((response: any) => {
    console.log(response.candidates?.[0].content.parts[0].text);
  });
}

buildPrompt(text: string) {
  return {
    contents: [
      {
        role: 'user',
        parts: [
          {
            text: text,
          },
        ],
      },
    ],
    safety_settings: {
      category: 'HARM_CATEGORY_SEXUALLY_EXPLICIT',
      threshold: 'BLOCK_LOW_AND_ABOVE',
    },
    generation_config: {
      temperature: 0.9,
      top_p: 1,
      top_k: 32,
      max_output_tokens: 100,
    },
  };
}

buildEndpointUrl(projectId: string) {
  const BASE_URL = 'https://us-central1-aiplatform.googleapis.com/';
  const API_VERSION = 'v1'; // may be different at this time
  const MODEL = 'gemini-pro';
  let url = BASE_URL; // base url
  url += API_VERSION; // api version
  url += '/projects/' + projectId; // project id
  url += '/locations/us-central1'; // google cloud region
  url += '/publishers/google'; // publisher
  url += '/models/' + MODEL; // model
  url += ':generateContent'; // action
  return url;
}

getAuthHeaders(accessToken: string) {
  const headers = new HttpHeaders().set(
    'Authorization',
    `Bearer ${accessToken}`
  );
  return headers;
}

Running the code

Uncomment the code you want to test within the ngOnInit body from the GitHub project.

ngOnInit(): void {
  // Google AI
  this.TestGeminiPro();
  //this.TestGeminiProChat();
  //this.TestGeminiProVisionImages();
  //this.TestGeminiProStreaming();
  
  // Vertex AI
  //this.TestGeminiProWithVertexAIViaREST();
}

In order to run the code run this command in the terminal and navigate to localhost:4200.

ng serve

Verifying the response

To verify the response, you can quickly check the Console output in your browser.

console.log(response.text());

The largest number with a name is a googolplex. A googolplex is a 1 followed by 100 zeroes.

Congratulations! You now have access to Gemini capabilities.

Conclusions

By completing this tutorial, you learned:

How to obtain an API Key and set up access to Gemini APIs
How to call Gemini Pro using text and chat
How to handle input images for Gemini Pro Vision
Bonus: How to setup and call Gemini using Vertex AI via the REST API
How to handle the response and outputs

You now have the foundation to start building AI-powered features like advanced text generation into your Angular apps using Gemini. The complete code is available on GitHub.

Want to see a more complex project?

I built a full blown Gemini Chatbot using Angular Material, ngx-quill and ngx-markdown showcasing text, chat and multimodal capabilities.

Feel free to fork the project here. If you like this project don’t forget to leave a star to show support of my work and to other contributors in the Angular community.

Thanks for reading!

Have you got any questions? Feel free to leave your comments below or reach out to me on Twitter at @gerardsans.

Resources

Build AI-Powered Angular Apps with Google Gemini was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

Getting Started with Generative AI in Angular

Gerard Sans — Mon, 14 Aug 2023 14:38:31 GMT

Vertex AI is Google Cloud platform for state-of-the-art Generative AI including PaLM 2

IMPORTANT: Google Gemini, a new model, has become the default and a new JavaScript client is available from Google AI. For an updated article check here

In this post, you are going to learn how to access to the just released PaLM APIs to create the next generation of AI-enabled Applications using Angular.

If you are based in the US, you can skip to the annex at the end using an API key from MakerSuite. This tutorial, covers accessing Vertex AI with Google Cloud credentials for countries where MakerSuite is not available yet.

For Firebase based projects, you can use Call PaLM API Securely, a Firebase extension. Find more details about how to set it up here.

We’ll use Angular to build a simple application that prompts the PaLM for Text model. Once you get this working, you are set to use any of the other models. Here’s what we’ll cover:

Introduction to Generative AI
Responsible AI
Getting API credentials for Vertex AI
Setup for API access via Google Cloud
Creating the Angular Application
Making the Prediction Request
Handling the Prediction Response
Conclusion
Annex 1: using API key from MakerSuite? Join the waitlist otherwise
Annex 2: Prompt basics: temperature, topK and TopP
Annex 3: Response Metadata: Citations, Safety Attributes and Billing

This tutorial will help you get set up to Vertex AI so you can start adding AI-enabled features using models like PaLM in your applications. Let’s get started!

Introduction to Generative AI

Vertex AI provides pre-trained models for tasks like text generation, chat, image editing, code, speech and more. You can access these via API requests. Alternatively, if you need managed services to create models from scratch and fine tune them you can also use these to create, host and manage your models for you.

Vertex AI: Foundational Models. Embeddings available for text and images.

What is Generative AI?

Generative AI is a very exciting new area within AI that falls into Artificial Intelligence (AI), Machine Learning (ML) and Neural Networks. The Transformer together with the attention mechanism released back in 2017 by Google is what powers, the new wave of Generative AI and the Foundational models.

Large language models are pre-trained on massive text with the goal of predicting the next word in a sequence. Once trained, they can generate new text relevant to the given prompt being an open-ended paragraph in a novel, a list of ingredients for a recipe or providing an answer to a general knowledge question.

The training data includes public data available on the Internet, books and code repositories similar to GitHub.

How are people using Generative AI today?

Models based on the Transformer have surpassed many traditional ML approaches. Most recently, research has focused in multi-modal architectures that can be trained and combine inputs and outputs not only as text but as images, video and audio in a single model. This is the case for Gemini, a model being developed by DeepMind, the creators of AlphaGo.

Some use cases for Generative AI sorted by output.

Some examples of how people are using Generative AI today:

Creative text generation
Chatbots and virtual assistants
Summarisation of text
Image generation and editing

Responsible AI

While Generative AI enables exciting new applications, these powerful models also introduce important ethical questions we must grapple with as developers. We have a responsibility to build AI thoughtfully by considering potential harms, setting appropriate constraints, and monitoring for misuse.

As we integrate these models into applications, we need to implement strategies like human review processes and sensitivity analysis to prevent abusive or dangerous uses of the technology.

Some questions you should ask before releasing any new AI features:

Auditing and traceability – How will we facilitate tracing back to the source of abusive or questionable AI usage? What logging and auditing capabilities should we implement?
Prompt framing and grounding – How will we ensure our application provides proper context for AI-generated content? Without context, misinterpretation can easily occur.
Outputs monitoring – If incorrect output does occur, how will it be corrected? What feedback mechanisms should exist for users?
Bias mitigation. How might the AI output negatively impact or unfairly characterise underrepresented groups? How will we proactively test for bias?

For more information, read Google’s approach to responsible AI.

Now that we’ve covered the key concepts, let’s walk through how to access Vertex AI and make requests from an Angular application.

Getting API Credentials for Vertex AI

In the previous section, we introduced Vertex AI and discussed the models and capabilities it provides. These are the steps to get credentials for accessing Vertex AI.

General overview of the architecture.

⚠️ Note that this approach is only temporary while no official Vertex AI Client for Web is available. If you are developing for the server-side do use available clients for Java, Node.JS, Python and Go.

Setup for API access via Google Cloud

To secure the API access, you need to create an account and get the credentials for your application so only you can access it. Here are the steps:

Sign up for a Google Cloud account and enable billing — this gives you access to Vertex AI.
Create a new project in the Cloud Console. Make note of the project ID.
Enable the Vertex AI API for your project.
Install the gcloud CLI and run gcloud auth print-access-token. Save the printed access token - you’ll use this for authentication.

Once you have the project ID and access token, we’re ready to move on to the Angular app. To verify everything is setup correctly you can try these curl commands.

Make sure that your access token remains private so you do not incur into expenses from unauthorised access.

Creating the Angular Application

We’ll use the Angular CLI to generate a new application:

ng new vertex-ai-palm2-angular

This scaffolds a new project with the latest Angular version.

To make API requests, we need the HttpClient module:

// app.module.ts
import {HttpClientModule} from '@angular/common/http';

@NgModule({
  imports: [
    HttpClientModule
  ] 
})

With this import, we can inject HttpClient into any component or service to make web requests.

Making the prediction request

Let’s create a component to run our prompt:

ng generate component predict

In predict.component.ts, we’ll use the code below to set up the HttpClient and prepare the stage for the API request:

import { Component, OnInit } from '@angular/core';
import { HttpClient, HttpHeaders } from '@angular/common/http';
import { createPrompt, TextRequest, TextResponse } from '../models/vertex-ai';

@Component(...)
export class PredictComponent implements OnInit {
  endpoint: string = "";
  headers: HttpHeaders | undefined;
  prompt: TextRequest = createPrompt("What is the largest number with a name?");

  constructor(public http: HttpClient) {
  }

  ngOnInit(): void {
    this.TestVertexAIWithoutApiKey();
  }
}

To make the call, We’ll break this down into a few steps.

Creating the prompt

To create the prompt, we are using a helper function which will return the request JSON object that we will be used to send our prediction.

/// src/app/models/vertex-ai.ts

export function createPrompt(
  prompt: string = "What is the largest number with a name?",
  temperature: number = 0.7,
  maxOutputTokens: number = 100,
  topP: number = 0.95,
  topK: number = 40
): TextRequest {
  const request : TextRequest = {
    "instances": [
      {
        "prompt": `${prompt}`
      }
    ],
    "parameters": {
      "temperature": temperature,
      "maxOutputTokens": maxOutputTokens,
      "topP": topP,
      "topK": topK
    }
  }
  return request;
}

Building the Endpoint URL

The endpoint requires consolidating the base URL, API version, your project ID, Google Cloud region, publisher, the model name, and the specific action. We concatenate these together to form the full prediction endpoint URL.

buildEndpointUrl(projectId: string) {
  const BASE_URL = "https://us-central1-aiplatform.googleapis.com/";
  const API_VERSION = 'v1';        // may be different at this time
  const MODEL = 'text-bison';      

  let url = BASE_URL;              // base url
  url += API_VERSION;              // api version
  url += "/projects/" + projectId; // project id
  url += "/locations/us-central1"; // google cloud region
  url += "/publishers/google";     // publisher
  url += "/models/" + MODEL;       // model
  url += ":predict";               // action

  this.endpoint = url;
}

This endpoint in particular calls the predict API on text-bison. If you want to try any of the other available APIs (chat, code, images, speech) find more details here. Just be aware that each API has different endpoints, requests and responses.

Authentication

Next, we added the access token retrieved earlier to the HTTP headers to authenticate the request:

getAuthHeaders(accessToken: string) {
  this.headers = new HttpHeaders()
    .set('Authorization', `Bearer ${accessToken}`);
}

In order to comply with the security credentials, we added an auth header.

This access token is temporary so you may need to renew it by running this command again gcloud auth print-access-token.

Making the request

Finally, we use all the previous steps to make a POST request to the prediction endpoint. To handle the response, we define a subscribe callback to print out to the console the content for the prediction. This is part of the response object as seen in the code below.

TestVertexAIWithoutApiKey() {
  const PROJECT_ID = '';
  const GCLOUD_AUTH_PRINT_ACCESS_TOKEN = '';
  
  this.buildEndpointUrl(PROJECT_ID);
  this.getAuthHeaders(GCLOUD_AUTH_PRINT_ACCESS_TOKEN);
  
  this.http.post(this.endpoint, this.prompt, { headers: this.headers })
    .subscribe(response => {
      console.log(response.predictions[0].content);
    });
}

Be sure to substitute your actual project ID and access token retrieved in the credential steps here.

Handling the Prediction Response

In order to run the code run this command in the terminal and navigate to localhost:4200.

ng serve

Verifying the response

To verify the response from the prediction call, you can quickly check the Console output in your browser.

console.log(response.predictions[0].content);

{
  "predictions": [
    {
      "content": "The largest number with a name is a googolplex. A googolplex is a 1 followed by 100 zeroes."
    }
  ],
}

Congratulations! You now have access to Vertex AI APIs. And that’s really all you need to integrate PaLM for Text predictions into an Angular application. The complete code is available on GitHub.

Conclusion

By completing this tutorial, you learned:

How to obtain credentials and set up access to the Vertex AI APIs
How to create an Angular application using the CLI
How to make a prediction request to a generative model
How to handle the response and output the generated text

You now have the foundation to start building AI-powered features like advanced text generation into your own Angular apps using Vertex AI.

Thanks for reading!

Have you got any questions? Feel free to leave your comments below or reach out on Twitter at @gerardsans.

Credits: AI Assistants

This post was co-created with the help of a few AI assistants. Claude collaborated with me throughout the writing process — from structuring ideas to drafting content. Their tireless efforts as my virtual writing buddies were instrumental in crafting this tutorial. Later, Bard and ChatGPT punctually help me out getting to the final version.

Try Claude for FREE here: https://claude.ai

Annex 1: Using API key from MakerSuite? Join the waitlist

In this annex you will learn how to use your API key to access Vertex AI. First, make sure you have a valid API key from MakerSuite or request joining the waitlist here if you live outside the US. Once you have it you can quickly verify it with a curl request.

Once you receive your API key follow these steps:

Changes to the Prediction Component

Now the code to make the prediction call becomes:

TestVertexAIWithApiKey() {
  const API_KEY = '';

  this.buildEndpointUrlApiKey(API_KEY);

  this.http.post(this.endpoint, this.prompt)
    .subscribe(response => {
      console.log(response.predictions[0].content);
    });
}

Replace API_KEY with yours, pass the prompt text in the request body, and make a POST request to the prediction endpoint. Let’s quickly go through the remaining code. I tried to broke buildEndpointUrl down so you can see how each piece connects together.

buildEndpointUrlApiKey(apikey: string) {
  const BASE_URL = "https://generativelanguage.googleapis.com/";
  const API_VERSION = 'v1beta2';   // may be different at this time Eg: v1, v2, etc
  const MODEL = 'text-bison-001';  // may be different at this time

  let url = BASE_URL;              // base url
  url += API_VERSION;              // api version
  url += "/models/" + MODEL        // model
  url += ":generateText";          // action
  url += "?key=" + apikey;         // api key

  this.endpoint = url;
}

This endpoint in particular calls the generateText API on text-bison-001. If you want to try any of the other available APIs (chat, code, images, speech) find the details here. Just be aware that each API has different endpoints, requests and responses.

Notice how in this version of the code the API key is part of the query string.

Full code is available on GitHub.

Annex 2: Understanding prompt basics: temperature, topK and TopP

Prompt parameters are some basic arguments that we can fiddle with to influence the output for a given prompt. For example, these can be used to get less common and more creative outputs.

Tokens vs words

While we submit prompts to the PaLM API in the form of text, the API converts these strings into chunks called tokens. A token is approximately four characters. 100 tokens correspond to roughly 60 to 80 words.

{
  "predictions": [
    {
      "content": "The largest number with a name is a googolplex. A googolplex is a 1 followed by 100 zeroes."
    }
  ],
  "metadata": {
    "tokenMetadata": {
      "inputTokenCount": {
        "totalTokens": 
      },
      "outputTokenCount": {
        "totalTokens": 
      }
    }
  }
}

Some tasks may require you to review the default settings for these parameters to get better results.

Diagrams by Cohere. How prompt parameters and temperature influence the output.

Let’s look at these in more detail:

prompt. This parameter becomes the text input. Eg: What is the largest number with a name? Feel free to change this prompt as is only to test access to the API. Mandatory.
temperature. This parameter allows you to introduce less common options during the decoding strategy thus rising up the chances for a more creative or less repetitive responses. When set to 0 a greedy strategy is used and the first option is picked. Recommended.
maxOutputTokens. This parameter limits the total amount of tokens generated by this prompt. Note output can be truncated leaving part of the answer missing. Be aware of costs while testing. The maximum value is 1024. Recommended.

Most of the time, the default values will be enough.

Extending the reach for your responses: TopK and TopP

Some times you will be testing a prompt and looking for different responses that can better fit your expectations or be a better match for the task at hand. In these cases you can use TopK and TopP to extend the scope for more and different responses.

Note that these parameters require a deeper understanding of the model and how probability distributions work for a given prompt.

As shown in the diagram below, given a certain temperature, you can introduce more variability by increasing topP and topK respectively. The final result will depend on the probability distribution present for that specific prompt.

Diagrams by Cohere. Shows outputs changing TopP and TopK for different Temperatures

As a review:

topK. This parameter limits the sampled options to k . Default value is set to 40. Optional.
topP. This parameter limits the aggregated probabilities up to p. Default value is 0.95. Values between 0 and 1. Optional.

When both topK and topP are provided topK takes precedence before applying topP. Finally a random selection is made within the sampled tokens. The probabilities for each token are the result after the training for that model ends.

Common strategies when testing your prompts:

Limit response to the most common tokens seen in training. Set temperature to zero and topP to 0.2. This will respect the probabilities from the model and chose a token that seats within the 20% of cases seen by the model during training. This will also limit the variability for long-tail distributions if that’s the case for your prompt.
Allow less common tokens and more creative responses: Set temperature to 0.7 and topK to 40. This will rise the probabilities for less common cases so they can show in the output. Note this strategy depends on each prompt and how many training data relevant to it was present during training.

Annex 3: Response Metadata: Citations, Safety Attributes and Billing

As part of the response you will also get access some interesting metadata around citations, safety and billing. See below the whole structure for reference. For more details, see Responsible AI.

{
  "predictions": [
    {
      "content": "The largest number with a name is a googolplex. A googolplex is a 1 followed by 100 zeroes."
      "citationMetadata": {
        "citations": []
      },
      "safetyAttributes": {
        "scores": [],
        "blocked": false,
        "categories": []
      }
    }
  ],
  "metadata": {
    "tokenMetadata": {
      "inputTokenCount": {
        "totalBillableCharacters": ,
        "totalTokens": 
      },
      "outputTokenCount": {
        "totalBillableCharacters": ,
        "totalTokens": 
      }
    }
  }
}

Safety

Generative AI is still very new and while adding AI-enabled features to your Apps be aware of the current PaLM API limitations, including:

Model Hallucinations, Grounding, and Factuality
Biases and Limited Domain Expertise

Learn how you can mitigate these by following the recommendations found in Responsible AI.

Vertex AI Billing

As a reference, 500 requests and corresponding 500 responses, will cost you (per 1,000 characters):

PaLM for Text: $1 (used in this post)
PaLM for Chat or Codey: $0.5
Embeddings for text or images: $0.1

Prices may vary over time. Note input and output are charged separately. See latest pricing.

Getting Started with Generative AI in Angular was originally published in Google Cloud - Community on Medium, where people are continuing the conversation by highlighting and responding to this story.

OpenAI — The Start of a New Era in AI

Gerard Sans — Tue, 11 Oct 2022 12:00:43 GMT

OpenAI — The Start of a New Era in AI

Discussion hosted by Darek Kalbarczyk featuring Gerard Sans

Image edition using DALL-E, one of the most popular products of OpenAI.

This is an edited and reviewed version of the transcript for JavaScript Master Podcast by Darek Kalbarczyk featuring Gerard Sans using GPT-3.

Introduction
How AI is going to replace some jobs?
Which jobs are more likely to be affected: writers, artists, or developers?
What is OpenAI?
What products are being offered today by OpenAI?
What is GPT-3 trying to solve?
Cooking recipes created by an AI!?!
How it could learn Japanese with almost no information?
What is Codex main purpose?
What is DALL-E and how people are using it?
Can you use DALL-E for interior design?
Can you use OpenAI today? How would that work?
What are the benefits of using those tools? Are there any drawbacks?
What the future of AI may look like in five years?
Do you think AI can help us build something we can’t do ourselves for now, like a super engine for a rocket?
Can you give some advice to people who are starting their career in IT today? And for those who are old-timers?

You can also listen to the original podcast at the end of this blogpost.

Darek: Hi, Gerard. How are you?

Gerard: Hello, Darek. I’m very good thanks!

Darek: please tell us a bit about yourself.

Gerard: my name is Gerard Sans. I’m a Developer Evangelist for Web and Cloud. I’ve been doing some research on Artificial Intelligence and Web3 for the last year and I’m here to share some of my learnings with you.

Darek: there’s been a lot of discussions around how AI is going to replace some jobs. What do you think about that?

Gerard: these are very exciting times. We’ve seen a lot of improvements in AI in the past few years. Some of them will certainly change how people do their jobs. The most significant changes have happened in text-to-image generation. This is where an AI model generates images from scratch by using a prompt. We’ve seen amazing progress in this area. You can easily generate illustrations or even photorealistic images, and it keeps getting better and better. A lot of designers and digital artists are seeing how their jobs are suddenly in the spotlight. At this time, it’s not clear if people will care if an artwork was created by a digital artist or an AI.

This white t-shirt doesn’t exist, it was created by DALL-E.

Darek: which jobs are more likely to be affected: writers, artists, or developers?

Gerard: AI’s latest developments have been very surprising for me. Initially, I was in awe looking at some of the text generated by AIs. This technology is very similar to what I explained for image generation but writing. These AI models have been trained using a very large amount of data including publicly available books from the last few centuries. So these models can generate text in all the styles that you can imagine — commercial books, self-help, cooking recipes — but also philosophy, math, and all of the sciences. If you think about important authors like Shakespeare or other famous authors in history, their styles are also embedded in these models. So today, you can effortlessly generate text that resembles any of the most famous writers in the world. Like DALL-E generating an image in the style of Van Gogh. This is not just limited to literature but also includes all sorts of specific domains like marketing.

For example, if you are a writer, you could use these AI tools to avoid what’s called writer’s block. You can get inspiration from a history that you are stuck on and get a fresh start following your same style. People are using this technology to generate anything from blog posts to cooking recipes and anything in between. It’s quite exciting but also quite uncertain how it will affect jobs like technical writers or even novelists. These jobs are all in the spotlight. Most people know about these AI technologies but they don’t think that they will be affected by them.

Most people know about these AI technologies but they don’t think that they will be affected by them

It was only last year that OpenAI released Codex, which is the model behind GitHub Copilot a product by Microsoft. Codex can write code from scratch but also introduce changes following commands. This is anything that you need as a developer. You can write code and then iterate over it as many times as you need. These are only a few examples of what these tools are capable of today. So yeah, it’s looking like we are facing the birth of a new era of AIs.

Using Codex you can specify what you need in a comment and it will create the code for you.

Darek: what is OpenAI?

Gerard: OpenAI is a company that was founded by Elon Musk, Sam Altman, and Greg Brockman. Their mission is to build a safe Artificial General Intelligence (AGI) that benefits all humanity.

OpenAI was founded back in 2015 as a non-profit company. It took them a few years until 2018 to create the first GPT model which is a Generative Pre-trained Transformer. It is also known as a Large Language Model (LLM). They published a public API in 2020 which is when everyone started playing with this technology and started discovering its many applications. The first applications were focused on writing but very soon they realised that GPT-3 was able to do things like writing valid JavaScript. That was when they started working on Codex which was released to the public in 2021. This is a new technology only been around for a few years and is just fantastic.

NLP models up to 2022 (not included)

Darek: what products are being offered today by OpenAI?

Gerard: there are three different products available from OpenAI.

The first is GPT-3, a model that can write text indistinguishable from human writing. It works by providing a starting text, a prompt, and GPT-3 will keep writing in a way that matches the existing text. For example, you can use it to create a story, or continue a conversation between characters. Some of these capabilities were not planned at all by their creators. This model was created to generate text by predicting the next word following a prompt, but then they realised that the model could identify more complex patterns, like a series of questions and answers, or even a general conversation. During the beta access, users found out that GPT-3 was able to do a lot of different things. You can, for example, ask for cooking recipes and it will give you the ingredients and how to cook them. That’s not text generation anymore, in the sense of writing a story or writing a book, but it’s more like identifying a question and providing an answer to that question. During the initial stages, a lot of different applications were identified. One of these was that it could write HTML, CSS, and JavaScript.

This led to the creation of Codex, a product specialising in coding. This is when OpenAI partnered with Microsoft, getting access to 59 million GitHub repositories. The outcome was that it learned how to use 12 different languages, including SQL and Bash commands.

Finally, DALL-E is the one I started talking about at the beginning.

Darek: what is GPT-3 trying to solve?

Gerard: GPT-3 is a Natural Language Processing (NLP) model that generates text. GPT-3 can be given a text input and then generate any number of words. Usually, a stop sequence is used, so that GPT-3 will stop after the first paragraph. However, you can create complex patterns, such as a question-and-answer text, and GPT-3 will follow that. You can also ask for instructions and it will provide the instructions to achieve a goal, such as cooking a recipe.

Most common usages for GPT-3 by OpenAI.

Darek: Cooking recipes created by an AI !?!

Gerard: yeah. It’s quite an exciting development. The idea behind it is that these models are very large. GPT-3 is 175 billion parameters large. That means it can generate more combinations than atoms in the universe. So this is a very big model that we cannot begin to imagine. And of course, one of the things it contains is cooking recipes. But this is not the most surprising thing that you can find inside these models.

The model can code in JavaScript! But it doesn’t stop there. Now it has also learned Typescript, Python, Ruby, Go, C sharp, Swift, Rust, PHP; and all of these languages after just a little bit less than half a terabyte of extra training.

Probably the weirdest thing is that it can speak more than a hundred languages, even though it was only trained in English. This is very surprising. In some languages, GPT-3 was able to learn using less than one percent of data. We don’t know how that happened. It was only supposed to learn English.

Training Dataset for GPT-3.

One of the largest sources of information was the Common Crawl. These are the top websites and all their accompanying raw files. That’s how it learned JavaScript because a lot of its content is HTML, CSS, and JavaScript files. But then it also took information from hundreds of years of public books (Books1–2). And funny enough, from Reddit too (WebText2). So there’s a lot of information from users and the links that these users provided. So anything you can imagine is in there.

The Common Crawl corpus contains petabytes of data collected over 12 years of web crawling.

Another resource was Wikipedia. So anything that is in Wikipedia is also part of the training data for GPT-3. It’s a massive amount of data. It can do things like translate between languages. So you can write something in French and it can give you the translation in Polish.

If you look at the complexity of a language like English, then you can understand how it’s able to learn JavaScript, because JavaScript is much simpler than English.

Darek: Yeah exactly.

Gerard: If you’re wondering how it could learn Japanese with almost no information, you’re not alone. This is just the beginning of what AI is capable of, and we’ve only seen a fraction of its potential.

GPT-3 is so large that it would be impossible for any one person to explore its entire latent space. What we’ve seen so far is just a drop in the bucket. For example, GPT-3 being able to generate cooking recipes is a small feat compared to what it’s capable of.

latent space is a mathematical space that maps what a neural network has learned from training

It’s been more than half a year since GPT-3 was first used in production with GitHub Copilot, and it’s still going strong. This shows that it’s not only capable of understanding JavaScript but also able to provide code to professional programmers who are building all sorts of applications.

In short, we haven’t seen anything yet. GPT-3 is just getting started, and its potential is massive.

Darek: what is Codex main purpose?

Gerard: the main purpose of Codex is to generate valid code. As we have explained before, it is capable of writing code in many different programming languages, which makes it a valuable tool for developers. However, it is important to keep in mind that Codex was trained with a snapshot of GitHub repositories from last year. Anything after that snapshot, will not be aware of. The reason for not being trained regularly is that the process is very expensive and requires massive amounts of computing.

Main languages supported by Codex.

Darek: what is DALL-E and how people are using it?

Gerard: DALL-E is a model from OpenAI that is trained to generate images from a prompt. It is by far the most popular. DALL-E is everywhere on social media, from TikTok to Twitter. We have seen how OpenAI models learned to generate text and code, and this is the same for images.

Everything that GPT-3 knows, DALL-E knows. This means that it’s a writer, it’s also a developer and now it has also data around images. DALL-E was trained with 400 million images along with its metadata (title, description, creation date, camera, lenses, geolocation, etc). This allowed DALL-E to establish relationships between images and words. As a result, it can generate any sort of image following a prompt very closely. You can either generate an image from scratch or edit images by giving it instructions.

For example, one of the experiments that I did recently was to create a photorealistic picture of a woman using DALL-E. I provided information about the composition, how the model was positioned, and the lighting, and the camera angle. I got the picture I was after, and then I changed the image and asked DALL-E to add a black leather jacket to the model. Not only did it add the jacket to the picture, but it also took into consideration the volume, the lighting, and just placed the jacket on the model in a realistic way!

From left to right: initial image, editing and final result. Face fixed using tencentarc/gfpgan.

DALL-E can not only do photography, but pixel art, 3D models, games, and illustrations too. Of course, photography is the most impressive, but you can also do things like hand-drawn illustrations. It’s something else.

People are using DALL-E to create games because creating 3D models requires quite a lot of work. Sometimes you may need to create textures. Let’s say you are creating a castle and there’s some pattern in the walls which is sometimes made of rock or wood. It’s a difficult task for a digital artist to create a lot of variations for these textures. Today, you can give it to DALL-E and it will create the textures and variations for you. You can then use it in a 3D model and in a few minutes, you have got everything ready to go. You can build games fast using tools like DALL-E.

Darek: That sounds amazing.

Gerard: Yeah, there are some fancy use cases for it. For example, I can see that you have a sofa in the background. So, you could take a picture of a room and ask DALL-E to add furniture. So, imagine something is missing, maybe you want to add some artwork. You can give this task to DALL-E and it will just decorate your room in different styles if you like.

People are also using DALL-E to recreate historic scenes. For example, you could ask it to give you a selfie of a soldier in Normandy during World War II. It’s anything that you can imagine. DALL-E can use all the information that you give it. It’s more like having a creative director that puts all the information together and makes it work.

Of course, the technology is very new. We are talking about a technology that didn’t exist last year. So, in just one year it has made a lot of progress. But we will see many other things happening with all of these different products. DALL-E is probably the most surprising because it’s very visual.

Darek: can you use OpenAI today? How would that work?

Gerard: the OpenAI API is available to anyone since 2021 by creating an account and buying some credits. You just need to get your API key, like you do with similar services, and you are ready to go. For Codex API, you will need to apply for beta access. So far there’s no API for DALL-E but you can use all of the models using the OpenAI Playground.

Available APIs from OpenAI.

Darek: what are the benefits of using those tools? Are there any drawbacks?

Gerard: I think one of the benefits of GPT-3 is that it can boost your productivity. So if you need to create a technical blog post, for example, GPT-3 can help you do that in a fraction of the time it would normally take. With other products, it may depend on what you want to use them for. Sometimes the quality isn’t quite there yet, but you can use parts of the results of the outputs to speed up your project. So it can just depend on what exactly you want to use it for, but GPT-3 can help you be more productive.

As a drawback, I would say that GPT-3’s reliance on statistical methods means that there is no easy way to know if the information it gives you is factual. So if you are writing a blog post and some of the information is not factual, it can be a problem. Depending on how much you rely on AI, you will need to review it. You cannot just take the output from GPT-3 and use it in a publication and release it without checking the facts as you may be embarrassed later on.

It just depends on what you’re trying to do. Sometimes when you try to do something complex, there can be some issues with the technology. For example, in DALL-E, when it creates an image, human faces in the pictures don’t always look perfect. But I think these errors or these drawbacks are going to be fixed and then it’s going to be much, much faster to use these tools.

Darek: what the future of AI may look like in five years?

Gerard: I just did a talk around this topic where I described the AI Revolution. This is from the work of different authors.

The first stage is called the Cognitive Revolution. This is where we use AI Assistants to get our job done faster. And depending on what you do, that will work best or not. I mean, even for GitHub Copilot, it’s not that the tool is creating everything for you. It’s just giving you some kind of productivity boost.

The next step, which is a little bit more far-fetched and it’s probably not going to be that soon, is the Robotics Revolution. This means fully autonomous robots. I think I saw the first robot in Silicon Valley some years ago. They go around a complex and they can give you information about what you can visit. But of course, this is not something that you can find everywhere yet, but in the future, we will see more examples of these.

The third stage is when AIs get smarter than humans which is called Singularity. At this point, the AIs will be similar to the character in Star Trek, Data. This is an Android that helps Star Trek characters when they struggle or they go on adventures. If there’s some danger and they need to make a decision, they ask Data.

People are not certain when all of these different stages will happen but there’s a recent quote from Elon Musk where he said that we are going to be overtaken by AIs in the next five years.

Darek: do you think AI can help us build something we can’t do ourselves for now, like a super engine for a rocket?

Gerard: yes. What is happening is that with these models we have created some kind of super intelligence. This means that they have all the information accessible at the same time. So one of these technologies that are being used behind the scenes is neural networks. This allows for a faster response time, as they don’t need to calculate. The neural network is more like a human in that it doesn’t even think, it just gives you the answer right away. When you use the systems GPT-3 and Codex, it is not taking seconds or even milliseconds to answer back. That means that all of the information is accessible to them right now.

Darek: so when you ask it for example how to build a rocket engine, it will give you the answer right away?

Gerard: exactly. The moment you ask, it will give you the answer right away. It doesn’t need to think about it for a while and give you an answer next week. No. It gives you the answer right away.

Darek: what’s your conclusion?

Gerard: my conclusion is that nobody knows what is happening with these models but it’s exciting because people find new use cases and they find new things every day that you can use these models for. There has also been news about open-sourcing some of these technologies so you can use them in your projects which is also very exciting.

Darek: can you give some advice to people who are starting their career in IT today? And for those who are old-timers?

Gerard: I think a good piece of advice is to keep your mind open because everything is changing. People think that things don’t change, but in the area of AI, things are changing fast. AI was not a technology that has been changing a lot. It’s kind of an old area, but the discoveries made in the last few years have exploded. It’s more like JavaScript. You know the new JavaScript that happened a few years back. And people were surprised like, “Oh, can you do this with JavaScript?” I mean before we didn’t use JavaScript in this way. So we are facing now a new generation of AI and this will change everything. A lot of the jobs that we do today won’t exist in the future. So if you’re not retiring, I mean if you’re an old-timer you maybe still have five, ten years, or 15 years more in your career, I think you should look into what’s happening in blockchain, what’s happening in web3. Keep your mind open, because you can become obsolete very fast.

New generations, I think should also be careful about what they focus on. If you focus on learning a specialised skill and then realise that anyone can be productive in that technology because of AI assistants, you will have wasted your time. Why would you want to memorise things or understand things when an AI can help you? So then it’s not clear that you want to go through specialisation of 15 years and then find out that someone who doesn’t even have a specialisation is using a tool that helps them achieve the same results in just five years. So then you have lost 10 years learning something that wasn’t necessary. So that’s something to be careful about.

Darek: thank you, Gerard, for joining today’s podcast and sharing your knowledge with us. I look forward to meeting you in Warsaw soon.

Gerard: thanks a lot for inviting me. I’m looking forward to another edition of the conference. Thank you so much. Bye-bye, everyone!

Thanks for reading!

Have you got any questions? Feel free to leave your comments below or reach out on Twitter at gerardsans.

Resources

You can also listen to the original podcast using the links below.

Spotify: https://spoti.fi/3CHg2Zb

Google: https://bit.ly/JSMP-5

Apple: https://apple.co/3CdOJUY

Anchor: https://bit.ly/JSMP-05

YouTube: https://bit.ly/JSMP-GS-5

Offline-first made easy with GraphQL, Amplify DataStore and Vue

Gerard Sans — Thu, 01 Oct 2020 15:33:39 GMT

Create a PWA cloud-enabled offline-first chatroom using Vue

In this article, you will create Chatty, a chatroom Progressive Web App (PWA) that stores data on the device while offline, and synchronises in real-time with other devices when online using Amplify DataStore, GraphQL and Amazon DynamoDB. We will cover:

Introduction to Offline-first, PWAs and Amplify DataStore
Setting up a new project with the Vue CLI
Creating a new GraphQL API
Amplify DataStore: setup, data models and usage
Creating the UI with Vue: the chatroom, sending and deleting messages; and doing real-time with subscriptions
Making Chatty a PWA
Adding a PWA custom configuration
Improving UX while offline
Publishing your app via the AWS Amplify Console
Installing Chatty in the Desktop and Mobile
Cleaning up cloud services

In order to follow this post you will need a basic knowledge of GraphQL. You can learn the basics following this tutorial at graphql.org. We will be referring to GraphQL schema, directives, types, queries, mutations, subscriptions and resolvers.

Please let me know if you have any questions or want to learn more at @gerardsans.

> Final solution and step by step tutorial in GitHub.

Introduction to Offline-first, PWAs and Amplify DataStore

Offline-first is an approach to software development, where an application is built to function, with or without an internet connection. Using this approach, data is stored locally on the user’s device and periodically uploaded and replicated into the cloud. Features include:

Offline ready: application works offline. This usually comes with some limitations regarding feature set and device capabilities. Eg: PWAs require an install step where the user needs to be online to download all the necessary application assets.
Great user experience: application loads fast and transitions seamlessly from online to offline improving user retention. The user is in control regarding app updates, data synchronisation, data conflict resolution and connectivity. Eg: App shell; user is informed about connectivity changes.
Native-like features: application behaves similar to a native app. Eg: PWAs are able to run on multiple operating systems as standalone applications and don’t require the user to interact with the browser.
Reliable storage: user can exit the application at any moment without losing data. Application stores user data securely on-device and synchronises with the cloud transparently when possible.

Progressive Web Apps

Progressive Web Apps combine the benefits from native apps, while being able to run on more platforms, leveraging browser technologies. PWA features allow Web apps to close the gap with native applications while creating similar user experiences. PWA features:

Works offline via service worker.
Great performance via app shell and pre-cached assets.
Access to device APIs via Web APIs.
Supports push notifications via service worker.
Works in Web, Mobile and Desktop via Web app manifest.

The most important requirements for PWA are: security, must be served via HTTPS, the service worker and the Web app manifest. We will incorporate all of these to Chatty.

Amplify DataStore

Amplify DataStore, is an on device persistent repository for interacting with local data, and able to automatically synchronise via GraphQL. Using Amplify DataStore, you can provide a great offline-first experience for your users, while using a simple programming model.

GraphQL knowledge is beneficial when using Amplify DataStore to understand its inner workings but not necessary to use DataStore APIs.

Amplify takes care of all the heavy lifting regarding hosting your data source in the cloud using Amazon DynamoDB; and scaling real-time messaging, synchronising data and running conflict resolution with AWS AppSync and GraphQL.

Amplify DataStore uses your GraphQL schema to generate data models. These will help keep your client code consistent with the type and fields defined in your GraphQL schema; and reduce type checks and data related run-time errors while enforcing GraphQL type safety.

Amplify DataStore behaves differently when in offline or online modes:

In offline mode, we use the DataStore API to manage the data on the device via the Storage Engine using GraphQL. The local Data Store being used will vary depending on the user device. This can either be IndexedDB in the browser or SQL Lite in Android and iOS.

Amplify DataStore data flow in offline mode

In online mode, Amplify DataStore, will synchronise with the cloud using the Sync Engine via GraphQL to automatically synchronise with your GraphQL API. At the query level, this is implemented as an AWS AppSync resolver accessing Amazon DynamoDB, AWS Lambda or Amazon ElasticSearch.

Amplify DataStore data flow in online mode

Amplify DataStore supports the following conflict resolution modes during synchronisation:

Optimistic concurrency, this mode rejects any incoming changes if there’s a mismatch in the tracking version between the incoming change and the item being changed. A further step on the client is required to resolve the conflict.
Auto merge (default), this mode doesn’t require further steps to resolve conflicts. It accommodates mutations on a field-by-field basis as long as scalar types don’t clash. If they do, they are ignored but all other fields are merged, if possible.
Custom, this mode uses AWS Lambda to allow more control and complex use cases.

For more details read conflict detection and sync.

Setting up a new project with the Vue CLI

Before moving to the next section, please complete the steps described in “Build your first full-stack serverless app with Vue”. Here you will set up the initial project, familiarise with Amplify CLI and add an authorisation flow so users can register themselves via an automated verification code sent to their email and login.

Creating a new GraphQL API

First, we are going to create the GraphQL API to service Chatty chatroom. To create it, we will use the following command:

amplify add api

Answer the following questions

Please select from one of the below mentioned services GraphQL
Provide API name: ChattyAPI
Choose the default authorization type for the API API key
Enter a description for the API key: (empty)
After how many days from now the API key should expire (1–365): 7

It’s important that you don’t miss the following steps to set up conflict detection. If you missed it, you can recover with amplify api update

Do you want to configure advanced settings for the GraphQL API Yes, I want to make some additional changes.
Configure additional auth types? No
Configure conflict detection? Yes
Select the default resolution strategy Auto Merge
Do you want to override default per model settings? No
Do you have an annotated GraphQL schema? No
Choose a schema template: Single object with fields (e.g., “Todo” with ID, name, description)
Do you want to edit the schema now? Yes

When prompted, replace the default schema with the following:

type Chatty @model {
  id: ID!
  user: String!
  message: String!
  createdAt: AWSDateTime
}

By adding@model to the Chatty type we are telling the Amplify CLI to create the resolvers to support queries, mutations and subscriptions in Amazon DynamoDB. Besides the regular CRUD operations, we are also getting some more advanced features like pagination, filtering and real-time synchronisation that we are going to use later.

Pushing your GraphQL API to the cloud

Let’s run the push command to create the GraphQL API:

amplify push

Are you sure you want to continue? Yes
Do you want to generate code for your newly created GraphQL API Yes
Choose the code generation language target javascript
Enter the file name pattern of graphql queries, mutations and subscriptions src/graphql/**/*.js
Do you want to generate/update all possible GraphQL operations — queries, mutations and subscriptions Yes
Enter maximum statement depth [increase from default if your schema is deeply nested] 2

Write down your GraphQL endpoint and API KEY.

This is the resulting architecture with our new GraphQL API managed by AWS AppSync and Amazon DynamoDB:

Run the command below to access the AWS AppSync console.

amplify console api

Please select from one of the below mentioned services GraphQL

Amplify DataStore setup

Run the command below to install the necessary dependencies:

npm install --save @aws-amplify/core @aws-amplify/datastore

Data model generation

Generate the data models to manage our messages for our ChattyAPI.

amplify codegen models

Important: DO NOT forget to regenerate your models every time you introduce a change in your schema.

After running this command, AWS Amplify CLI has generated the necessary data models in a new folder. The files in this folder hold your data model classes and schema.


    |_ src
      |_ models

Creating a message

Now that the GraphQL API and data models are created, we can begin using the DataStore API. The first thing we’ll do is create a new message passing the generated data model to save.

Using data models allow us to be always compliant with the GraphQL schema and prevent errors right in the client.

import { DataStore } from "@aws-amplify/datastore";
import { Chatty } from "./models";

await DataStore.save(new Chatty({
  user: "amplify-user",
  message: "Hi everyone!",
  createdAt: new Date().toISOString()
}))

This will create a record locally in your device and synchronise it in the background using the underlying GraphQL API when in online mode.

Querying data

Let’s see how we can query the data using Amplify DataStore. In order to query our data model we will use a query and a predicate to indicate that we want all records.

import { DataStore, Predicates } from "@aws-amplify/datastore";
import { Chatty } from "./models";

const messages = await DataStore.query(Chatty, Predicates.ALL);

This will return an array of messages that we can display in our UI. Predicates also support filters for common types like Strings, Numbers and Lists.

Find all supported filters at query with Predicates.

Creating the UI with Vue

Now, let’s look at how we can create the UI to create and display messages for Chatty chatroom.

After the user signs in, we call loadMessages. This fetches all the messages in our local store. We are sorting the list with a computed method sorted as we want our messages to show the most recent at the top. We are also using moment to format our time variables.

Creating a message

Now, let’s look at how we create new messages.

We use a form to read the message from the user and call sendMessage with save passing the user and current date along with the message. Once the message has been created we reset the input field and reload all messages. Note how we use Amplify DataStore as our source of truth in this implementation.

Deleting all messages

One of the main advantages of working using Amplify DataStore is being able to run a series of GraphQL mutations without having to use a series of individual operations. See below how we can use delete together with a predicate to remove all messages.

DataStore.delete(Chatty, Predicates.ALL).then(() => {
  console.log('messages deleted!');
});

When using Amplify DataStore records are never removed, but marked for removal. Records will then be automatically removed as defined in their TTL attribute. At the time of writing, this is set to 30 days.

Real-time with GraphQL subscriptions

Next, let’s see how we can create a GraphQL subscription. To do so, we will listen for changes of data in our API, and update the state whenever a new piece of data comes through. When the component is destroyed, we will unsubscribe to avoid memory leaks.

As we are using Amplify DataStore, as our source of truth, we don’t need another copy of it at the app level so we just need to load the latest state with loadMessages.

At this point, the chatroom is fully working including authentication flow and data synchronisation in real-time.

Real-time synchronisation using Chrome and Firefox clients side-by-side.

On the other hand, we haven’t made much progress in making it work offline or having native-like features. In order to tick these boxes, we are going to make our app a PWA.

Making Chatty a PWA

The PWA CLI plugin can help us making Chatty a PWA by running a single command:

vue add @vue/pwa

This will make few changes to the project and add a few new assets necessary to run the app as a PWA. The most important change happens in src/main.js where we are now registering a service worker that will allow Chatty to work offline.

Using the default settings, the service worker is configured to pre-cache all the assets resulting from the build (js, css and public folder). Using this setup, the browser can access the cache to load the app while offline. See the diagram below:

Service worker serving assets from the cache while offline

Learn more about service workers at using service workers.

Run the build command to see changes:

yarn build

In your dist folder you should find the following:

└── dist
  ├── css
  │   └── app..css
  ├── img/icons
  │   ├── android-chrome-.png
  │   └── ...
  ├── js
  │   ├── app..png
  │   └── ...
  ├── favicon.ico
  ├── index.html
  ├── manifest.json
  ├── precache-manifest..json
  ├── robots.txt
  └── service-worker.js

The index.html has been modified to include the Web app manifest manifest.json along with some meta tags and entries to support different devices and platforms. You can further customise these settings. The most important ismanifest.json/display set to standalone. This setting will give the app a look and feel like a native application for Desktop and Mobile. Learn about all the options available in Web app manifest.

Another important file is service-worker.js. This contains the implementation for the service worker which determines the behaviour of the app while offline. As part of it, precache-manifest..json includes all files to be installed and pre-cached as part of the app. This will make the app load almost instantly as it won’t use the network. These files were created by the PWA CLI plugin and include all the assets from your build.

Test the app running the following commands on a new terminal window:

cd dist
python -m SimpleHTTPServer 8887  // open localhost:8887

Remember that in order to test the app you need to install the service worker first. This requires at least loading the app once.

To test the app while offline you can use the Developer Tools or switch off your internet connection. See below a screen capture after refreshing the page while offline:

Chatty app while offline.

As you can see in the screenshot above using Chrome, in the Network tab, the files are served by the service worker while offline (in green). Unfortunately the default setup won’t include any custom assets like our SVG logo in the header (in red) or external fonts.

Adding a PWA custom configuration

Let’s fix the issue with our logo by using a custom configuration. Create vue.config.js with the following content at the root of your project:

// vue.config.js
const manifest = require('./public/manifest.json')

module.exports = {
  pwa: {
    name: manifest.short_name,
    themeColor: manifest.theme_color,
    msTileColor: manifest.background_color,
    appleMobileWebAppCapable: 'yes',
    appleMobileWebAppStatusBarStyle: 'black',
    workboxPluginMode: 'InjectManifest',
    workboxOptions: {
      swSrc: 'src/service-worker.js',
    }
  }
}

This file controls the PWA CLI plugin and deal with settings in the Web app manifest and UX for different devices.

The custom configuration uses workbox in injectManifest mode and points to the service worker implementation we want to use. Find all configuration options at the official website. Now create src/service-worker.js with the following code

// src/service-worker.js
workbox.core.setCacheNameDetails({ prefix: 'amplify-datastore' })

workbox.core.skipWaiting()
workbox.core.clientsClaim()

const cacheFiles = [{
 "revision": "e653ab4d124bf16b5232",
 "url": "https://aws-amplify.github.io/img/amplify.svg"
}]

self.__precacheManifest = cacheFiles.concat(self.__precacheManifest || [])
workbox.precaching.precacheAndRoute(self.__precacheManifest, {})

For our service worker, we are setting a prefix for our cache to easily find our files while using the Developer Tools. We are also setting the behaviour for new service worker versions. These will be applied immediately replacing any existing ones with skipWaiting and clientsClaim as explained here. To fix our SVG issue, we are adding the url to the SVG file, to the pre-cached files defined in the Web app manifest.

Improving UX while offline

We are almost done. Our app can survive offline reloads, can store messages while offline and can synchronise when back online thanks to Amplify DataStore.

The only remaining issue now is improving the user experience as the user doesn’t know when the app is operating online or offline. This is not a great UX, as sometimes the user might inadvertently go offline and be confused as for not receiving any feedback or explanation from the app. We can fix this by using the code below:

// You are offline.

//

// App.vue
import { Hub } from 'aws-amplify';export default {
  data() {
    return { offline: undefined };
  },
  created() {
    this.listener = Hub.listen('datastore', {payload:{event}} => {
      if (event === 'networkStatus') {
        this.offline = !data.active;
      }
    })
  }
}

We are using the Amplify Hub to detect network changes and use it to set the offline flag. Once in place, we can use it to display messages or style the UI accordingly as shown at the beginning of the code above. We are now ready to publish Chatty to the cloud and test it using different devices.

Chatty PWA informing the user of changes in the network status.

Publishing your app via the AWS Amplify Console

The first thing you need to do is create a new repo for this project. Once you’ve created the repo, copy the URL for the project to the clipboard and initialise git in your local project:

git init
git remote add origin repo@repoofyourchoice.com:username/project-name.git
git add .git commit -m 'initial commit'git push origin master

Next visit the AWS Amplify Console in your AWS account. Click Get Started to create a new deployment. Next, authorise your repository provider as the repository service. Next, choose the new repository and branch for the project you just created and click Next. In the next screen, create a new role and use this role to allow the AWS Amplify Console to deploy these resources and click Next. Finally, click Save and Deploy to deploy your application!

AWS Amplify Console deployment steps.

This will generate a public url that uses HTTPS, which is a requirement for PWAs. Congratulations! You can now test it in your browser, Desktop and Mobile.

Installing the Chatty app in the Desktop and Mobile

Due to the inclusion of the Web app manifest in our index.html page, you can now install the Chatty app in the Desktop and Mobile devices in Windows, Mac and Linux. The way it works, changes slightly depending the device and operating system you are using. Try it out following the steps in Add to Home screen.

Use add to Home screen to install in Desktop and Mobile.

Cleaning up cloud services

If at any time, you would like to delete a service from your project and your AWS Account, you can do this by running:

amplify delete

Conclusion

Congratulations! You successfully built your first offline-first app using Vue and Amplify. You created a chatroom PWA that stores data on the device while offline and when online synchronises in real-time with other devices using Amplify DataStore, AWS AppSync, GraphQL and Amazon DynamoDB. Thanks for following this tutorial.

If you prefer, you can follow the instructions in this video to build the chatroom app.

https://medium.com/media/23b1a303a95bff9ab82a2d445a6198ef/href

Thanks for reading!

Have you got any questions about this tutorial or AWS Amplify? Feel free to reach me anytime at @gerardsans.

My Name is Gerard Sans. I am a Developer Advocate at AWS Mobile working with AWS Amplify and AWS AppSync teams.

GraphQL is an open source data query and manipulation language for APIs, and a runtime for fulfilling queries with existing data.

Offline-first made easy with GraphQL, Amplify DataStore and Vue was originally published in Vue.js Developers on Medium, where people are continuing the conversation by highlighting and responding to this story.

Offline-first made easy with GraphQL, Amplify DataStore and Vue

Gerard Sans — Thu, 01 Oct 2020 10:30:10 GMT

Create a PWA cloud-enabled offline-first chatroom using Vue

Introduction to Offline-first, PWAs and Amplify DataStore
Setting up a new project with the Vue CLI
Creating a new GraphQL API
Amplify DataStore: setup, data models and usage
Creating the UI with Vue: the chatroom, sending and deleting messages; and doing real-time with subscriptions
Making Chatty a PWA
Adding a PWA custom configuration
Improving UX while offline
Publishing your app via the AWS Amplify Console
Installing Chatty in the Desktop and Mobile
Cleaning up cloud services

In order to follow this post you will need a basic knowledge of GraphQL. You can learn the basics following this tutorial at graphql.org. We will be referring to GraphQL schema, directives, types, queries, mutations, subscriptions and resolvers.

Please let me know if you have any questions or want to learn more at @gerardsans.

> Final solution and step by step tutorial in GitHub.

Introduction to Offline-first, PWAs and Amplify DataStore

Offline ready: application works offline. This usually comes with some limitations regarding feature set and device capabilities. Eg: PWAs require an install step where the user needs to be online to download all the necessary application assets.
Great user experience: application loads fast and transitions seamlessly from online to offline improving user retention. The user is in control regarding app updates, data synchronisation, data conflict resolution and connectivity. Eg: App shell; user is informed about connectivity changes.
Native-like features: application behaves similar to a native app. Eg: PWAs are able to run on multiple operating systems as standalone applications and don’t require the user to interact with the browser.
Reliable storage: user can exit the application at any moment without losing data. Application stores user data securely on-device and synchronises with the cloud transparently when possible.

Progressive Web Apps

Works offline via service worker.
Great performance via app shell and pre-cached assets.
Access to device APIs via Web APIs.
Supports push notifications via service worker.
Works in Web, Mobile and Desktop via Web app manifest.

The most important requirements for PWA are: security, must be served via HTTPS, the service worker and the Web app manifest. We will incorporate all of these to Chatty.

Amplify DataStore

GraphQL knowledge is beneficial when using Amplify DataStore to understand its inner workings but not necessary to use DataStore APIs.

Amplify DataStore behaves differently when in offline or online modes:

In offline mode, we use the DataStore API to manage the data on the device via the Storage Engine using GraphQL. The local Data Store being used will vary depending on the user device. This can either be IndexedDB in the browser or SQL Lite in Android and iOS.

Amplify DataStore data flow in offline mode

In online mode, Amplify DataStore, will synchronise with the cloud using the Sync Engine via GraphQL to automatically synchronise with your GraphQL API. At the query level, this is implemented as an AWS AppSync resolver accessing Amazon DynamoDB, AWS Lambda or Amazon ElasticSearch.

Amplify DataStore data flow in online mode

Amplify DataStore supports the following conflict resolution modes during synchronisation:

Optimistic concurrency, this mode rejects any incoming changes if there’s a mismatch in the tracking version between the incoming change and the item being changed. A further step on the client is required to resolve the conflict.
Auto merge (default), this mode doesn’t require further steps to resolve conflicts. It accommodates mutations on a field-by-field basis as long as scalar types don’t clash. If they do, they are ignored but all other fields are merged, if possible.
Custom, this mode uses AWS Lambda to allow more control and complex use cases.

For more details read conflict detection and sync.

Setting up a new project with the Vue CLI

Creating a new GraphQL API

First, we are going to create the GraphQL API to service Chatty chatroom. To create it, we will use the following command:

amplify add api

Answer the following questions

Please select from one of the below mentioned services GraphQL
Provide API name: ChattyAPI
Choose the default authorization type for the API API key
Enter a description for the API key: (empty)
After how many days from now the API key should expire (1–365): 7

It’s important that you don’t miss the following steps to set up conflict detection. If you missed it, you can recover with amplify api update

Do you want to configure advanced settings for the GraphQL API Yes, I want to make some additional changes.
Configure additional auth types? No
Configure conflict detection? Yes
Select the default resolution strategy Auto Merge
Do you want to override default per model settings? No
Do you have an annotated GraphQL schema? No
Choose a schema template: Single object with fields (e.g., “Todo” with ID, name, description)
Do you want to edit the schema now? Yes

When prompted, replace the default schema with the following:

type Chatty @model {
  id: ID!
  user: String!
  message: String!
  createdAt: AWSDateTime
}

Pushing your GraphQL API to the cloud

Let’s run the push command to create the GraphQL API:

amplify push

Are you sure you want to continue? Yes
Do you want to generate code for your newly created GraphQL API Yes
Choose the code generation language target javascript
Enter the file name pattern of graphql queries, mutations and subscriptions src/graphql/**/*.js
Do you want to generate/update all possible GraphQL operations — queries, mutations and subscriptions Yes
Enter maximum statement depth [increase from default if your schema is deeply nested] 2

Write down your GraphQL endpoint and API KEY.

This is the resulting architecture with our new GraphQL API managed by AWS AppSync and Amazon DynamoDB:

Run the command below to access the AWS AppSync console.

amplify console api

Please select from one of the below mentioned services GraphQL

Amplify DataStore setup

Run the command below to install the necessary dependencies:

npm install --save @aws-amplify/core @aws-amplify/datastore

Data model generation

Generate the data models to manage our messages for our ChattyAPI.

amplify codegen models

Important: DO NOT forget to regenerate your models every time you introduce a change in your schema.

After running this command, AWS Amplify CLI has generated the necessary data models in a new folder. The files in this folder hold your data model classes and schema.


    |_ src
      |_ models

Creating a message

Now that the GraphQL API and data models are created, we can begin using the DataStore API. The first thing we’ll do is create a new message passing the generated data model to save.

Using data models allow us to be always compliant with the GraphQL schema and prevent errors right in the client.

import { DataStore } from "@aws-amplify/datastore";
import { Chatty } from "./models";

await DataStore.save(new Chatty({
  user: "amplify-user",
  message: "Hi everyone!",
  createdAt: new Date().toISOString()
}))

This will create a record locally in your device and synchronise it in the background using the underlying GraphQL API when in online mode.

Querying data

Let’s see how we can query the data using Amplify DataStore. In order to query our data model we will use a query and a predicate to indicate that we want all records.

import { DataStore, Predicates } from "@aws-amplify/datastore";
import { Chatty } from "./models";

const messages = await DataStore.query(Chatty, Predicates.ALL);

This will return an array of messages that we can display in our UI. Predicates also support filters for common types like Strings, Numbers and Lists.

Find all supported filters at query with Predicates.

Creating the UI with Vue

Now, let’s look at how we can create the UI to create and display messages for Chatty chatroom.

Creating a message

Now, let’s look at how we create new messages.

Deleting all messages

DataStore.delete(Chatty, Predicates.ALL).then(() => {
  console.log('messages deleted!');
});

When using Amplify DataStore records are never removed, but marked for removal. Records will then be automatically removed as defined in their TTL attribute. At the time of writing, this is set to 30 days.

Real-time with GraphQL subscriptions

As we are using Amplify DataStore, as our source of truth, we don’t need another copy of it at the app level so we just need to load the latest state with loadMessages.

At this point, the chatroom is fully working including authentication flow and data synchronisation in real-time.

Real-time synchronisation using Chrome and Firefox clients side-by-side.

On the other hand, we haven’t made much progress in making it work offline or having native-like features. In order to tick these boxes, we are going to make our app a PWA.

Making Chatty a PWA

The PWA CLI plugin can help us making Chatty a PWA by running a single command:

vue add @vue/pwa

Service worker serving assets from the cache while offline

Learn more about service workers at using service workers.

Run the build command to see changes:

yarn build

In your dist folder you should find the following:

└── dist
  ├── css
  │   └── app..css
  ├── img/icons
  │   ├── android-chrome-.png
  │   └── ...
  ├── js
  │   ├── app..png
  │   └── ...
  ├── favicon.ico
  ├── index.html
  ├── manifest.json
  ├── precache-manifest..json
  ├── robots.txt
  └── service-worker.js

Test the app running the following commands on a new terminal window:

cd dist
python -m SimpleHTTPServer 8887  // open localhost:8887

Remember that in order to test the app you need to install the service worker first. This requires at least loading the app once.

To test the app while offline you can use the Developer Tools or switch off your internet connection. See below a screen capture after refreshing the page while offline:

Chatty app while offline.

Adding a PWA custom configuration

Let’s fix the issue with our logo by using a custom configuration. Create vue.config.js with the following content at the root of your project:

// vue.config.js
const manifest = require('./public/manifest.json')

module.exports = {
  pwa: {
    name: manifest.short_name,
    themeColor: manifest.theme_color,
    msTileColor: manifest.background_color,
    appleMobileWebAppCapable: 'yes',
    appleMobileWebAppStatusBarStyle: 'black',
    workboxPluginMode: 'InjectManifest',
    workboxOptions: {
      swSrc: 'src/service-worker.js',
    }
  }
}

This file controls the PWA CLI plugin and deal with settings in the Web app manifest and UX for different devices.

// src/service-worker.js
workbox.core.setCacheNameDetails({ prefix: 'amplify-datastore' })

workbox.core.skipWaiting()
workbox.core.clientsClaim()

const cacheFiles = [{
 "revision": "e653ab4d124bf16b5232",
 "url": "https://aws-amplify.github.io/img/amplify.svg"
}]

self.__precacheManifest = cacheFiles.concat(self.__precacheManifest || [])
workbox.precaching.precacheAndRoute(self.__precacheManifest, {})

Improving UX while offline

We are almost done. Our app can survive offline reloads, can store messages while offline and can synchronise when back online thanks to Amplify DataStore.

// You are offline.

//

// App.vue
import { Hub } from 'aws-amplify';

export default {
  data() {
    return { offline: undefined };
  },
  created() {
    this.listener = Hub.listen('datastore', {payload:{event}} => {
      if (event === 'networkStatus') {
        this.offline = !data.active;
      }
    })
  }
}

Chatty PWA informing the user of changes in the network status.

Publishing your app via the AWS Amplify Console

The first thing you need to do is create a new repo for this project. Once you’ve created the repo, copy the URL for the project to the clipboard and initialise git in your local project:

git init
git remote add origin repo@repoofyourchoice.com:username/project-name.git
git add .git commit -m 'initial commit'git push origin master

AWS Amplify Console deployment steps.

This will generate a public url that uses HTTPS, which is a requirement for PWAs. Congratulations! You can now test it in your browser, Desktop and Mobile.

Installing the Chatty app in the Desktop and Mobile

Use add to Home screen to install in Desktop and Mobile.

Cleaning up cloud services

If at any time, you would like to delete a service from your project and your AWS Account, you can do this by running:

amplify delete

Conclusion

If you prefer, you can follow the instructions in this video to build the chatroom app.

https://medium.com/media/23b1a303a95bff9ab82a2d445a6198ef/href

Thanks for reading!

Have you got any questions about this tutorial or AWS Amplify? Feel free to reach me anytime at @gerardsans.

My Name is Gerard Sans. I am a Developer Advocate at AWS Mobile working with AWS Amplify and AWS AppSync teams.

GraphQL is an open source data query and manipulation language for APIs, and a runtime for fulfilling queries with existing data.

Create a REST API integrated with Amazon DynamoDB using AWS Amplify and Vue

Gerard Sans — Fri, 25 Sep 2020 14:45:58 GMT

Refresh your REST APIs skills using AWS Amplify and Vue

In this article you will create a REST API integrated with Amazon DynamoDB using AWS Amplify including CRUD operations and publication. Access to the REST API will allow both registered users and guests. In order to test it you will create a client including an authentication flow using Vue.

Setting up a new project with the Vue CLI
Creating a REST API with Amplify CLI
Creating a new REST API
Implementing CRUD operations using Amazon DynamoDB
Pushing your REST API to the cloud
Publishing your app via the AWS Amplify Console
Cleaning up cloud services

Please let me know if you have any questions or want to learn more at @gerardsans.

> Final solution in GitHub.

Setting up a new project with the Vue CLI

Creating a REST API with Amplify CLI

The Amplify CLI provides a guided workflow to easily add, develop, test and manage REST APIs to access your AWS resources from your web and mobile applications.

A REST API or HTTP endpoint will be composed by one or more paths. Eg: /todos. Each path will use a Lambda function to handle HTTP requests and responses. Amplify CLI creates a single resource in Amazon API Gateway so you can handle all routes, HTTP Methods and paths, with a single Lambda function via a Lambda Proxy integration. HTTP proxy integrations forward all requests and responses directly through to your HTTP endpoint. Eg: /todos.

Architecture diagram for the Todo App

Creating a new REST API

In this post, you are going to create a REST API to service a todo app with a single endpoint /todos using Amazon DynamoDB as a data source. To create it, use the following command:

amplify add api

Answer the following questions

Please select from one of the below mentioned services: REST
Provide a friendly name for your resource to be used as a label for this category in the project: todosApi
Provide a path (e.g., /book/{isbn}): /todos

This will be the configuration for /todos path in Amazon API Gateway:

/                        
 |_ /todos        Main resource. Eg: /todos
    ANY           Methods: DELETE,GET,HEAD,OPTIONS,PATCH,POST,PUT
    OPTIONS       Allow pre-flight requests in CORS by browser
    |_ /{proxy+}  Eg: /todos/, /todos/id
       ANY        Methods: DELETE,GET,HEAD,OPTIONS,PATCH,POST,PUT
       OPTIONS    Allow pre-flight requests in CORS by browser

By default, Amplify CLI creates a greedy path variable /todos/{proxy+} that catches all child resources for a path and forwards them to your Lambda. This will match all child routes including /todos/id.

Choose a Lambda source Create a new Lambda function
Provide a friendly name for your resource to be used as a label for this category in the project: todosLambda
Provide the AWS Lambda function name: todosLambda
Choose the runtime that you want to use: NodeJS
Choose the function template that you want to use: CRUD function for Amazon DynamoDB

The Lambda function template, CRUD function for Amazon DynamoDB implements route handlers for GET, POST, PUT and DELETE Http Methods and paths for /todos and /todos/*. Some possible routes examples include:

GET /todos         List all todos
GET /todos/1       Load a todo by id
POST /todos        Create a todo
PUT /todos         Update a todo
DELETE /todos/1    Delete a todo by id

Do you want to access other resources in this project from your Lambda function? No
Do you want to invoke this function on a recurring schedule? No
Do you want to configure Lambda layers for this function? No
Do you want to edit the local lambda function now? Yes

We are going to change this template later but it’s good that you have it open as you follow the next steps.

Press enter to continue
Restrict API access Yes
Who should have access? Authenticated and Guest users
What kind of access do you want for Authenticated users? create, read, update, delete
What kind of access do you want for Guest users? read

Amplify CLI restricts API access combining Amazon Cognito for authentication and AWS IAM (Identity and Access Management) for granting execution permissions on routes.

Do you want to add another path? No

That’s all! Before we publish to the cloud though, let’s see how to change the default template to implement create, read, update and delete operations for your todo app.

Implementing CRUD operations with Amazon DynamoDB

In order to manage your todo app you want to implement all CRUD operations, these are: create, read, update and delete. The template we picked in the last section uses AWS Serverless Express. Amazon API Gateway will proxy incoming requests to your todosLambda.

For the implementation, you need to add route handlers to match all HTTP methods and paths you want to support. The first to match with an incoming request will be then executed. This will be in the same order as you have define them. If there’s no match an error will be returned.

See below, how different HTTP methods and operations match route definitions in AWS Serverless Express and DynamoDB Document Client.

REST API mapping between HTTP requests, AWS Serverless Express and Document Client calls.

Before we start, you need to install a library to generate the ids for new todos. To install this dependency run the following commands from the root of your project:

cd /amplify/backend/function/todosLambda/src
npm i --save uuid

Open /amplify/backend/function/todosLambda/src/app.js and replace its content for:

const AWS = require('aws-sdk')

var awsServerlessExpressMiddleware = require('aws-serverless-express/middleware')
var bodyParser = require('body-parser')
var express = require('express')
const { v4: uuidv4 } = require('uuid')

AWS.config.update({ region: process.env.TABLE_REGION });
const dynamodb = new AWS.DynamoDB.DocumentClient();

let tableName = "todosTable";
if (process.env.ENV && process.env.ENV !== "NONE") {
  tableName = tableName + '-' + process.env.ENV;
}

var app = express()
app.use(bodyParser.json())
app.use(awsServerlessExpressMiddleware.eventContext())

app.use(function (request, response, next) {
  response.header("Access-Control-Allow-Origin", "*")
  response.header("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept")
  next()
});

This is the same code from the default template adding the uuid library. This takes care of initialising DynamoDB Document Client API to interact with DynamoDB, the DynamoDB table name taking into account the current environment and enabling CORS headers.

Let’s handle the first route, we will use the HTTP Method and path by the title for reference as in the next section. For each operation, first we will include the code from the client; and then, followed by the code serving it in the todosLambda.

Fetching todos (GET, /todos)

In the client, we use the Amplify JavaScript library aws-amplify to consume the REST API. In order to fetch the todos useAPI.get with the todosApi followed by the path /todos and an empty payload. The result is an array with the todos as part of the body. Remember to use JSON.parse as the result.body needs to be in plain text due to HTTP transport. The result object also contains the statusCode and path as url.

// client request: fetching todos
import { API } from 'aws-amplify';

API.get('todosApi', '/todos', {}).then(result => {
 this.todos = JSON.parse(result.body);
}).catch(err => {
 console.log(err);
})

Let’s look at the code in todosLambda. To fetch todos, map a handler function to the GET Method and the /todos path. Then run a dynamodb.scan operation. This fetches all records in the table specified. Limit the results to 100 items. Find more details in the Developer Guide from Amazon DynamoDB.

Scan returns all the data in a table so you may consider to change it for a query if you expect more than just few records.

// todosLambda route handler: fetching todos

app.get("/todos", function (request, response) {
  let params = {
    TableName: tableName,
    limit: 100
  }
  dynamodb.scan(params, (error, result) => {
    if (error) {
      response.json({ statusCode: 500, error: error.message });
    } else {
      response.json({ statusCode: 200, url: request.url, body: JSON.stringify(result.Items) })
    }
  });
});

This returns the todos array as part of the body. Don’t forget to use JSON.stringifyas it needs to be in plain text due to HTTP transport.

Once the operation finishes, check for errors and create a JSON response accordingly. You will use the same approach in the rest of operations. Just note the highlighted parts on the code for important changes.

Fetching a todo by id (GET, /todos/:id)

Let’s quickly see the code necessary on the client below.

// client request: fetching a todo by id

import { API } from 'aws-amplify';

API.get('todosApi', `/todos/${id}`, {}).then((result) => {
  this.todo = JSON.parse(result.body);
}).catch(err => {
  console.log(err);
})

Notice how to use API.get with a path using a template string generating the path. Eg: `/todos/${id}` becomes "/todos/c71f0be2–607d-48bc-b721–4cb77c90a58f". The result receives the todo details.

Let’s see how to get the details from a single todo in todosLambda. In order to capture the todo id use route parameters, a feature from AWS Serverless Express. Route parameters use a semicolon to capture values at specific position in the URL like in /users/:userId/books/:bookId. Values are then available in the request object as request.params.userId and request.params.bookId.

// todosLambda route handler: fetching a todo by id

app.get("/todos/:id", function (request, response) {
  let params = {
    TableName: tableName,
    Key: {
      id: request.params.id
    }
  }
  dynamodb.get(params, (error, result) => {
    if (error) {
      response.json({ statusCode: 500, error: error.message });
    } else {
      response.json({ statusCode: 200, url: request.url, body: JSON.stringify(result.Item) })
    }
  });
});

As in the code above, use dynamodb.get to set your table and partition key Key.id from the request parameters.

For todosApi we only have a partition key, if you have a composed key (partition key + sort key) include the sort key too as part of the Key.sk.

Creating a new todo (POST, /todos)

In order to create a new todo, use API.post and set the payload to include the new todo description text within the request body.

// client request: creating a new todo

import { API } from 'aws-amplify';

API.post('todosApi', '/todos', {
  body: {
    text: "todo-1"
  }
}).then(result => {
  this.todo = JSON.parse(result.body);
}).catch(err => {
  console.log(err);
})

In todosLambda, creating a new todo is a bit more elaborated as we need to provide some new attributes and values. As we saw in the code above, the request.bodycontains the description for the new todo: text. Generate a new id, set complete to false, addcreatedAtand updatedAt timestamps and include the userId. To create the todo use dynamodb.put.

// todosLambda route handler: creating a new todo

app.post("/todos", function (request, response) {
  const timestamp = new Date().toISOString();
  let params = {
    TableName: tableName,
    Item: {
      ...request.body,
      id: uuidv4(),               // auto-generate id
      complete: false,            // default for new todos
      createdAt: timestamp,
      updatedAt: timestamp,
      userId: getUserId(request)  // userId from request
    }
  }

  dynamodb.put(params, (error, result) => {
    if (error) {
      response.json({ statusCode: 500, error: error.message, url: request.url });
    } else {
      response.json({ statusCode: 200, url: request.url, body: JSON.stringify(params.Item) })
    }
  });
});

The helper function getUserId below extracts the user id from Amazon Cognito.

// todosLambda route handler: helper function

const getUserId = (request) => {
  try {
    const reqContext = request.apiGateway.event.requestContext;
    const authProvider = reqContext.identity.cognitoAuthenticationProvider;
    return authProvider ? authProvider.split(":CognitoSignIn:").pop() : "UNAUTH";
  } catch (error) {
    return "UNAUTH";
  }
}

This code tries to capture the user id from the request identity context and if unsuccessful, returns an unauthorised token with UNAUTH as the value.

Updating a todo (PUT, /todos)

At this point you should know how to use the Amplify API. In order to update a todo use API.put and the /todos path. As you did creating a new todo provide the todo changes including its id as part of the body. As in the code below, change both the description and the complete values. The result contains any fields that were changed and their new values.

// client request: updating a todo

import { API } from 'aws-amplify';

API.put('todosApi', `/todos`, { 
  body: {
    id: id,
    text: "todo-2",
    complete: true
  }
}).then(result => {
  this.todo = JSON.parse(result.body);
}).catch(err => {
  console.log(err);
})

The update in todosLambda uses most of the code you are already familiar but including a new update expression. This expression includes optional attributes so we need to create it dynamically. Eg: to service a request changing only the description text but not the other attributes. We used UPDATED_NEW to return only the updated attributes in DynamoDB but if you need all attributes use ALL_NEW.

// todosLambda route handler: updating a todo

app.put("/todos", function (request, response) {
  const timestamp = new Date().toISOString();
  const params = {
    TableName: tableName,
    Key: {
      id: request.body.id,
    },
    ExpressionAttributeNames: { '#text': 'text' },
    ExpressionAttributeValues: {},
    ReturnValues: 'UPDATED_NEW',
  };
  params.UpdateExpression = 'SET ';
  if (request.body.text) {
    params.ExpressionAttributeValues[':text'] = request.body.text;
    params.UpdateExpression += '#text = :text, ';
  }
  if (request.body.complete) {
    params.ExpressionAttributeValues[':complete'] = request.body.complete;
    params.UpdateExpression += 'complete = :complete, ';
  }
  if (request.body.text || request.body.complete) {
    params.ExpressionAttributeValues[':updatedAt'] = timestamp;
    params.UpdateExpression += 'updatedAt = :updatedAt';
  }
  dynamodb.update(params, (error, result) => {
    if (error) {
      response.json({ statusCode: 500, error: error.message, url: request.url });
    } else {
      response.json({ statusCode: 200, url: request.url, body: JSON.stringify(result.Attributes) })
    }
  });
});

To keep the semantics of HTTP PUT use the todo id from request.body.id instead of the request.params.id.

Deleting a todo by id (DELETE, /todos/:id)

Let’s implement the request to delete a todo. On the client, use API.del with a path including the id as a route parameter. If successful, the result body will be {} with statusCode:200.

// client request: deleting a todo by id

import { API } from 'aws-amplify';

API.del('todosApi', `/todos/${id}`, {}).then(result => {
  console.log(result);
}).catch(err => {
  console.log(err);
})

In todosLambda, use the same techniques from the previous route handlers now using dynamodb.delete operation.

// todosLambda route handler: deleting a todo by id

app.delete("/todos/:id", function (request, response) {
  let params = {
    TableName: tableName,
    Key: {
      id: request.params.id
    }
  }
  dynamodb.delete(params, (error, result) => {
    if (error) {
      response.json({ statusCode: 500, error: error.message, url: request.url });
    } else {
      response.json({ statusCode: 200, url: request.url, body: JSON.stringify(result) })
    }
  });
});

This completes all the CRUD operations. Before we can test it we need to publish it.

Pushing your new REST API to the cloud

Let’s deploy the new REST API in the cloud by running this command in the root of your project:

amplify push

At the end of this command you can take note of your new REST API url.

REST APIs follow this pattern https://{restapi-id}.execute-api.{region}.amazonaws.com/{environment}/{path}.

Let’s see an overview of all the resources created by Amplify CLI.

REST
 |_ /todos (path)
    |_ todosApi (Amazon API Gateway) 
       |_ todosLambda (AWS Lambda)
          |_ Logs (Amazon CloudWatch)

We have covered all AWS Services but Amazon CloudWatch. This service will allow you to monitor usage and access to logs during development and testing.

Publishing your app via the AWS Amplify Console

The first thing you need to do is create a new repo for this project. Once you’ve created the repo, copy the URL for the project to the clipboard and initialise git in your local project:

git init
git remote add origin repo@repoofyourchoice.com:username/project-name.git
git add .git commit -m 'initial commit'git push origin master

AWS Amplify Console deployment steps.

Cleaning up cloud services

If at any time, you would like to delete the services from your project and your AWS Account, you can do this by running:

amplify delete

Conclusion

Congratulations! You successfully created a REST API using AWS Amplify, implemented all CRUD operations using Amazon DynamoDB and created a client to consume it restricting access to both registered users and guests using Vue. Thanks for following this tutorial.

If you prefer, you can also follow this video to achieve the same result.

https://medium.com/media/729f2b6d5dfa3026b6759d2731bf11ff/href

Thanks for reading!

Have you got any questions about this tutorial or AWS Amplify? Feel free to reach me anytime at @gerardsans.

My Name is Gerard Sans. I am a Developer Advocate at AWS Mobile working with AWS Amplify and AWS AppSync teams.

Create a cloud-enabled GraphQL API with AWS Amplify and Vue

Gerard Sans — Tue, 22 Sep 2020 15:50:35 GMT

Full-stack serverless made easy using AWS Amplify and Vue

In this tutorial, we will learn how to build a GraphQL data-driven serverless app using Vue, AWS Amplify and AWS AppSync. We will create a new project with Vue, generate a serverless GraphQL API using AWS AppSync and build a client to consume it including real-time. We will cover:

Introduction to AWS AppSync
Setting up a new project with the Vue CLI
Creating a new GraphQL API and using your first GraphQL transform
Pushing your GraphQL API to the cloud
Building the client: adding queries, mutations and real-time
Publishing your app via the AWS Amplify Console
Cleaning up cloud services

In order to follow this post you will need a basic knowledge of GraphQL. You can learn the basics following this tutorial at graphql.org.

> Final solution and step by step tutorial in GitHub.

Please let me know if you have any questions or want to learn more at @gerardsans.

Introduction to AWS AppSync

AWS AppSync allows you to create highly scalable serverless GraphQL data-driven mobile and web apps by providing an open source Amplify Framework (consisting of AWS AppSync Clients, GraphQL transforms and the CLI), integrations with multiple data sources, and the AWS AppSync console.

GraphQL client: client-side library to securely access data from your GraphQL API. Built-in support for real-time and offline/delta client synchronisation.
Amplify CLI: set of commands, via theapi category, to automate the setup and provision of resources for AWS AppSync and Amazon API Gateway cloud services. Support for REST and GraphQL APIs.
GraphQL Transforms: custom GraphQL schema directives that can be used in your GraphQL schema to enable custom workflows.
Data sources: databases (Amazon DynamoDB for NoSQL, Amazon Aurora for RDBMS), searches (Amazon Elasticsearch Service), and AWS Lambda functions.

By using AWS AppSync, teams can quickly create highly scalable serverless GraphQL data-driven apps for mobile and web while leaving the heavy lifting to the Amplify CLI.

AWS AppSync data-driven apps architecture overview

Setting up a new project with the Vue CLI

Before moving to the next section, please complete the steps described in “Build your first full-stack serverless app with Vue”.

Creating a new GraphQL API

For this post we are going to create a GraphQL API to list our favourite restaurants. To create it, we will use the following command:

amplify add api

Answer the following questions

Please select from one of the below mentioned services GraphQL
Provide API name: RestaurantAPI
Choose the default authorization type for the API API key
Enter a description for the API key: (empty)
After how many days from now the API key should expire (1–365): 7
Do you want to configure advanced settings for the GraphQL API No, I am done.
Do you have an annotated GraphQL schema? No
Choose a schema template: Single object with fields (e.g., “Todo” with ID, name, description)
Do you want to edit the schema now? Yes

When prompted, replace the default schema with the following:

type Restaurant @model {
  id: ID!
  name: String!
  description: String!
  city: String!
}

You should be familiar with everything in the GraphQL schema above but for the @model directive. This is a GraphQL transform provided by AppSync.

Using your first GraphQL transform

GraphQL transforms allow AppSync to provide further customisation and support for common scenarios, so you don’t have to.

By adding@model to theRestauranttype we are telling the Amplify CLI runtime to create the resolvers to support queries, mutations and subscriptions in DynamoDB. Besides the regular CRUD operations, we are also getting some more advanced features like pagination and filtering that we are going to cover later.

Once we apply@model to a type we get access to these other transforms:

@key to configure custom index structures in DynamoDB
@searchable to enable searches using Amazon Elasticsearch Service
@connection to add relationships between types
@lambda to generate AWS Lambda resolvers
@auth to add fine grained multi-authorisation support
@versioned to add conflict resolution for offline scenarios

Learn how to use each transform in detail from the official docs.

GraphQL Transforms are implemented using custom GraphQL schema directives as defined in the GraphQL specification.

Pushing your GraphQL API to the cloud

Let’s run the push command to create the GraphQL API and see the results of using the @model transform:

amplify push

Are you sure you want to continue? Yes
Do you want to generate code for your newly created GraphQL API Yes
Choose the code generation language target javascript
Enter the file name pattern of graphql queries, mutations and subscriptions src/graphql/**/*.js
Do you want to generate/update all possible GraphQL operations — queries, mutations and subscriptions Yes
Enter maximum statement depth [increase from default if your schema is deeply nested] 2

Write down your GraphQL endpoint and API KEY.

Run the command below to access the AWS AppSync console.

amplify console api

Please select from one of the below mentioned services GraphQL

Testing your new GraphQL API

Once in the AWS AppSync console, click on Queries on the left side. This will open an editor that we can use to edit and test GraphQL queries.

Execute the following mutation to create a new restaurant:

mutation createRestaurant {
  createRestaurant(input: {
    name: "Nobu"
    description: "Great Sushi"
    city: "New York"
  }) {
    id name description city
  }
}

Now, run this query to list the restaurant we just added:

query listRestaurants {
  listRestaurants {
    items {
      id
      name
      description
      city
    }
  }
}

Besides creating all common CRUD operations for your types, AWS AppSync also creates extra utility features like filters. Run this query to try using filters:

query searchRestaurants {
  listRestaurants(filter: {
    city: {
      contains: "New York"
    }
  }) {
    items {
      id
      name
      description
      city
    }
  }
}

AWS AppSync creates filters on GraphQL scalar types (ID, String, Int, Float, Boolean) for list operations like listRestaurants.

Querying data with queries

In the earlier sections, we made sure we had a working GraphQL API. On the client, we will start by displaying a list.

From the code above, we are using API.graphqlto run the listRestaurantsquery within getData. Queries were created by the Amplify CLI during the push command. This call returns a promise that we handle by using async/await and render the result using v-for.

You might be wondering why we used listRestaurants.items. This is because AWS AppSync created an intermediate type ModelRestaurantConnection. This is so we can handle pagination scenarios together withlimit and nextTokenparameters.

listRestaurants(filter: ModelRestaurantFilterInput, limit: Int, nextToken: String): ModelRestaurantConnection

type ModelRestaurantConnection {
 items: [Restaurant]
 nextToken: String
}

Creating data with mutations

In order to add new restaurants, we are going to create a data entry using form to take the required user input and pass it forward to the createRestaurant mutation.

Note how AWS AppSync instead of passing each field as an argument created an input type CreateRestaurantInput which makes our client code simpler and convenient.

type Mutation {
 createRestaurant(input: CreateRestaurantInput!): Restaurant
}input CreateRestaurantInput {
 id: ID
 name: String!
 description: String!
 city: String!
}

As we did before with queries, we are using async/await to run our mutation as we submit the form via onCreate. Once we get the result, we use the form data to update the list and clear the form inputs.

Adding real-time with subscriptions

To demonstrate real-time, we are going to use a subscription to update AppSync clients when new restaurants are added. AppSync clients will subscribe to listen for changes going through our GraphQL API. More specifically, restaurant creation mutations.

import { onCreateRestaurant } from './graphql/subscriptions';

export default {
  name: 'app',
  created() {
    //Subscribe to changes
    API.graphql(graphqlOperation(onCreateRestaurant))
    .subscribe((sourceData) => {
      const newRestaurant = sourceData.value.data.onCreateRestaurant
      if (newRestaurant) {
        // skip our own mutations and duplicates
        if (this.restaurants.some(r => r.id == newRestaurant.id)) return;
        this.restaurants = [newRestaurant, ...this.restaurants];
      } 
    });
  },
}

We use createdto setup the subscription and immediately subscribe to start listening for notifications. Then for each notification, we will update our list of restaurants with changes.

Publishing your app via the AWS Amplify Console

The first thing you need to do is create a new repo for this project. Once you’ve created the repo, copy the URL for the project to the clipboard and initialise git in your local project:

git init
git remote add origin repo@repoofyourchoice.com:username/project-name.git
git add .git commit -m 'initial commit'git push origin master

AWS Amplify Console deployment steps.

Cleaning up cloud services

If at any time, you would like to delete a service from your project and your AWS Account, you can do this by running:

amplify delete

Conclusion

Congratulations! You successfully built your first GraphQL API using Vue and AWS AppSync. Thanks for following this tutorial.

If you prefer, you can follow the instructions in this video to build the same app but deploying to Amazon Cloudfront and Amazon S3.

https://medium.com/media/bcbb4bda9b42a357ecd616d7e74c06ea/href

Thanks for reading!

Have you got any questions about this tutorial or AWS Amplify? Feel free to reach me anytime at @gerardsans.

My Name is Gerard Sans. I am a Developer Advocate at AWS Mobile working with AWS Amplify and AWS AppSync teams.

GraphQL is an open source data query and manipulation language for APIs, and a runtime for fulfilling queries with existing data.

Build your first full-stack serverless app with Vue and AWS Amplify

Gerard Sans — Mon, 14 Sep 2020 12:04:38 GMT

Build flexible, scalable, and reliable apps with AWS Amplify

In this tutorial, you will learn how to build a full-stack serverless app using Vue and AWS Amplify. You will create a new project and add a full authorisation flow using the authenticator component. This includes:

Please let me know if you have any questions or want to learn more on the above at @gerardsans.

Introduction to AWS Amplify

Amplify makes developing, releasing and operating modern full-stack serverless apps easy and delightful. Mobile and frontend web developers are being supported throughout the app life cycle via an open source Amplify Framework (consisting of the Amplify libraries and Amplify CLI) and seamless integrations with AWS cloud services, and the AWS Amplify Console.

Amplify libraries: in this article we will be using aws-amplify and @aws-amplify/ui-vue.
Amplify CLI: command line tool for configuring and integrating cloud services.
UI components: authenticator, photo picker, photo album and chat bot.
Cloud services: authentication, storage, analytics, notifications, AWS Lambda functions, REST and GraphQL APIs, predictions, chat bots and extended reality (AR/VR).
Offline-first support: Amplify DataStore provides a programming model for leveraging shared and distributed data without writing additional code for data reconciliation between offline and online scenarios.

By using AWS Amplify, teams can focus on development while the Amplify team enforces best patterns and practices throughout the AWS Amplify stack.

Amplify CLI

The Amplify CLI provides a set of commands to help with repetitive tasks and automating cloud service setup and provision.

Some commands will prompt questions and provide sensible defaults to assist you during its execution. These are some common tasks. Run:

amplify init, to setup a new environment. Eg: dev, test, dist.
amplify push, to provision local resources to the cloud.
amplify status, to list local resources and their current status.

The Amplify CLI uses AWS CloudFormation to manage service configuration and resource provisioning via templates. This a declarative and atomic approach to configuration. Once a template is executed, it will either fail or succeed.

Setting up a new project with the Vue CLI

To get started, create a new project using the Vue CLI. If you already have it, skip to the next step. If not, install it and create the app using:

yarn global add @vue/cli
vue create amplify-app

Navigate to the new directory and check everything checks out before continuing

cd amplify-app
yarn serve

Prerequisites

Before going forward make sure you have gone through the instructions in our docs to sign up to your AWS Account and install and configure the Amplify CLI.

Setting up your Amplify project

AWS Amplify allows you to create different environments to define your preferences and settings. For any new project you need to run the command below and answer as follows:

amplify init

Enter a name for the project: amplify-app
Enter a name for the environment: dev
Choose your default editor: Visual Studio Code
Please choose the type of app that you’re building javascript
What javascript framework are you using vue
Source Directory Path: src
Distribution Directory Path: dist
Build Command: npm run-script build
Start Command: npm run-script serve
Do you want to use an AWS profile? Yes
Please choose the profile you want to use default

At this point, the Amplify CLI has initialised a new project and a new folder: amplify. The files in this folder hold your project configuration.


    |_ amplify
      |_ .config
      |_ #current-cloud-backend
      |_ backend
      team-provider-info.json

Installing the AWS Amplify dependencies

Install the required dependencies for AWS Amplify and Vue using:

yarn add aws-amplify @aws-amplify/ui-vue

Adding authentication

AWS Amplify provides authentication via the auth category which gives us access to AWS Cognito. To add authentication use the following command:

amplify add auth

When prompted choose:

Do you want to use default authentication and security configuration?: Default configuration
How do you want users to be able to sign in when using your Cognito User Pool?: Username
Do you want to configure advanced settings? No

Pushing changes to the cloud

By running the push command, the cloud resources will be provisioned and created in your AWS account.

amplify push

To quickly check your newly created Cognito User Pool you can run

amplify status

To access the AWS Cognito Console at any time, go to the dashboard at https://console.aws.amazon.com/cognito. Also be sure that your region is set correctly.

Your resources have been created and you can start using them!

Configuring the Vue application

Reference the auto-generated aws-exports.js file that is now in your src folder. To configure the app, open main.ts and add the following code below the last import:

import Vue from 'vue'
import App from './App.vue'

import Amplify from 'aws-amplify';
import '@aws-amplify/ui-vue';
import aws_exports from './aws-exports';

Amplify.configure(aws_exports);

Vue.config.productionTip = false

new Vue({
  render: h => h(App),
}).$mount('#app')

Using the Authenticator Component

AWS Amplify provides UI components that you can use in your app. Let’s add these components to the project

In order to use the authenticator component add it to src/App.vue:

You can run the app and see that an authentication flow has been added in front of your app component. This flow gives users the ability to sign up and sign in.

To view any users that were created, go back to the Cognito Dashboard at https://console.aws.amazon.com/cognito. Also be sure that your region is set correctly.

Alternatively you can also use:

amplify console auth

Accessing User Data

To access the user’s info using the Auth API. This will return a promise.

import { Auth } from 'aws-amplify';

Auth.currentAuthenticatedUser().then(console.log)

Publishing your app

To deploy and host your app on AWS, we can use the hosting category.

amplify add hosting

Select the plugin module to execute: Amazon CloudFront and S3
Select the environment setup: DEV
hosting bucket name YOURBUCKETNAME
index doc for the website index.html
error doc for the website index.html

Now, everything is set up & we can publish it:

amplify publish

Cleaning up Services

To wipe out all resources from your project and your AWS Account, you can do this by running:

amplify delete

Conclusion

Congratulations! You successfully built your first full-stack serverless app using Vue and AWS Amplify. Thanks for following this tutorial.

You have learnt how to provide an authentication flow using the authenticator component or via the service API and how to use Amplify CLI to execute common tasks including adding and removing services.

Thanks for reading!

Have you got any questions regarding this tutorial or AWS Amplify? Feel free to reach out to me anytime at @gerardsans.

My Name is Gerard Sans. I am a Developer Advocate at AWS Mobile working with AWS Amplify and AWS AppSync teams.