Create speech-enabled apps with Microsoft Foundry

Introduction

Azure Speech provides APIs that you can use to build speech-enabled applications. This includes:

• Speech to text: An API that enables speech recognition in which your application can accept spoken input.
• Text to speech: An API that enables speech synthesis in which your application can provide spoken output.
• Speech Translation: An API that you can use to translate spoken input into multiple languages.
• Keyword Recognition: An API that enables your application to recognize keywords or short phrases.
• Intent Recognition: An API that uses conversational language understanding to determine the semantic meaning of spoken input.

This module focuses on speech recognition and speech synthesis, which are core capabilities of any speech-enabled application.

Note

The code examples in this module are provided in Python, but you can use any of the available Azure Speech SDK packages to develop speech-enabled applications in your preferred language. Available SDK packages include:

• azure-cognitiveservices-speech for Python
• Microsoft.CognitiveServices.Speech for Microsoft .NET
• microsoft-cognitiveservices-speech-sdk for JavaScript
• Microsoft Cognitive Services Speech SDK For Java

Provision an Azure resource for speech

Before you can use Azure Speech, you need to create an Azure Speech resource in your Azure subscription. You can use either a dedicated Azure Speech resource or a Microsoft Foundry resource.

After you create your resource, you’ll need the following information to use it from a client application through one of the supported SDKs:

• The location in which the resource is deployed (for example, eastus)
• One of the keys assigned to your resource.

You can view of these values on the Keys and Endpoint page for your resource in the Azure portal.

While the specific syntax and parameters can vary between language-specific SDKs, most interactions with the Azure Speech service start with the creation of a SpeechConfig object that encapsulates the connection to your Azure Speech resource.

For example, the following Python code instantiates a SpeechConfig object based on an Azure Speech resource in the East US region:

Python

import azure.cognitiveservices.speech as speech_sdk

speech_config = speech_sdk.SpeechConfig(your_project_key, ‘eastus’)

Note

This example assumes that the Speech SDK package for python has been installed, like this:

pip install azure-cognitiveservices-speech

Use the Azure Speech to Text API

The Azure Speech service supports speech recognition through the following features:

• Real-time transcription: Instant transcription with intermediate results for live audio inputs.
• Fast transcription: Fastest synchronous output for situations with predictable latency.
• Batch transcription: Efficient processing for large volumes of prerecorded audio.
• Custom speech: Models with enhanced accuracy for specific domains and conditions.

Using the Azure Speech SDK

While the specific details vary, depending on the SDK being used (Python, C#, and so on); there’s a consistent pattern for using the Speech to text API:

1. Use a SpeechConfig object to encapsulate the information required to connect to your Azure Speech resource. Specifically, its location and key.
2. Optionally, use an AudioConfig to define the input source for the audio to be transcribed. By default, this is the default system microphone, but you can also specify an audio file.
3. Use the SpeechConfig and AudioConfig to create a SpeechRecognizer object. This object is a proxy client for the Speech to text API.
4. Use the methods of the SpeechRecognizer object to call the underlying API functions. For example, the RecognizeOnceAsync() method uses the Azure Speech service to asynchronously transcribe a single spoken utterance.
5. Process the response from the Azure Speech service. In the case of the RecognizeOnceAsync() method, the result is a SpeechRecognitionResult object that includes the following properties:

• Duration
• OffsetInTicks
• Properties
• Reason
• ResultId
• Text

If the operation was successful, the Reason property has the enumerated value RecognizedSpeech, and the Text property contains the transcription. Other possible values for Result include NoMatch (indicating that the audio was successfully parsed but no speech was recognized) or Canceled, indicating that an error occurred (in which case, you can check the Properties collection for the CancellationReason property to determine what went wrong).

Configure audio format and voices

When synthesizing speech, you can use a SpeechConfig object to customize the audio that is returned by the Azure Speech service.

Audio format

The Azure Speech service supports multiple output formats for the audio stream that is generated by speech synthesis. Depending on your specific needs, you can choose a format based on the required:

• Audio file type
• Sample-rate
• Bit-depth

For example, the following Python code sets the speech output format for a previously defined SpeechConfig object named speech_config:

Python

speech_config.set_speech_synthesis_output_format(SpeechSynthesisOutputFormat.Riff24Khz16BitMonoPcm)

For a full list of supported formats and their enumeration values, see the Azure Speech SDK documentation.

Voices

The Azure Speech service provides multiple voices that you can use to personalize your speech-enabled applications. Voices are identified by names that indicate a locale and a person’s name — for example en-GB-George.

The following Python example code sets the voice to be used

Python

speech_config.speech_synthesis_voice_name = “en-GB-George”

For information about voices, see the Azure Speech SDK documentation.

Use Speech Synthesis Markup Language

While the Azure Speech SDK enables you to submit plain text to be synthesized into speech, the service also supports an XML-based syntax for describing characteristics of the speech you want to generate. This Speech Synthesis Markup Language (SSML) syntax offers greater control over how the spoken output sounds, enabling you to:

• Specify a speaking style, such as “excited” or “cheerful” when using a neural voice.
• Insert pauses or silence.
• Specify phonemes (phonetic pronunciations), for example to pronounce the text “SQL” as “sequel”.
• Adjust the prosody of the voice (affecting the pitch, timbre, and speaking rate).
• Use common “say-as” rules, for example to specify that a given string should be expressed as a date, time, telephone number, or other form.
• Insert recorded speech or audio, for example to include a standard recorded message or simulate background noise.

For example, consider the following SSML:

XML

<speak version=”1.0" xmlns=”http://www.w3.org/2001/10/synthesis"

xmlns:mstts=”https://www.w3.org/2001/mstts" xml:lang=”en-US”>

<voice name=”en-US-AriaNeural”>

<mstts:express-as style=”cheerful”>

I say tomato

</mstts:express-as>

</voice>

<voice name=”en-US-GuyNeural”>

I say <phoneme alphabet=”sapi” ph=”t ao m ae t ow”> tomato </phoneme>.

<break strength=”weak”/>Lets call the whole thing off!

</voice>

</speak>

This SSML specifies a spoken dialog between two different neural voices, like this:

• Ariana (cheerfully): “I say tomato:
• Guy: “I say tomato (pronounced tom-ah-toe) … Let’s call the whole thing off!”

To submit an SSML description to the Speech service, you can use an appropriate method of a SpeechSynthesizer object, like this:

Python

speech_synthesizer.speak_ssml(‘<speak>…’);

For more information about SSML, see the Azure Speech SDK documentation.

Exercise — Create a speech-enabled app

In this exercise, build a speech enabled app for both speech recognition and synthesis.

Note

To complete this lab, you need an Azure subscription.

Launch the exercise and follow the instructions.

Tip

After completing the exercise, if you’ve finished exploring Foundry Tools, delete the Azure resources that you created during the exercise.

Recognize and synthesize speech

Azure AI Speech is a service that provides speech-related functionality, including:

• A speech-to-text API that enables you to implement speech recognition (converting audible spoken words into text).
• A text-to-speech API that enables you to implement speech synthesis (converting text into audible speech).

In this exercise, you’ll use both of these APIs to implement a speaking clock application.

While this exercise is based on Python, you can develop speech applications using multiple language-specific SDKs; including:

• Azure AI Speech SDK for Python
• Azure AI Speech SDK for .NET
• Azure AI Speech SDK for JavaScript

This exercise takes approximately 30 minutes.

NOTE This exercise is designed to be completed in the Azure cloud shell, where direct access to your computer’s sound hardware is not supported. The lab will therefore use audio files for speech input and output streams. The code to achieve the same results using a mic and speaker is provided for your reference.

Create an Azure AI Speech resource

Let’s start by creating an Azure AI Speech resource.

1. Open the Azure portal at https://portal.azure.com, and sign in using the Microsoft account associated with your Azure subscription.
2. In the top search field, search for Speech service. Select it from the list, then select Create.
3. Provision the resource using the following settings:

• Subscription: Your Azure subscription.
• Resource group: Choose or create a resource group.
• Region:Choose any available region
• Name: Enter a unique name.
• Pricing tier: Select F0 (free), or S (standard) if F is not available.

1. Select Review + create, then select Create to provision the resource.
2. Wait for deployment to complete, and then go to the deployed resource.
3. View the Keys and Endpoint page in the Resource Management section. You will need the information on this page later in the exercise.

Prepare and configure the speaking clock app

1. Leaving the Keys and Endpoint page open, use the [>_] button to the right of the search bar at the top of the page to create a new Cloud Shell in the Azure portal, selecting a PowerShell environment. The cloud shell provides a command line interface in a pane at the bottom of the Azure portal.

Note: If you have previously created a cloud shell that uses a Bash environment, switch it to PowerShell.

1. In the cloud shell toolbar, in the Settings menu, select Go to Classic version (this is required to use the code editor).

Ensure you’ve switched to the classic version of the cloud shell before continuing.

1. In the PowerShell pane, enter the following commands to clone the GitHub repo for this exercise:

code

rm -r mslearn-ai-language -f

git clone https://github.com/microsoftlearning/mslearn-ai-language

Tip: As you enter commands into the cloudshell, the ouput may take up a large amount of the screen buffer. You can clear the screen by entering the cls command to make it easier to focus on each task.

1. After the repo has been cloned, navigate to the folder containing the speaking clock application code files:

code

cd mslearn-ai-language/Labfiles/07-speech/Python/speaking-clock

1. In the command line pane, run the following command to view the code files in the speaking-clock folder:

code

ls -a -l

The files include a configuration file (.env) and a code file (speaking-clock.py). The audio files your application will use are in the audio subfolder.

1. Create a Python virtual environment and install the Azure AI Speech SDK package and other required packages by running the following command:

code

python -m venv labenv

./labenv/bin/Activate.ps1

Get Aram Radif’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

pip install -r requirements.txt azure-cognitiveservices-speech==1.42.0

1. Enter the following command to edit the configuration file:

code

code .env

The file is opened in a code editor.

1. Update the configuration values to include the region and a key from the Azure AI Speech resource you created (available on the Keys and Endpoint page for your Azure AI Translator resource in the Azure portal).
2. After you’ve replaced the placeholders, use the CTRL+S command to save your changes and then use the CTRL+Q command to close the code editor while keeping the cloud shell command line open.

Add code to use the Azure AI Speech SDK

Tip: As you add code, be sure to maintain the correct indentation.

1. Enter the following command to edit the code file that has been provided:

code

code speaking-clock.py

1. At the top of the code file, under the existing namespace references, find the comment Import namespaces. Then, under this comment, add the following language-specific code to import the namespaces you will need to use the Azure AI Speech SDK:

code

# Import namespaces

from azure.core.credentials import AzureKeyCredential

import azure.cognitiveservices.speech as speech_sdk

1. In the main function, under the comment Get config settings, note that the code loads the key and region you defined in the configuration file.
2. Find the comment Configure speech service, and add the following code to use the AI Services key and your region to configure your connection to the Azure AI Services Speech endpoint:

code

# Configure speech service

speech_config = speech_sdk.SpeechConfig(speech_key, speech_region)

print(‘Ready to use speech service in:’, speech_config.region)

1. Save your changes (CTRL+S), but leave the code editor open.

Run the app

So far, the app doesn’t do anything other than connect to your Azure AI Speech service, but it’s useful to run it and check that it works before adding speech functionality.

1. In the command line, enter the following command to run the speaking clock app:

code

python speaking-clock.py

The code should display the region of the speech service resource the application will use. A successful run indicates that the app has connected to your Azure AI Speech resource.

Add code to recognize speech

Now that you have a SpeechConfig for the speech service in your project’s Azure AI Services resource, you can use the Speech-to-text API to recognize speech and transcribe it to text.

In this procedure, the speech input is captured from an audio file, which you can play here:

1. In the code file, note that the code uses the TranscribeCommand function to accept spoken input. Then in the TranscribeCommand function, find the comment Configure speech recognition and add the appropriate code below to create a SpeechRecognizer client that can be used to recognize and transcribe speech from an audio file:

code

# Configure speech recognition

current_dir = os.getcwd()

audioFile = current_dir + ‘/time.wav’

audio_config = speech_sdk.AudioConfig(filename=audioFile)

speech_recognizer = speech_sdk.SpeechRecognizer(speech_config, audio_config)

1. In the TranscribeCommand function, under the comment Process speech input, add the following code to listen for spoken input, being careful not to replace the code at the end of the function that returns the command:

code

# Process speech input

print(“Listening…”)

speech = speech_recognizer.recognize_once_async().get()

if speech.reason == speech_sdk.ResultReason.RecognizedSpeech:

command = speech.text

print(command)

else:

print(speech.reason)

if speech.reason == speech_sdk.ResultReason.Canceled:

cancellation = speech.cancellation_details

print(cancellation.reason)

print(cancellation.error_details)

1. Save your changes (CTRL+S), and then in the command line below the code editor, re-run the program:
2. Review the output, which should successfully “hear” the speech in the audio file and return an appropriate response (note that your Azure cloud shell may be running on a server that is in a different time-zone to yours!)

Tip: If the SpeechRecognizer encounters an error, it produces a result of “Cancelled”. The code in the application will then display the error message. The most likely cause is an incorrect region value in the configuration file.

Synthesize speech

Your speaking clock application accepts spoken input, but it doesn’t actually speak! Let’s fix that by adding code to synthesize speech.

Once again, due to the hardware limitations of the cloud shell we’ll direct the synthesized speech output to a file.

1. In the code file, note that the code uses the TellTime function to tell the user the current time.
2. In the TellTime function, under the comment Configure speech synthesis, add the following code to create a SpeechSynthesizer client that can be used to generate spoken output:

code

# Configure speech synthesis

output_file = “output.wav”

speech_config.speech_synthesis_voice_name = “en-GB-RyanNeural”

audio_config = speech_sdk.audio.AudioConfig(filename=output_file)

speech_synthesizer = speech_sdk.SpeechSynthesizer(speech_config, audio_config,)

1. In the TellTime function, under the comment Synthesize spoken output, add the following code to generate spoken output, being careful not to replace the code at the end of the function that prints the response:

code

# Synthesize spoken output

speak = speech_synthesizer.speak_text_async(response_text).get()

if speak.reason != speech_sdk.ResultReason.SynthesizingAudioCompleted:

print(speak.reason)

else:

print(“Spoken output saved in “ + output_file)

1. Save your changes (CTRL+S) and re-run the program, which should indicate that the spoken output was saved in a file.
2. If you have a media player capable of playing .wav audio files, download the file that was generated by entering the following command:

code

download ./output.wav

The download command creates a popup link at the bottom right of your browser, which you can select to download and open the file.

The file should sound similar to this:

Use Speech Synthesis Markup Language

Speech Synthesis Markup Language (SSML) enables you to customize the way your speech is synthesized using an XML-based format.

1. In the TellTime function, replace all of the current code under the comment Synthesize spoken output with the following code (leave the code under the comment Print the response):

code

# Synthesize spoken output

responseSsml = “ \

<speak version=’1.0' xmlns=’http://www.w3.org/2001/10/synthesis' xml:lang=’en-US’> \

<voice name=’en-GB-LibbyNeural’> \

{} \

<break strength=’weak’/> \

Time to end this lab! \

</voice> \

</speak>”.format(response_text)

speak = speech_synthesizer.speak_ssml_async(responseSsml).get()

if speak.reason != speech_sdk.ResultReason.SynthesizingAudioCompleted:

print(speak.reason)

else:

print(“Spoken output saved in “ + output_file)

1. Save your changes and re-run the program, which should once again indicate that the spoken output was saved in a file.
2. Download and play the generated file, which should sound similar to this:

Clean up

If you’ve finished exploring Azure AI Speech, you should delete the resources you have created in this exercise to avoid incurring unnecessary Azure costs.

1. Close the Azure cloud shell pane
2. In the Azure portal, browse to the Azure AI Speech resource you created in this lab.
3. On the resource page, select Delete and follow the instructions to delete the resource.

What if you have a mic and speaker?

In this exercise, the Azure Cloud Shell environment we used doesn’t support audio hardware, so you used audio files for the speech input and output. Let’s see how the code can be modified to use audio hardware if you have it available.

Using speech recognition with a microphone

If you have a mic, you can use the following code to capture spoken input for speech recognition:

code

# Configure speech recognition

audio_config = speech_sdk.AudioConfig(use_default_microphone=True)

speech_recognizer = speech_sdk.SpeechRecognizer(speech_config, audio_config)

print(‘Speak now…’)

# Process speech input

speech = speech_recognizer.recognize_once_async().get()

if speech.reason == speech_sdk.ResultReason.RecognizedSpeech:

command = speech.text

print(command)

else:

print(speech.reason)

if speech.reason == speech_sdk.ResultReason.Canceled:

cancellation = speech.cancellation_details

print(cancellation.reason)

print(cancellation.error_details)

Note: The system default microphone is the default audio input, so you could also just omit the AudioConfig altogether!

Using speech synthesis with a speaker

If you have a speaker, you can use the following code to synthesize speech.

code

response_text = ‘The time is {}:{:02d}’.format(now.hour,now.minute)

# Configure speech synthesis

speech_config.speech_synthesis_voice_name = “en-GB-RyanNeural”

audio_config = speech_sdk.audio.AudioOutputConfig(use_default_speaker=True)

speech_synthesizer = speech_sdk.SpeechSynthesizer(speech_config, audio_config)

# Synthesize spoken output

speak = speech_synthesizer.speak_text_async(response_text).get()

if speak.reason != speech_sdk.ResultReason.SynthesizingAudioCompleted:

print(speak.reason)

Note: The system default speaker is the default audio output, so you could also just omit the AudioConfig altogether!

More information

For more information about using the Speech-to-text and Text-to-speech APIs, see the Speech-to-text documentation and Text-to-speech documentation.

Summary

• Provision an Azure resource for the Azure Speech service
• Use the Speech to text API to implement speech recognition
• Use the Text to speech API to implement speech synthesis
• Configure audio format and voices
• Use Speech Synthesis Markup Language (SSML)

To learn more about the Azure Speech, refer to the Azure Speech service documentation.