An Incident Reporting App powered by AI.

— A PowerApp Integrated with Microsoft Custom Vision API and Bing Maps API.

This Blog is a part of the Microsoft Developer Stories Initiative by Microsoft Student Partners (India) Program, please follow this link for further details https://studentpartners.microsoft.com/

Microsoft’s goal “To Empower People” has really lead to developing so many new technologies and Instruments through which people could create an app, train a model by just clicking some buttons.

This Less Code has approach has really lead people to come up with some really great ideas. In this blog, I’ll teach you how I merged two Less Code Platforms(Powerapps and Custom Vision) and created an awesome Incident Reporting app.

So This was my First Powerapp which basically started as a Simple Image Classification based on Custom Vision Model. In order to make something out of this amazing tech, I came up with an Idea through which you could create an incident reporting system in which the User be able to report an incident from the app and the Officer would be able to see the exact location from where the incident was reported and I used Custom Vision Algorithm to detect the image so that fraud reporting would also be reduced.

Part 1: Create a Model Using Custom Vision AI

· Go to customvision.ai, Then log in to your Microsoft Account with an Active Azure Subscription.

· Select New Project -> Enter Name, Description, Select a Resource -> Select Classification -> Multiclass(Single tag per Image) -> General Domain -> Click Create Project

Create New Project Screen

· Add Images -> Select your Images -> Then Add the Preferred Tag to it -> Upload.

Note: Make sure you have 2 Classes with 5 Images in it at least.

· After Uploading the Images with the tags, Click on Train — For now use the Quick Training Feature, Later you can train the model in Advanced Mode.

I’ll be adding Images of Garbage and Humans so that the User can detect Garbage on streets and report back to the Officer.

· When the Training is completed, you would see Performance Screen, in which you’ll able to the Precision, Recall, and AP of your model. Click the Publish button, Give a Name to the Project and then click Publish. Now You’ll get a Prediction key for your model.

Now Once that part is done Let’s move to the PowerApps Section. This is the fun part and you’ll see how much of #LessCode is required.

Part 2: Create the App Using Microsoft PowerApps

· Go to make.powerapps.com, Then log in to your Microsoft Account with an Active Azure Subscription.

· Click on the ‘Canvas app from blank’ ->Write an App Name -> Select your App Layout Format (I’ll stick to Phone Layout as it is more common).
· So now, you’ll be on the default starting screen of the app.

Default Screen of PowerApps

· So Let us Start by Adding a Simple Button to during the App Startup and when it is clicked it would navigate you to the Main Screen.

1] Go to Insert, Click on Button and a Button will appear on your screen, drag it to a place and if you double-click it you will be able to Rename the text of the button. So Let us Create 2 Buttons, One being for the User and the other being for an Officier.

It would look something like this, then if you see on the Left-Hand Side, you might be able to see the Tree View of your Powerapp. Rename the Screen1 to MainScreen.

Tree View of the Powerapp

So once that’s done, Click New Screen at the top and then select ‘Blank’. This will create a New Screen in the Tree View, Rename that Screen to ‘UserScreen’.

In the UserScreen, Click the Drop Down Menu of Media and Select ‘Add Picture’ Option. This will add create an “Add Image” Section on the screen. Now Create a Button and Rename it has “Scan”.

UserScreen View

Now we need to Program the button in such a way that, after we add an image to the section and click Scan, we should get the result from the model. For that, we’ll be using Custom Vision Connector Tool.

Click on View, then Data Sources, In it search for “Custom Vision”

Click “Add a Connection”, then It’ll ask you the Prediction Key and The Site URL for the Model, You can get both of them through the “Prediction URL” button from custom vision website.

Prediction URL Windows Example

Once you are able to connect the model to your Powerapps, You’re good to go to use the Model in your powerapps. So now go back to your UserScreen and click the Button, Change its “OnSelect Property” to ClearCollect(Collection,Item) where

Collection:- Give the name of your choice for the Image Collection database
Item:- Here we need to implement our Model, So We’ll use ‘CustomVision.ClassifyImageV2(“Project-id”, “Published Name”, UploadedImage1).predictions’

So, in the end, it should look something similar to this,

Example Function

So now you have configured the app to detect the image, now you need to something to display it, So Go to Insert->Gallery->Select Vertical. Then Select the Collection name which you had set to the button previously.

Get Sunder Eshwar’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Then Change the Gallery Layout to ‘Title and Subtitle’. Now, change the “Text” property of the Title to “ThisItem.tagName” and Subtitle to “Round(ThisItem.probability*100,2) & “%”. After selecting the property it should look something like this.

Text Property for Subtitle

UserScreen Layout for the Powerapp

Now you can start the app and test if the function is working correctly or not. I’ll test with adding an Image of Trash and see whether it is detecting or not.

Test Detection of the App

Okay so now we have the User Screen where he could scan and image, now let us set up the Map Console Screen for the Officer.

Now Add a New Blank Screen and name it has OfficerScreen. So we’ll be using Microsoft’s Bing Map. So Go to “Data Sources” then search for “Bing Maps”, select it and add the API key for the Map.

Now go back to OfficerScreen, then Select Media -> Select Image and set it on the screen of your App.

Now Create an Excel Sheet in your Onedrive and Make a Table in it as follows,

Excel Sheet Example

Then Go to the Data Sources and add “OneDrive” and then Select the Table that you have created and now you can use this table to store the data of the locations.

Now once you have set up the Database, go back to OfficerScreen

Select Data Table->Select the Onedrive Table->Click select fields->Add both the Latitude and Longitude fields.

Now Add 2 Text Inputs, with the ‘Default’ Property being ‘Value(DataTable1.Selected.Latitute)’ and Value(DataTable1.Selected.Longitude). This will fetch the active selection in the data table.

Now Select the Image Module, change its ‘Image’ value to BingMaps.GetMapV2(“Aerial”,18,TextInput1,TextInput2)

Now in the end it should look something like this.

Example OfficerScreen of the Powerapp

The Resulting Screen would be as follows. So now you need to just add a function that would add the Coordinates in the Database through which the Maps can take the data.

Go back to the User Screen, Add a Button and Rename it has “Send Data”. Then Change the OnSelect Property to “Patch( Table1, Defaults( Table1 ), {Latitude: Location.Latitude},{Longitude:Location.Longitude} )”

‘Send Data’ Button OnSelect Property

This Function will fetch your Device’s Location Coordinates and adds it to your Database.

Before we start testing go back to the Main Screen and Click the User Button and Set its OnSelect Property to “Navigate(UserScreen)” and the Officer Button to “Navigate(OfficerScreen)”. Also Just a back button in each screen so that we could go back and forth the screens.

So now we are ready to test the app.

So hit Start, Click User

Select Add an Image and Click Scan, After getting the Output, Click Send Data.

Now Go Back, to the Main Screen, Click Officer. After you go to the OfficerScreen, you’ll a have screen similar to like this.

So the Coordinates were added to the database and the Bing Maps was fetching the coordinates from it. You can report many times and the one you select in the data table would be shown on the Map Screen.

So this was a basic Incident Reporting App that I made for the demo purposes, You can even implement features like
-Capturing Images Straight from the Camera.
-Making the Officer Screen Password Protected.
-Making a Screen in which you can edit, remove the data entries of the excel sheet.

Challenges you can do

Give these features a try for yourself. I have also done a Full Detailed Video Tutorial of this blog at youtu.be/fWbZwUsA6vY and I have also recorded a separate showcase video of the app at youtu.be/FwIIl_BNO1U.

So this was a basic demonstration of how we can use the Microsoft PowerPlatform with Custom Vision integrated with Microsoft’s Bing Map API and Onedrive to make a completely functioning #LessCode Incident Reporting PowerApp.

Before I end this blog, I’d like to thank MSP Developer Stories for giving me this amazing opportunity to showcase my app as a blog and I really did enjoy writing this.

Thank You.