Human-First Engineering: Building an Accessibility App for My Blind Grandma

Gen

My grandmother is blind and illiterate. For her even the simple task of counting money means having to depend on someone else. I wanted to change that.

Braille is the general solution used but for someone who cant read it would be very hard to teach to learn braille. The Bureau of Engraving and Printing does offer a U.S. Currency Reader Program but the problem is that my grandmother lives in India, speaking Hindi and Punjabi, and uses Rupees. Every solution was either too difficult for her, only in English or only for U.S. currency.

After searching and searching I decided that if I couldn’t find a solution, I would create the solution. I needed to make something that would require minimal user to device interaction. Something with no reading, typing or menus. I did some research and learned how to create a simple application that would help solve her money identification problem.

I created an application where she would have to place her money on a surface, place her phone over the general area where they are, double tap the back of her phone and her phone would tell her how much money she has in her native language. It was amazing to see my grandmother who is constantly depending on people be able to complete a task on her own.

This is how I made it happen. I realized the most efficient way to create this application would be through the Apple Shortcut app and using Google Gemini’s API. Let’s break down how it works.

1.) The Trigger (The Back Tap): The shortcut is activated by the double tap on the back of the Iphone

2.) The Runner (iOS Shortcut): The phone uses the back camera to snap a photo, resizes it so it’s smaller and converts it to JPEG, this makes it so that it is a lot quicker when sending to the AI.

3.) The Expert (Google Gemini’s API): The phone sends the image to the AI with the specific task “Identify this Indian currency note. Answer ONLY with the value in Punjabi.” I created a GoogleAI studio account & generated the API key to call the Gemini API from the shortcut app

4.) The Voice: AI identifies the amount in rupees (example: 500 rupees), sends the text back and the Iphone speaks it out loud

From this experience I learned that sometimes all it takes to make impactful change is an idea and the right tools. But this isn’t something that is only available to my grandmother. Anyone with an Iphone can create this application

For anyone interested, step by step instructions are provided below.

Demo video: https://drive.google.com/file/d/1MFn-H5xJNHxWlya9oF9k6YO1Dpq4bz1M/view?usp=sharing

Shortcut Video Walkthrough : https://drive.google.com/file/d/1Me7Pf3LxOHxKk52dfTyx6o4QdNUAI5RE/view?usp=sharing

The Architecture

• Trigger: Tap the back of the iPhone twice (Back Tap).
• Input: The phone silently captures a photo of the currency.
• Intelligence: The image is sent to Gemini 2.5 Flash for instant recognition.
• Output: The phone speaks the result aloud in Hindi.

Phase 1: Get the API Key

To let the iPhone “see,” we need access to Google’s Gemini API. It’s powerful, fast, and offers a free tier as well

• Go to Google AI Studio on your computer.
• Sign in with your Google account.
• Click “Get API Key” and then “Create API Key.”
• Copy this key and send it to your iPhone

Phase 2: Build the Shortcut

Get alisha’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Open the Shortcuts app on the iPhone and tap the + icon. Name the shortcut “CurrencyReader”

1. Capture and Optimize the Image

We need to capture a photo and shrink it so it uploads instantly.

• Take Photo: Search for “Take Photo.” Tap the arrow and turn OFF “Show Camera Preview.”
• Resize Image: Search for “Resize Image.” Set the width to 800 (height on Auto). This keeps the file small but clear enough for AI.
• Convert Image: Search for “Convert Image.” Set it to convert the resized image to JPEG.
• Base64 Encode: Search for “Base64 Encode.” Set the input to the Converted Image and ensure Line Breaks are set to None. This turns the photo into a text string the AI can read.

2. The “Text” Action

This step packages the photo and instructions into a format Google’s servers understand. Add a Text action and paste the following JSON code:

JSON
{
  "contents": [
  {
  "parts": [
  {
  "text": "Identify this Indian currency note. Answer ONLY with the value in Hindi."
  },
  {
  "inline_data": {
  "mime_type": "image/jpeg",
  "data": "INSERT_VARIABLE_HERE"
  }
  }
  ]
  }
  ]
}

Crucial: Highlight INSERT_VARIABLE_HERE, delete it, and select the Base64 Encoded variable from the bar above your keyboard. It should appear as a blue token.

3. Send to Gemini (Get Contents of URL)

Add the “Get Contents of URL” action.

• URL: Paste the following link, replacing YOUR_API_KEY with the key from Phase 1: https://generativelanguage.googleapis.com/v1beta/models/gemini-flash-latest:generateContent?key=YOUR_API_KEY
• Method: Change to POST.
• Headers: Add a new header with Key: Content-Type and Value: application/json.
• Request Body: Change to File and choose the Text variable from Step 2.

4. Extract and Speak the Result

• Get Dictionary Value: Type the following path exactly into the key field: candidates.1.content.parts.1.text. Set the source to the Contents of URL.
• Speak Text: Add the “Speak” action. Tap the arrow to select the Language (Hindi).

Phase 3: The Back Tap Trigger

Now, enable back tap trigger

• Go to iPhone Settings > Accessibility > Touch.
• Scroll down to Back Tap.
• Choose Double Tap.
• Scroll down to the “Shortcuts” section and select “CurrencyReader”