AI Virtual Mouse
Contributor Usman Noor
Artificial Intelligence (AI) is an emerging field, that has gained a lot of popularity in recent times. However, AI technologies have been under development for quite some time and can be dated back to 1940’s.
The use of AI is in varied in both industries and human lives as well. With the capacity to increase productivity, enhance decision-making, and improve overall human experiences, AI has the potential to revolutionize the world as we know it. We, however are not building AI models, rather using pre-trained AI model, to build the Virtual Mouse that can controlled using hand gestures. The model we use is Mediapipe, trained and developed by google. The programming language to be used is Python version 3.10.5. And the basic GUI was developed using tkinter.
We used mediapipe and opencv2 for python to take the image feed from the user and process it. We used mediapipe hand solutions, which provide 21 unique landmarks for the hand. We use those landmarks to identify different gestures and can perform a multitude of actions through it.
The solution we have developed is object oriented based with two main classes. The HandDetector class and GestureController class, both classes are linked together with hand detector mainly being used for carrying out different calculations on the landmarks and Gesture controller identifies different gestures and maps the functionality of keyboard and mouse to it.
The Gestures are defined in the user manual, there are different gestures for left and right hands. The total gestures that are mapped are 17 in total. It is advised to read the Manual before running the code.
Acquiring Landmarks
The opencv2 module was used to get the image feed from camera 0 (webcam by default). The image is retrieved and inverted in colors for mediapipe. Since Mediapipe works in RGB colors and OpenCV works in BRG. Then using the below code the Frame Rate is calculated and displayed on the screen. Furthermore, a functionality is added to end the task if q (“Q”) key is pressed. This helps a lot in development stages when there is a frequent need to check how the gestures perform.
def main():
pyg.FAILSAFE = False
cap = cv2.VideoCapture(0)
wcam = cap.get(3)
hcam = cap.get(4)
cap.set(cv2.CAP_PROP_FRAME_WIDTH, wcam)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, hcam)
#declaring objects
detector = handDetector()
controller = GestureControl()
ptime = 0
ctime = 0
while True:
success, img = cap.read()
#calling the instance methods
img = detector.draw_hands(img)
lmdict = detector.findPosition(img)
#Adds in the extra landmark for calculating distance
if len(lmdict) != 0:
if 'Left' in lmdict:
cl = detector.cl(lmdict)
cv2.circle(img, (cl[0], cl[1]), 5, (0, 0, 225), cv2.FILLED)
if ('Right' in lmdict):
cr = detector.cr(lmdict)
cv2.circle(img, (cr[0], cr[1]), 5, (0, 0, 225), cv2.FILLED)
#finding the gesture and performing functions
Fingers = detector.FindFingers(lmdict)
controller.Gestures(Fingers, lmdict, img)
key = cv2.waitKey(1)
if key == ord('q'):
cv2.destroyAllWindows()
break
ctime = time.time()
fps = 1/(ctime-ptime)
ptime = ctime
img = cv2.flip(img, 1)
cv2.putText(img, 'FPS: ' + str(int(fps)), (10, 30),
cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 0, 255), 2)
cv2.imshow("Virtual Mouse", img)This the main program for running the code and only 6 lines of codes calling different instance methods in both classes were added to standard OpenCV main program in python, to achieve our goal.
Find Fingers and Position
Since in some gestures right and left hand showing the same gesture perform different actions, like in case of all four fingers up of right hand changes brightness whereas the left hand changes volume. To implement this we had to list the landmarks in order of left and right hands. Since the screen is flipped the mediapipe module assigned the flipped labels to both hands and hence they were flipped in code to compensate for that.
We converted the continuous coordinates of the landmarks to pixel values, this was achieved by multiply the x coordinate with width of screen and y coordinate with height of the screen. This helped us a lot in the determining the change in hand in both x and y directions.
For actually using our gestures, we established whether each finger and thumb were raised or not. By adding them to their separate lists helped us differentiate in gestures of both hands.
def findPosition(self, img, draw=True):
# lmdict is a dictionary containing all the landmarks as a tupple and hand label as key
lmdict = {}
h, w, c = img.shape
# only runs if the hand(s) are detected
if self.result.multi_hand_landmarks:
for handType, handLms in zip(self.result.multi_handedness, self.result.multi_hand_landmarks):
# land mark List
mylmList = []
for id, lm in enumerate(handLms.landmark):
# converting landmarks to pixel coordinate sytem
px, py, pz = int(lm.x * w), int(lm.y * h), int(lm.z * w)
mylmList.append((id, px, py))
# compensating the labels for the flipped image
if handType.classification[0].label == "Right":
lmdict["Left"] = mylmList
else:
lmdict["Right"] = mylmList
return lmdict
def FindFingers(self, lmdict: dict):
# land mark values of tips of each finger and thumb
indexids = [4, 8, 12, 16, 20]
fingers_l = []
fingers_r = []
self.all_fingers = []
# only calculates if the hands were detected
if len(lmdict) != 0:
# for left hand
if ('Left' in lmdict):
fingers_l = [0, 0, 0, 0, 0]
# thumb caused a few errors if we calculated using landmark 5
midlandlt = (lmdict['Left'][indexids[0]][1] + lmdict['Left'][5][1])/2
if lmdict['Left'][indexids[0]][1] < midlandlt:
fingers_l[0] = 1
# rest of the four fingers
for id in range(1, 5):
if lmdict['Left'][indexids[id]][2] < lmdict['Left'][indexids[id]-2][2]:
fingers_l[id] = 1
# for right hand
if ('Right' in lmdict):
fingers_r = [0, 0, 0, 0, 0]
# thumb caused a few errors if we calculated using landmark 5
midlandrt = (lmdict['Right'][indexids[0]][1] + lmdict['Right'][5][1])/2
if lmdict['Right'][indexids[0]][1] > midlandrt:
fingers_r[0] = 1
# rest of the four fingers
for id in range(1, 5):
if lmdict['Right'][indexids[id]][2] < lmdict['Right'][indexids[id]-2][2]:
fingers_r[id] = 1
self.all_fingers.append(fingers_l)
self.all_fingers.append(fingers_r)
return self.all_fingersAdditional Landmark
This landmark was solely created and added for the functionality and ease of calculation in finding the change in movement, rather than using different points for each gesture and causing a wide variety of issues. We generalized the position, which was found by taking the average of point 9 and point zero of the hand. Then using the logic similar to one used for frame rate (previous and current values) the change in distances were found for the landmark 21.
Gesture Controller (Class)
Using the fingerlist and different libraries of python including pyautogui, winsound and screen_brightness_control.
Get Abiam Asif Khalid’s stories in your inbox
Join Medium for free to get updates from this writer.
Pyautogui is a versatile library offering almost all the functinality of keyboard and mouse available. Easy to use and understand. Winsound library was mainly used to add the screenshot sound in the background as an audio output of the command. Finally screen_brightness_control was used to control the screen brightness.
#For Taking a ScreenShot
if i == [1, 0, 0, 0, 1]:
pyg.hotkey("win", "prntscrn")
winsound.PlaySound("*", winsound.SND_ALIAS)
cv2.waitKey(1500)
# For Left Clicking
if (i == [1, 1, 1, 0, 0] or i == [0, 1, 1, 0, 0]) and (self.FindDist(8, 12, lmdict) > 50):
pyg.click(clicks=2)
pyg.keyUp('alt')
cv2.waitKey(1000)Above code block shows sample code for two gestures, showcasing both the keyboard and mouse functionality of pyauto. The FindDistance function calculates the distance between two fingers.
GUI (Graphical User Interface)
The GUI was build by using tkinker library of python. The background of the canvas was chosen as a gradient of purple and peach colors. Using the classes of the tkinter buttons were made to continue to main program that translates the gestures and reading the user manual. The Pillow library was used to allow mediapipe and OpenCV to work with tkinter interface.
import tkinter as tk
from PIL import Image, ImageTk
win.tk.geometry("1080x720+0+0")
win.title('Welcome To Virtual Mouse')
frame_1 = tk.Canvas(win, width=1080, height=720)
frame_1.place(x=0, y=0)
frame_1.pack()
def select_img():
_, img = cap.read()
img = cv2.resize(img, (w, h))
rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
image = Image.fromarray(rgb)
finalImage = ImageTk.PhotoImage(image)
label1.configure(image=finalImage)
label1.image = finalImage
win.after(1, select_img)
select_img()
win.mainloop()
#This code is essential for integrating Mediapipe/OpenCV in Tkinter based GUIVisual Sample
Below is a visual sample, demonstrating a few functions of the AI Virtual Mouse. (Performed by Usman Noor)
References
Elements of code were inspired by the following links:
- https://www.youtube.com/watch?v=01sAkU_NvOY&ab_channel=freeCodeCamp.org
- https://www.youtube.com/watch?v=yQSEXcf6s2I&list=PLCC34OHNcOtoC6GglhF3ncJ5rLwQrLGnV&ab_channel=Codemy.com
Code
Complete code can be found on github.
https://gist.github.com/usmanNoor5/776ce6ddcd3581e62697fc891a3725c3
