Description
Describe the bug
InputImage.fromMediaImage and InputImage.fromBitmap do not seem to be applying parameter rotationDegrees properly, at least when used with text-recognition.
I'm attempting to fix MLKit results when used with react-native-vision-camera
, and discovered that all rotationDegrees
values supplied as input result in the exact same detection outcomes as not rotating the image at all (180 degrees rotation from upright sensor returns text lines in the opposite order). The data is YUV_420_888 directly from Camera2/CameraX as Image
.
Converting the YUV to Bitmap manually and using InputImage.fromBitmap
returned the same results.
The only way I could circumvent this was by rotating the bitmap before creating the InputImage, or setting automatic rotation of the buffers in CameraX.
My original issue in vision-camera
mrousavy/react-native-vision-camera#3490
To Reproduce
Take Image from CameraX or Camera 2 when rotated to 180 degrees from upright sensor, process with InputImage, and process with TextRecognizer. Check the order of the lines in returned text.
Getting the sample app up and running and modifying all relevant parts to reproduce this is a huge task for me, but here's the callback logic I have tested with vision-camera
private var recognizer = TextRecognition.getClient(TextRecognizerOptions.DEFAULT_OPTIONS)
override fun callback(frame: Frame, arguments: Map<String, Any>?): HashMap<String, Any>? {
val data = WritableNativeMap()
val mediaImage: Image = frame.image
val image =
InputImage.fromMediaImage(mediaImage, 180) // Hard coded for testing purposes, no values worked as expected
val task: Task<Text> = recognizer.process(image)
try {
val text: Text = Tasks.await(task)
if (text.text.isEmpty()) {
return WritableNativeMap().toHashMap()
}
data.putString("resultText", text.text) // This text comes in reversed order when sensor is at 180 deg
data.putArray("blocks", getBlocks(text.textBlocks))
return data.toHashMap()
} catch (e: Exception) {
e.printStackTrace()
return null
}
}
Expected behavior
Expect resulting MediaImage to be rotated and result in expected Text Recognition output ordering.
SDK Info:
- com.google.android.gms:play-services-mlkit-text-recognition:19.0.1
- com.google.android.gms:play-services-mlkit-text-recognition-chinese:16.0.1 (+ others at same version)
- com.google.mlkit:translate:17.0.3
Also tested with latest non-play services variants.
Smartphone:
- Device/Simulator: Pixel 9 Pro, Asus Zenfone 8
- Device/Simulator OS: Android API 33, Android API 35
Development Environment:
(For Android issue feel free to skip this section)
- IDE Eversion: Android Studio Meerkat | 2024.3.1 Patch 1
- Laptop/Desktop: MacBook Pro M3 Max
- Laptop/Desktop OS/version: MacOS 15.4