Skip to content

Ocv instr test #364

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 90 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
90 commits
Select commit Hold shift + click to select a range
9e6a1bb
call native test method
vkocheryzhkin Apr 15, 2018
17566d2
Add custom detector
Apr 16, 2018
2191f3b
send bitmap to native
vkocheryzhkin Apr 17, 2018
73f2f2f
add dlib as submodule
vkocheryzhkin Apr 17, 2018
30b8869
link with dlib
vkocheryzhkin Apr 17, 2018
6bc6767
start conversion of bmp to rgb dlib image
vkocheryzhkin Apr 18, 2018
58cd257
basic dirty recognition
Apr 18, 2018
5205730
try to build release
Apr 18, 2018
e007fb9
play with rgb in native
Apr 19, 2018
2294feb
add basic python to get face vector
Apr 19, 2018
c1cca81
fix thresh for face rec
vkocheryzhkin Apr 19, 2018
5bc87c3
decrease threshold
vkocheryzhkin Apr 19, 2018
89d9677
restyle for demo
vkocheryzhkin Apr 19, 2018
95bf942
demo prepare #2
Apr 20, 2018
349fe5a
add identify button to use dlib from photo
vkocheryzhkin Apr 28, 2018
923e0b4
use separate sp and net
vkocheryzhkin Apr 28, 2018
86365a8
add basic instrumentation test
vkocheryzhkin May 4, 2018
29672e5
add travis
vkocheryzhkin May 8, 2018
3c295df
modify travis
vkocheryzhkin May 8, 2018
5d60921
update travis
vkocheryzhkin May 8, 2018
845d8ac
update travis
vkocheryzhkin May 8, 2018
6561285
update travis 4
vkocheryzhkin May 8, 2018
3f9e256
update travis 5
vkocheryzhkin May 8, 2018
c4684a3
remove abs path to signing
vkocheryzhkin May 8, 2018
831f491
update travis
vkocheryzhkin May 8, 2018
93ccda9
dup key for release
vkocheryzhkin May 8, 2018
d840398
disable lint temprarily
vkocheryzhkin May 8, 2018
2d43257
add simple native test
May 15, 2018
fb12260
start refactoring
vkocheryzhkin May 15, 2018
081986f
try to move all logic into FaceRecognizer
May 16, 2018
d37b412
try to read image from assets
vkocheryzhkin May 16, 2018
5530658
fix test
vkocheryzhkin May 16, 2018
1d2d322
refactor recognize methods
vkocheryzhkin May 18, 2018
a962582
try orientation with exif
vkocheryzhkin May 21, 2018
74faab8
rotate cropped bitmap
May 22, 2018
bb1078e
implement all orientations
vkocheryzhkin May 22, 2018
c4ca415
clean up
vkocheryzhkin May 22, 2018
e778aaf
delete exif
vkocheryzhkin May 22, 2018
f58b237
Merge pull request #1 from hoopoe/bitmap_orientation
vkocheryzhkin May 23, 2018
63f075a
This AS app is OpenCV and Dlib based configured as two different modu…
aloruss1 May 24, 2018
1153e58
build with opencv and dlib
vkocheryzhkin May 25, 2018
e49a4d8
remove opencv for java
vkocheryzhkin May 27, 2018
cf3db1e
remove jni libs
vkocheryzhkin May 27, 2018
8442d59
fix load dir
vkocheryzhkin May 27, 2018
b696585
merge master
vkocheryzhkin May 27, 2018
0da1b19
fix build
vkocheryzhkin May 27, 2018
8d299cb
fix local build for opencv
vkocheryzhkin May 27, 2018
df71024
Merge pull request #2 from hoopoe/AS-OpenCV-DLib-2modules
vkocheryzhkin May 27, 2018
5306431
restructure project
vkocheryzhkin May 27, 2018
a88b3c7
Update README.md
vkocheryzhkin May 27, 2018
ad52dfe
clean up
vkocheryzhkin May 28, 2018
1a593d6
disable identify button temporarly. try to fix build
vkocheryzhkin May 28, 2018
cd61e96
fix build
vkocheryzhkin May 28, 2018
2906463
enable auto focus
vkocheryzhkin May 30, 2018
d044290
update readme for rt statement
vkocheryzhkin May 30, 2018
3dcab26
update readme
vkocheryzhkin May 31, 2018
9701c10
request permissions for camera and sdcard together
vkocheryzhkin Jun 1, 2018
6bb8264
get cam resolution
vkocheryzhkin Jun 2, 2018
0142554
try to fix permission issue
vkocheryzhkin Jun 2, 2018
b0a9375
add Spc detector
sweetdream779 Jun 3, 2018
36ca54d
add Spc detector
sweetdream779 Jun 3, 2018
dc5a8f1
fix initial portrait orientation
vkocheryzhkin Jun 3, 2018
c7e7768
Merge pull request #3 from hoopoe/permissions
vkocheryzhkin Jun 4, 2018
cb6d7af
refactor to use 2 activities
vkocheryzhkin Jun 4, 2018
d609727
change readme; change build gradle
sweetdream779 Jun 4, 2018
9e4d3ad
change readme
sweetdream779 Jun 4, 2018
43e6d4b
Merge pull request #4 from hoopoe/activity
vkocheryzhkin Jun 5, 2018
6277ec9
merge projects
sweetdream779 Jun 6, 2018
b68dba6
merge projects
sweetdream779 Jun 6, 2018
7c1ba3d
remove .externalNativeBuild
sweetdream779 Jun 7, 2018
6b8416d
get tensorflow before build
vkocheryzhkin Jun 7, 2018
7624cfc
update travis
vkocheryzhkin Jun 7, 2018
e606a10
Merge pull request #6 from hoopoe/activity
vkocheryzhkin Jun 7, 2018
867dbf5
Merge branch 'master' into dlib-opencv-tf
vkocheryzhkin Jun 8, 2018
4e73909
change readme; add OPENCV_ANDROID_SDK var in CMakeLists
sweetdream779 Jun 8, 2018
3245ee7
Merge branch 'dlib-opencv-tf' of https://github.com/sweetdream779/FR-…
sweetdream779 Jun 8, 2018
f9a876c
Update README.md
sweetdream779 Jun 8, 2018
d211a05
Update README.md
sweetdream779 Jun 8, 2018
18edb6f
change paths; updates readme
sweetdream779 Jun 8, 2018
555ec04
change paths; updates readme
sweetdream779 Jun 8, 2018
a259d1e
return $ENV{OPENCV_ANDROID_SDK}
sweetdream779 Jun 8, 2018
f6d82fb
dlib as submodule
sweetdream779 Jun 8, 2018
60e3c66
change abiFilters
sweetdream779 Jun 8, 2018
359b090
del dlib
sweetdream779 Jun 9, 2018
01348d4
add submodule
sweetdream779 Jun 9, 2018
f3be575
Merge pull request #5 from sweetdream779/dlib-opencv-tf
vkocheryzhkin Jun 13, 2018
a5fcd0e
clean up. reorder activities
vkocheryzhkin Jun 14, 2018
6ba4f0c
merge of tensorflow and OpenCV with google-vision Dlib
aloruss1 Jun 19, 2018
0544881
added lint options for OpenCV and excluded .externalNativeBuild folde…
aloruss1 Jun 19, 2018
a48ff43
.externalNativeBuild deletion
aloruss1 Jun 19, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
try orientation with exif
  • Loading branch information
vkocheryzhkin committed May 21, 2018
commit a962582c0d3467a40a9b95832075e5ad38d03615
81 changes: 39 additions & 42 deletions visionSamples/FaceTracker/app/src/main/cpp/native-lib.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -325,51 +325,48 @@ Java_dlib_android_FaceRecognizer_recognizeFace(JNIEnv *env, jobject instance, jo
}

//todo: smth wrong with colors
//dlib::save_bmp(img, "/storage/emulated/0/Download/test.bmp");

std::vector<dlib::rectangle> dets = detector(img);
LOGI("detected size %d", dets.size());

float min_dist = 0.0;
if(dets.size() > 0 ){
auto face = dets.front();
std::vector<matrix<rgb_pixel>> faces;
int x = face.left();
int y = face.top();
int width = face.width();
int height = face.height();

auto shape = sp(img, face);
matrix<rgb_pixel> face_chip;
extract_image_chip(img, get_face_chip_details(shape, 150, 0.25), face_chip);
faces.push_back(move(face_chip));

std::vector<matrix<float, 0, 1>> face_descriptors = net(faces);

if (face_descriptors.size() > 0)
{
matrix<float, 0, 1> face_desc = face_descriptors[0];
for (auto& i : known_faces) {
float dist = length(face_desc - i.second );
if (dist < min_dist){
min_dist = dist;
}
if( dist < FACE_RECOGNIZE_THRESH) //todo: extract thresh
{
LOGI("recognized");
return env->NewStringUTF(i.first.c_str());
}
}
}
LOGI("not recognized, max dist %0.2f", min_dist);
}

dlib::save_bmp(img, "/sdcard/Download/res.bmp");

// std::vector<dlib::rectangle> dets = detector(img);
// LOGI("detected size %d", dets.size());
//
// float min_dist = 0.0;
// if(dets.size() > 0 ){
// auto face = dets.front();
// std::vector<matrix<rgb_pixel>> faces;
// int x = face.left();
// int y = face.top();
// int width = face.width();
// int height = face.height();
//
// auto shape = sp(img, face);
// matrix<rgb_pixel> face_chip;
// extract_image_chip(img, get_face_chip_details(shape, 150, 0.25), face_chip);
// faces.push_back(move(face_chip));
//
// std::vector<matrix<float, 0, 1>> face_descriptors = net(faces);
//
// if (face_descriptors.size() > 0)
// {
// matrix<float, 0, 1> face_desc = face_descriptors[0];
// for (auto& i : known_faces) {
// float dist = length(face_desc - i.second );
// if (dist < min_dist){
// min_dist = dist;
// }
// if( dist < FACE_RECOGNIZE_THRESH) //todo: extract thresh
// {
// LOGI("recognized");
// return env->NewStringUTF(i.first.c_str());
// }
// }
// }
// LOGI("not recognized, max dist %0.2f", min_dist);
// }
//
LOGI("unlocking pixels");
AndroidBitmap_unlockPixels(env, bmp);

//std::string returnValue = "Unknown" + std::to_string(min_dist);


std::string returnValue = "Unknown";
return env->NewStringUTF(returnValue.c_str());
}
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
import java.io.FileWriter;
import java.io.IOException;

import dlib.android.Exif;
import dlib.android.FaceRecognizer;

import static android.os.Environment.getExternalStorageDirectory;
Expand Down Expand Up @@ -83,6 +84,8 @@ public SparseArray<Face> detect(Frame frame) {
yuvImage.compressToJpeg(new Rect(0, 0, frame.getMetadata().getWidth(),
frame.getMetadata().getHeight()), 100, byteArrayOutputStream);
byte[] jpegArray = byteArrayOutputStream.toByteArray();
int orientation = Exif.getOrientation(jpegArray);
Log.w(TAG, String.format("Orientation %d", orientation));
Bitmap tmpBitmap = BitmapFactory.decodeByteArray(jpegArray, 0, jpegArray.length);
final Bitmap cropped = Bitmap.createBitmap(tmpBitmap, x, y, w, h);
// try {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ public final class FaceTrackerActivity extends AppCompatActivity {
private GraphicOverlay mGraphicOverlay;
private Button mBtnDetect;
private CustomDetector customDetector;
private FaceDetector mPictureDetector;
//private FaceDetector mPictureDetector;

private static final int RC_HANDLE_GMS = 9001;
// permission request codes need to be < 256
Expand Down Expand Up @@ -185,16 +185,16 @@ private void createCameraSource() {
.setClassificationType(FaceDetector.NO_CLASSIFICATIONS)
.setProminentFaceOnly(false)
.setMode(FaceDetector.ACCURATE_MODE)
.setMinFaceSize(0.05f)
.setMinFaceSize(0.015f)
.build();

mPictureDetector = new FaceDetector.Builder(context)
.setTrackingEnabled(false)
.setProminentFaceOnly(false)
.setMinFaceSize(0.015f) // 80 / 5312 detect up to 80 pixels head width
.setMode(FaceDetector.ACCURATE_MODE)
.setClassificationType(FaceDetector.NO_CLASSIFICATIONS)
.build();
// mPictureDetector = new FaceDetector.Builder(context)
// .setTrackingEnabled(false)
// .setProminentFaceOnly(false)
// .setMinFaceSize(0.015f) // 80 / 5312 detect up to 80 pixels head width
// .setMode(FaceDetector.ACCURATE_MODE)
// .setClassificationType(FaceDetector.NO_CLASSIFICATIONS)
// .build();

//mFaceRecognizer
customDetector = new CustomDetector(detector, mFaceRecognizer);
Expand All @@ -215,9 +215,9 @@ private void createCameraSource() {
Log.w(TAG, "Face detector dependencies are not yet available.");
}

if (!mPictureDetector.isOperational()) {
Log.w(TAG, "mPictureDetector dependencies are not yet available.");
}
// if (!mPictureDetector.isOperational()) {
// Log.w(TAG, "mPictureDetector dependencies are not yet available.");
// }

//.setRequestedPreviewSize(640, 480)
//.setRequestedFps(30.0f)
Expand Down
122 changes: 122 additions & 0 deletions visionSamples/FaceTracker/app/src/main/java/dlib/android/Exif.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
package dlib.android;

import android.util.Log;

//https://github.com/googlesamples/android-vision/issues/124
public class Exif {
private static final String TAG = "CameraExif";

// Returns the degrees in clockwise. Values are 0, 90, 180, or 270.
public static int getOrientation(byte[] jpeg) {
if (jpeg == null) {
return 0;
}

int offset = 0;
int length = 0;

// ISO/IEC 10918-1:1993(E)
while (offset + 3 < jpeg.length && (jpeg[offset++] & 0xFF) == 0xFF) {
int marker = jpeg[offset] & 0xFF;

// Check if the marker is a padding.
if (marker == 0xFF) {
continue;
}
offset++;

// Check if the marker is SOI or TEM.
if (marker == 0xD8 || marker == 0x01) {
continue;
}
// Check if the marker is EOI or SOS.
if (marker == 0xD9 || marker == 0xDA) {
break;
}

// Get the length and check if it is reasonable.
length = pack(jpeg, offset, 2, false);
if (length < 2 || offset + length > jpeg.length) {
Log.e(TAG, "Invalid length");
return 0;
}

// Break if the marker is EXIF in APP1.
if (marker == 0xE1 && length >= 8 &&
pack(jpeg, offset + 2, 4, false) == 0x45786966 &&
pack(jpeg, offset + 6, 2, false) == 0) {
offset += 8;
length -= 8;
break;
}

// Skip other markers.
offset += length;
length = 0;
}

// JEITA CP-3451 Exif Version 2.2
if (length > 8) {
// Identify the byte order.
int tag = pack(jpeg, offset, 4, false);
if (tag != 0x49492A00 && tag != 0x4D4D002A) {
Log.e(TAG, "Invalid byte order");
return 0;
}
boolean littleEndian = (tag == 0x49492A00);

// Get the offset and check if it is reasonable.
int count = pack(jpeg, offset + 4, 4, littleEndian) + 2;
if (count < 10 || count > length) {
Log.e(TAG, "Invalid offset");
return 0;
}
offset += count;
length -= count;

// Get the count and go through all the elements.
count = pack(jpeg, offset - 2, 2, littleEndian);
while (count-- > 0 && length >= 12) {
// Get the tag and check if it is orientation.
tag = pack(jpeg, offset, 2, littleEndian);
if (tag == 0x0112) {
// We do not really care about type and count, do we?
int orientation = pack(jpeg, offset + 8, 2, littleEndian);
switch (orientation) {
case 1:
return 0;
case 3:
return 180;
case 6:
return 90;
case 8:
return 270;
}
Log.i(TAG, "Unsupported orientation");
return 0;
}
offset += 12;
length -= 12;
}
}

Log.i(TAG, "Orientation not found");
return 0;
}

private static int pack(byte[] bytes, int offset, int length,
boolean littleEndian) {
int step = 1;
if (littleEndian) {
offset += length - 1;
step = -1;
}

int value = 0;
while (length-- > 0) {
value = (value << 8) | (bytes[offset] & 0xFF);
offset += step;
}
return value;
}
}