CPU Picking (low-latency raycast)

[gibssa]

Hi Filament team,

I was hoping you could clarify your previous advice on directly raycasting touches.

I've been stuck a few days debugging my (CPU) screen touch -> world coordinate function. I'm using Filament's C++ API on iOS:

Goal is:

• camera looking down at a table (hard-coded Y-axis height in world-space).
• ball sitting on the table.
• ball follows user touch & drag.
• I can't use Filament's pick() API
  • (app requires low-latency touches, so can't wait for CPU -> GPU -> CPU PickingQueryResult)
• probably can't use MapManipulator as I'll need to animate camera movements later and the Manipulator API doesn't support directly setting its mEye field.
  • (note, I've tried MapManipulator raycasts here too, which I also can't get to work correctly in this demo)

I'm converting NDC -> View-Space -> World-Space, following @pixelflinger's previous advice here:

You want to take your position in clip space and apply the inverse of the projection matrix, which will give you a ray in view space, which you need to normalize.

If you need world space, you need to also apply the inverse transpose of the model matrix.

To go from mouse coords to clip space, it's just a remapping of x and y between -1 and 1. And z doesn't matter, so just use 0.
Also don't forget that clip space (-1,-1) is left-bottom (not left-top).

Here's my code (touches appear slightly off, misses more badly as I pan the camera left/right):

// NDC works fine, I can verify with printf().
// View-space, world-space, parametric ray not so much.
double clip_space_x = (x_pixels / screen_width  - 0.5f) * 2.0f;
double clip_space_y = (y_pixels / screen_height - 0.5f) * -2.0f;
math::double4 clip_space_coordinate{clip_space_x, clip_space_y, 0, _camera->getNear()};

// Convert clip-space -> view-space (multiply by inverse projection matrix).
// Then normalize it.
math::mat4 inverse_projection_matrix = Camera::inverseProjection(_camera->getProjectionMatrix());
math::double4 view_space_ray = inverse_projection_matrix * clip_space_coordinate;
math::double4 view_space_ray_normalized = normalize(view_space_ray);

// Convert ray in (normalized view-space) -> ray in (world-space).
math::mat4 model_matrix = _camera->getModelMatrix();
math::mat4 inverse_tranpose = inverse(transpose(model_matrix));

math::double4 world_space_ray = inverse_tranpose * view_space_ray_normalized;
math::double4 world_space_ray_normalized = normalize(world_space_ray);

// Assume ray starts at the camera's position in world-space.
math::float3 origin = _camera->getPosition();

// Treat ray as parametric equation, solve for hard-coded table
// position in world-space.
double hitbox_y_axis = -0.25;
double t = (hitbox_y_axis - origin.y) / world_space_ray_normalized.y;
double x = origin.x + (world_space_ray_normalized.x * t);
double z = origin.z + (world_space_ray_normalized.z * t);

printf("done: %f, %f, %f\n", x, hitbox_y_axis, z);

I have a few specific questions:

• PerViewUniforms.cpp passes a similar worldFromClip matrix to its shaders, but directly uses the inverse MVP matrices (no transpose on the inverse view matrix). Why no tranpose?
• as per the previous advice, my clip-space vector ignores z, but w seems to matter. _camera->getNear() seems to be the most correct for me so far, but do you have advice on what we should use for w in clip-space?
• it seems that normalizing the view-space ray doesn't affect my result as long as I normalize the world-space ray afterwards. Do we have to normalize twice?
  • (once in view-space, again in world-space?)

Thanks!

[pixelflinger]

You should put 1.0 in the w component of your clip-space coordinate.

You are forgetting to "divide-by-w" when you apply the inverse(projection). You should be doing:

view_space_ray = view_space_ray / view_space_ray.w;

The "inverse transpose" business is just how you transform vectors (like normals or other), for vectors, you can't apply the matrix just like you would for vertices. e.g. if you want to transform a vertex v by the matrix M, you do M*v, but if you need to transform a vector (e.g. a normal) you need to do inverse(transpose(M))*v. At least that's what most people use, but it is actually incorrect in some case, the "real" correct way to do it is to use the cofactor(M). So I suggest you actually use our getTransformForNormals instead (the comments in the code has tons of reference on this):

double3 world_space_ray = mat3::getTransformForNormals(model_matrix.upperLeft())) * view_space_ray_normalized.xyz;

I think your biggest "mistake" is to not set w to 1 and to not do the divide-by-w.

Hope this helps! Let us know.

[gibssa]

Works perfectly, thanks @pixelflinger !