I've been puzzling over a problem that exhibits itself on flat surfaces. Cubemap reflections appear "jittery" as if they are been rounded off to a lower-precision float. At first I thought this might mean that our best-fit normals routine using 8 bits per channel might lack the precision needed for cubemap reflections. However, when I changed the normal texture in the gbuffer to a 32-bit floating.point RGBA texture, the problem remained. This error is also visible in Igor's SSLR shader.

I found that the camera matrix multiplication in the model vertex shader was causing this, but I couldn;t figure out why. In the model shader, the normal is multiplied by the camera normal matrix:

mat3 nmat = camerainversenormalmatrix;
nmat *= mat3(entitymatrix[0].xyz,entitymatrix[1].xyz,entitymatrix[2].xyz);
ex_normal = normalize(nmat * vertex_normal);
ex_tangent = normalize(nmat * vertex_tangent);
ex_binormal = normalize(nmat * vertex_binormal);

Then that operation is reversed in the probe lighting shader:

shadowcoord = cameranormalmatrix * reflect(screennormal*flipcoord,normal);

However, if I stored the fragment normal in world space rather than camera space, the error went away. This makes some sense because in world space the value is not constantly changing as you rotate the camera. In the video below I switch the code back and forth to adjust the two shaders and show the difference.

Changing this will require the matrix multiplication being changed in pretty much every lighting, post-processing, and model shader, which isn't a huge deal. I still don't understand why using a 32-bit floating point RGBA texture for the normal texture didn't fix the problem though. I also verified that the camera matrix was not changing between the model drawing and the probe drawing.

Part of the idea for the design for this system came from watching footage of the new Doom game, which makes good use of a parallax cubemapping effect all over the environment. You can see lots of lovely reflections in the pools of blood on the floor here: