Accurately modeling the eye is a challenging task as it exhibits refraction and reflection at the cornea, complex iris texture, and self-occlusion and shadowing due to eyelids and eyelashes. To address these challenges, we present a system for learning a hybrid relightable eye model which can be relit under near-field point lights. Our hybrid model leverages an eyeball mesh for explicitly representing the cornea surface and the reflections on it while learning the geometry and light transport of the periocular region and eye interior implicitly. To account for refraction, we explicitly handle the refraction of camera rays using Snell's law and predict the refraction of incident light rays using a neural network. Furthermore, we propose an extension of our method which enables us to relight the eye using a fringe projector to simulate structured light. Through experiments, we demonstrate that our method results in higher fidelity rendering under novel viewpoint and lighting conditions, improves learned iris geometry, and more accurately simulates structured light fringe patterns on the eye.
Our model renders the eye from viewpoints not seen during training while preserving fine detail. It recovers detailed iris texture behind the refractive cornea, and the explicit eyeball mesh captures the shape of the cornea and the reflections on it. Learning the periocular region implicitly further allows our model to reconstruct thin geometric structures such as eyelashes.
Because our model learns the light transport of the periocular region and eye interior, the eye can be relit under arbitrary near-field point lights. To handle refraction at the cornea, we refract camera rays using Snell's law and predict the refracted path of incident light rays with our Direct Illumination Ray network. Below, we show subjects rendered under continuously changing illumination, at novel light positions, and with an increasing number of point lights.
Without ground truth iris geometry, we evaluate how well the subject's gaze direction can be recovered from our reconstructed iris. We fit a plane to the reconstructed iris and measure the angular error (AE) between its normal (red arrow) and a reference optic axis from fitting the pose of our eye mesh (blue arrow). Iris points are colored by depth along the optic axis. Without refraction (NR), the iris is noticeably curved and takes on the shape of the cornea, while accounting for camera and light ray refraction (CRR + LRR) yields a much flatter iris and improves gaze estimation by 3.5°.
NR
CRR + LRR
We extend our method to relight the eye with a fringe projector to simulate structured light. To evaluate our simulation, we capture the same subject with both our light stage and a structured light rig and compare real and simulated fringe patterns. We compute the wrapped phase using a 3-step phase-shift profilometry (PSP) method and compare cross sections of the real and simulated phase. Ignoring refraction (NR) or modeling only camera ray refraction (CRR) leads to blurred and warped fringes, while our full pipeline (CRR + LRR) recovers the fringe pattern with minimal distortion and produces the most accurate phase.
| NR | CRR | CRR + LRR | |
|---|---|---|---|
| Angular Error | 8.65 | 5.09 | 5.12 |
Absolute error between simulated and real phase averaged over different cross sections of the iris.
(a) Structured Light
(b) Wrapped Phase
(c) Phase Cross Section
Real Capture
NR
CRR
CRR + LRR
(a) Comparison of real and simulated structured light. (b) Wrapped phase computed using a 3-step PSP method. (c) Cross section comparison of the real and simulated wrapped phase.
@inproceedings{khademi2026reconstructing,
title={Reconstructing Realistic and Relightable Eyes},
author={Khademi, Wesley and Kundu, Jogendra Nath and An, Yatong and Fix, Alexander and Colmenares, David},
booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
year={2026}
}