Aberration Separation

Revealing the preference of deep-learning model on optical aberrations

Abstract:

The joint design of the optical system and the downstream algorithm is a challenging and promising task. Due to the demand for balancing the global optima of imaging systems and the computational cost of physical simulation, existing methods cannot achieve efficient joint design of complex systems such as smartphones and drones. In this work, starting from the perspective of the optical design, we characterize the optics with separated aberrations. Additionally, to bridge the hardware and software without gradients, an image simulation system is presented to reproduce the genuine imaging procedure of lenses with large field-of-views. As for aberration correction, we propose a network to perceive and correct the spatially varying aberrations and validate its superiority over state-of-the-art methods. Comprehensive experiments reveal that the preference for correcting separated aberrations in joint design is as follows: longitudinal chromatic aberration, lateral chromatic aberration, spherical aberration, field curvature, and coma, with astigmatism coming last. Drawing from the preference, a 10% reduction in the total track length of the consumer-level mobile phone lens module is accomplished. Moreover, this procedure spares more space for manufacturing deviations, realizing extreme-quality enhancement of computational photography. The optimization paradigm provides innovative insight into the practical joint design of sophisticated optical systems and post-processing algorithms.

Background:

With the popularity of mobile photography (e.g., smartphones, action cameras, drones, etc.), the optics of cameras tend to be more miniaturized and lightweight for imaging. The engaged methods can be divided into several categories: 1. from the perspective of image processing and 2. from the consideration of the whole system with end-to-end optimization. While they all have some imitations in implementation. So what if we start from the perspective of the optical design?

Comparison of the imaging system’s design methods: (a) Fixed lens design remedied by post-processing algorithms, (b) End-to-end method, and (c) Ours (Step-by-step joint design).

Method:

Our aim is to investigate the preference for correcting separated aberrations in joint optic-image design.

Overview of our joint optic-image design method.

Compound lens design has various degrees of freedom, and traversing all the potential designs is time-consuming. Following the vector quantization in auto-regressive, we note that mapping intricate data (e.g., lens configuration) to a highdimensional representation (e.g., aberrations coefficients) could realize a more comprehensive characterization. Fortunately, imaging quality is mostly corrupted by the degradation of primary aberrations, which happens to be quantified by Seidel coefficients. Therefore, considering the trade-off between accuracy and concision, Seidel is a better option in the view of aberration decomposition. Here we show the design results of our aberration separation strategy

And we analysis the distribution of degradation across all designs. This results shows that our strategy could obtain adquately sampled aberration/degration space.

Distribution analysis of degradation across all designs. From the perspectives of the MTF difference between tangential and sagittal directions (the upper half sector in (a)) and the PSF energy dispersion size (the upper half sector in (b)), (a) and (b) outline the differences in deterioration produced by various types of designs.

By the design strategy of aberration separation, we work with the same restoration pipeline used in Imaging Simulation. In this way, we could analyse which aberrations are easier for correction, and which aberrations are more difficult to restore.

The line charts of (a) PSNR and (b) SSIM evaluation before vs. after correction.

From the results above, we can easily find out that the deep learning model has a preference to different optical aberrations. The sequence is given as: CL > CT > SPHA > FCUR > COMA > ASTI (larger means easier to correct).

After getting this prior, we start to use this as a guidance for photography object lens design. And we realize the optimization of 10% height decrease in the main camera of mobile phone (from 6.1mm to 5.5mm).

lower TTL lens, but could have evenly matched imaging quality after the same deep-learning reconstruction.

We do the same analysis procedures on two different objective lens configuration, and the conclusions are the same!

Our Main Observation:

What optical aberrations do you need to control first in optical designing?

Follow me and say ASTI > COMA > FCUR > SPHA > CT > CL (loudly)

Let this MANTRA makes your LENS DESIGN BETTER!!!

References

2024

Revealing the preference for correcting separated aberrations in joint optic-image design

Jingwen Zhou, Shiqi Chen*, Zheng Ren, and 5 more authors

Optics and Lasers in Engineering, Sep 2024

Abs Bib PDF

The joint design of the optical system and the downstream algorithm is a challenging and promising task. Due to the demand for balancing the global optima of imaging systems and the computational cost of physical simulation, existing methods cannot achieve efficient joint design of complex systems such as smartphones and drones. In this work, starting from the perspective of the optical design, we characterize the optics with separated aberrations. Additionally, to bridge the hardware and software without gradients, an image simulation system is presented to reproduce the genuine imaging procedure of lenses with large field-of-views. As for aberration correction, we propose a network to perceive and correct the spatially varying aberrations and validate its superiority over state-of-the-art methods. Comprehensive experiments reveal that the preference for correcting separated aberrations in joint design is as follows: longitudinal chromatic aberration, lateral chromatic aberration, spherical aberration, field curvature, and coma, with astigmatism coming last. Drawing from the preference, a 10% reduction in the total track length of the consumer-level mobile phone lens module is accomplished. Moreover, this procedure spares more space for manufacturing deviations, realizing extreme-quality enhancement of computational photography. The optimization paradigm provides innovative insight into the practical joint design of sophisticated optical systems and post-processing algorithms.
@article{ZHOU2024108220, title = {Revealing the preference for correcting separated aberrations in joint optic-image design}, journal = {Optics and Lasers in Engineering}, volume = {178}, pages = {108220}, year = {2024}, issn = {0143-8166}, doi = {https://doi.org/10.1016/j.optlaseng.2024.108220}, url = {https://www.sciencedirect.com/science/article/pii/S0143816624001994}, author = {Zhou, Jingwen and Chen*, Shiqi and Ren, Zheng and Zhang, Wenguan and Yan, Jiapu and Feng, Huajun and Li, Qi and Chen, Yueting}, keywords = {Joint optic-image design, Aberration correction preference, Manufacturing deviations, Imaging simulation, Computational photography}, }

2023

Computational Optics for Mobile Terminals in Mass Production

Shiqi Chen, Ting Lin, Huajun Feng, and 3 more authors

IEEE Transactions on Pattern Analysis and Machine Intelligence, Apr 2023

Abs Bib PDF

Correcting the optical aberrations and the manufacturing deviations of cameras is a challenging task. Due to the limitation on volume and the demand for mass production, existing mobile terminals cannot rectify optical degradation. In this work, we systematically construct the perturbed lens system model to illustrate the relationship between the deviated system parameters and the spatial frequency response (SFR) measured from photographs. To further address this issue, an optimization framework is proposed based on this model to build proxy cameras from the machining samples’ SFRs. Engaging with the proxy cameras, we synthetic data pairs, which encode the optical aberrations and the random manufacturing biases, for training the learning-based algorithms. In correcting aberration, although promising results have been shown recently with convolutional neural networks, they are hard to generalize to stochastic machining biases. Therefore, we propose a dilated Omni-dimensional dynamic convolution (DOConv) and implement it in post-processing to account for the manufacturing degradation. Extensive experiments which evaluate multiple samples of two representative devices demonstrate that the proposed optimization framework accurately constructs the proxy camera. And the dynamic processing model is well-adapted to manufacturing deviations of different cameras, realizing perfect computational photography. The evaluation shows that the proposed method bridges the gap between optical design, system machining, and post-processing pipeline, shedding light on the joint of image signal reception (lens and sensor) and image signal processing (ISP).
@article{Chen_2022_TPAMI, author = {Chen, Shiqi and Lin, Ting and Feng, Huajun and Xu, Zhihai and Li, Qi and Chen, Yueting}, journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence}, title = {Computational Optics for Mobile Terminals in Mass Production}, month = apr, year = {2023}, volume = {45}, number = {4}, pages = {4245-4259}, }