Bridging the Simulation-to-Reality Gap in Electron Microscope Calibration via VAE-EM Estimation
Electron microscopy has enabled many scientific breakthroughs across multiple fields. A key challenge is the tuning of microscope parameters based on images to overcome optical aberrations that deteriorate image quality. This calibration problem is challenging due to the high-dimensional and noisy nature of the diagnostic images, and the fact that optimal parameters cannot be identified from a single image. We tackle the calibration problem for Scanning Transmission Electron Microscopes (STEM) by employing variational autoencoders (VAEs), trained on simulated data, to learn low-dimensional representations of images, whereas most existing methods extract only scalar values. We then simultaneously estimate the model that maps calibration parameters to encoded representations and the optimal calibration parameters using an expectation maximization (EM) approach. This joint estimation explicitly addresses the simulation-to-reality gap inherent in data-driven methods that train on simulated data from a digital twin. We leverage the known symmetry property of the optical system to establish global identifiability of the joint estimation problem, ensuring that a unique optimum exists. We demonstrate that our approach is substantially faster and more consistent than existing methods on a real STEM, achieving a 2x reduction in estimation error while requiring fewer observations. This represents a notable advance in automated STEM calibration and demonstrates the potential of VAEs for information compression in images. Beyond microscopy, the VAE-EM framework applies to inverse problems where simulated training data introduces a reality gap and where non-injective mappings would otherwise prevent unique solutions.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Bridging the reality gap in quantum devices with physics-aware machine learning
The discrepancies between reality and simulation impede the optimisation and scalability of solid-state quantum devices. Disorder induced by the unpredictable distribution of material defects is one of the major contribu…
Bayesian InferenceBIG-bench Machine LearningAttention-Based Synthetic Data Generation for Calibration-Enhanced Survival Analysis: A Case Study for Chronic Kidney Disease Using Electronic Health Records
Access to real-world healthcare data is limited by stringent privacy regulations and data imbalances, hindering advancements in research and clinical applications. Synthetic data presents a promising solution, yet existi…
Survival AnalysisSynthetic Data Generation2D SEM images turn into 3D object models
The scanning electron microscopy (SEM) is probably one the most fascinating examination approach that has been used since more than two decades to detailed inspection of micro scale objects. Most of the scanning electron…
ObjectSurface ReconstructionTowards Closing the Sim-to-Real Gap in Collaborative Multi-Robot Deep Reinforcement Learning
Current research directions in deep reinforcement learning include bridging the simulation-reality gap, improving sample efficiency of experiences in distributed multi-agent reinforcement learning, together with the deve…
Deep Reinforcement LearningMulti-agent Reinforcement Learningreinforcement-learningReinforcement Learning+1Towards Augmented Microscopy with Reinforcement Learning-Enhanced Workflows
Here, we report a case study implementation of reinforcement learning (RL) to automate operations in the scanning transmission electron microscopy (STEM) workflow. To do so, we design a virtual, prototypical RL environme…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)