Artificial intelligence optical hardware empowers high-resolution hyperspectral video understanding at 1.2 Tb/s
Foundation models, exemplified by GPT technology, are discovering new horizons in artificial intelligence by executing tasks beyond their designers' expectations. While the present generation provides fundamental advances in understanding language and images, the next frontier is video comprehension. Progress in this area must overcome the 1 Tb/s data rate demanded to grasp real-time multidimensional video information. This speed limit lies well beyond the capabilities of the existing generation of hardware, imposing a roadblock to further advances. This work introduces a hardware-accelerated integrated optoelectronic platform for multidimensional video understanding in real-time. The technology platform combines artificial intelligence hardware, processing information optically, with state-of-the-art machine vision networks, resulting in a data processing speed of 1.2 Tb/s with hundreds of frequency bands and megapixel spatial resolution at video rates. Such performance, validated in the AI tasks of video semantic segmentation and object understanding in indoor and aerial applications, surpasses the speed of the closest technologies with similar spectral resolution by three to four orders of magnitude. This platform opens up new avenues for research in real-time AI video understanding of multidimensional visual information, helping the empowerment of future human-machine interactions and cognitive processing developments.
Code (0)
등록된 구현이 없습니다.
Tasks
Semantic SegmentationVideo Semantic SegmentationVideo UnderstandingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Programmable metasurfaces for future photonic artificial intelligence
Photonic neural networks (PNNs), which share the inherent benefits of photonic systems, such as high parallelism and low power consumption, could challenge traditional digital neural networks in terms of energy efficienc…
Transfer LearningPhotonic reservoir computing enabled by stimulated Brillouin scattering
Artificial Intelligence (AI) drives the creation of future technologies that disrupt the way humans live and work, creating new solutions that change the way we approach tasks and activities, but it requires a lot of dat…
A Hardware-Efficient Photonic Tensor Core: Accelerating Deep Neural Networks with Structured Compression
Recent advancements in artificial intelligence (AI) and deep neural networks (DNNs) have revolutionized numerous fields, enabling complex tasks by extracting intricate features from large datasets. However, the exponenti…
Optical Computing for Deep Neural Network Acceleration: Foundations, Recent Developments, and Emerging Directions
Emerging artificial intelligence applications across the domains of computer vision, natural language processing, graph processing, and sequence prediction increasingly rely on deep neural networks (DNNs). These DNNs req…
Generating retinal flow maps from structural optical coherence tomography with artificial intelligence
Despite significant advances in artificial intelligence (AI) for computer vision, its application in medical imaging has been limited by the burden and limits of expert-generated labels. We used images from optical coher…