ESTemd: A Distributed Processing Framework for Environmental Monitoring based on Apache Kafka Streaming Engine
Distributed networks and real-time systems are becoming the most important components for the new computer age, the Internet of Things (IoT), with huge data streams or data sets generated from sensors and data generated from existing legacy systems. The data generated offers the ability to measure, infer and understand environmental indicators, from delicate ecologies and natural resources to urban environments. This can be achieved through the analysis of the heterogeneous data sources (structured and unstructured). In this paper, we propose a distributed framework Event STream Processing Engine for Environmental Monitoring Domain (ESTemd) for the application of stream processing on heterogeneous environmental data. Our work in this area demonstrates the useful role big data techniques can play in an environmental decision support system, early warning and forecasting systems. The proposed framework addresses the challenges of data heterogeneity from heterogeneous systems and real time processing of huge environmental datasets through a publish/subscribe method via a unified data pipeline with the application of Apache Kafka for real time analytics.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Enhancing Air Quality Monitoring: A Brief Review of Federated Learning Advances
Monitoring air quality and environmental conditions is crucial for public health and effective urban planning. Current environmental monitoring approaches often rely on centralized data collection and processing, which p…
Federated LearningDistributed Dynamic Invariant Causal Prediction in Environmental Time Series
The extraction of invariant causal relationships from time series data with environmental attributes is critical for robust decision-making in domains such as climate science and environmental monitoring. However, existi…
Weather ForecastingCausal InferenceMonitoring exposure-length variations in submarine power cables using distributed fiber-optic sensing
This study proposes an anomaly-detection framework for monitoring exposure-length variations in submarine free-span cables using Distributed Acoustic Sensing (DAS), which is one of the distributed fiber-optic sensing tec…
Binary ClassificationAnomaly DetectionUncertainty Aware Deep Neural Network for Multistatic Localization with Application to Ultrasonic Structural Health Monitoring
Guided ultrasonic wave localization uses spatially distributed multistatic sensor arrays and generalized beamforming strategies to detect and locate damage across a structure. The propagation channel is often very comple…
Structural Health MonitoringAdvanced Distributed Submarine Cable Monitoring and Environmental Sensing using Constant Power Probe Signals and Coherent Detection
In this work we demonstrate an FPGA-based coherent optical frequency domain reflectometry setup for cable monitoring. Using coherent detection for averaging and narrowband filtering, we significantly improve the signal-t…