Integrating pre-processing pipelines in ODC based framework
Using on-demand processing pipelines to generate virtual geospatial products is beneficial to optimizing resource management and decreasing processing requirements and data storage space. Additionally, pre-processed products improve data quality for data-driven analytical algorithms, such as machine learning or deep learning models. This paper proposes a method to integrate virtual products based on integrating open-source processing pipelines. In order to validate and evaluate the functioning of this approach, we have integrated it into a geo-imagery management framework based on Open Data Cube (ODC). To validate the methodology, we have performed three experiments developing on-demand processing pipelines using multi-sensor remote sensing data, for instance, Sentinel-1 and Sentinel-2. These pipelines are integrated using open-source processing frameworks.
Code (0)
등록된 구현이 없습니다.
Tasks
ManagementSimilar Papers 제목 키워드 기반
AutoStreamPipe: LLM Assisted Automatic Generation of Data Stream Processing Pipelines
Data pipelines are essential in stream processing as they enable the efficient collection, processing, and delivery of real-time data, supporting rapid data analysis. In this paper, we present AutoStreamPipe, a novel fra…
GPUs, CPUs, and... NICs: Rethinking the Network's Role in Serving Complex AI Pipelines
The increasing prominence of AI necessitates the deployment of inference platforms for efficient and effective management of AI pipelines and compute resources. As these pipelines grow in complexity, the demand for distr…
ManagementCompendium Manager: a tool for coordination of workflow management instances for bulk data processing in Python
Compendium Manager is a command-line tool written in Python to automate the provisioning, launch, and evaluation of bioinformatics pipelines. Although workflow management tools such as Snakemake and Nextflow enable users…
ManagementIntegration of Workflow and Pipeline for Language Service Composition
Integrating language resources and language services is a critical part of building natural language processing applications. Service workflow and processing pipeline are two approaches for sharing and combining language…
Service CompositionRASL: Relational Algebra in Scikit-Learn Pipelines
Integrating data preparation with machine-learning (ML) pipelines has been a long- standing challenge. Prior work tried to solve it by building new data processing platforms such as MapReduce or Spark, and then implement…
BIG-bench Machine Learning