BannerField

AI-Enabled Sensing

AI-enabled sensing serves as the entry point for the GEMINI pipeline. Traditional phenotyping relies heavily on manual measurements, which limits both scale and consistency. By combining field imaging technologies with machine learning, GEMINI replaces these manual workflows with an automated, high-throughput pipeline that transforms raw imagery into quantitative traits: calibrating crop models, informing genetic analyses, and powering downstream predictions.

Graphic showing horn speakers, studio racks, and a smartphone above a cost–fidelity scale

Data collection begins in the field using a combination of aerial and ground-based imaging systems, including drones, rover platforms, and handheld devices. These platforms capture complementary views of crop structure and development across spatial scales, ranging from broad canopy-level patterns down to individual plant features.

Farm workers harvesting red peppers in dense rows, trucks, white tarp and orange cones behind
Person facing away in white shirt in a green field at sunset holding a phone on a long pole; colored flags
Heesup Yun kneeling in crop field, adjusting a multirotor drone
Two people wearing hats inspect tall bunch of sorghum in a field under a clear blue sky

GEMINI App

Field imagery is processed through the GEMINI App, an open-source platform that provides a standardized workflow for data management, preprocessing, model deployment, and trait extraction. Supported by collaborative development across the project network, the app moves data efficiently from raw capture to analysis.

Graphic diagram of GEMINI sensing pipeline: sensors → data → processing → training cloud

Within this pipeline, machine learning models quantify a wide range of phenotypic traits from imagery:

  • Structural traits: Plant density and canopy cover
  • Developmental traits: Flowering timing and intensity
  • Complex field tasks: Separating crops from weeds and identifying reproductive structures under variable field conditions

Because imagery is collected repeatedly across the growing season, the pipeline captures crop development as a dynamic, time-series process rather than a single snapshot. Beyond visible features, GEMINI derives traits reflecting underlying biological processes - for instance, tracking changes in canopy color and structure over time to estimate physiological variables like leaf nitrogen content, the onset of flowering, and resource redistribution.

Learn more about the GEMINI app here.

Real to Sim

Training reliable AI models typically requires massive amounts of manually labeled images, while ensuring those models perform consistently across diverse locations and weather conditions remains a central challenge. To solve both problems, GEMINI uses 3D crop simulations powered by the Helios 3D crop modeling framework to generate photorealistic synthetic training datasets with automatically labeled features. Recent work shows that models trained on synthetic data, especially when fine-tuned with a small number of real labeled examples, generalize effectively to real-world imagery. While current models show strong performance within individual environments and promising transferability across genotypes, expanding these synthetic datasets is a primary strategy for improving model robustness across entirely new environments and imaging systems without extensive human annotation.

To support the parameterization and validation of Helios, the primary goal is to fit the Helios procedural plant model to field-collected images, making the simulation as close to reality and as biologically accurate as possible. Most field-collected images are top-view, with a limited number of side-view images. The goal is to fit the plant models to this limited visual information using vision foundation models, differentiable plant renderer optimization, procedural plant modeling constraints, and organ priors.

Side-by-side aerial photos: three vertical green shrub rows with soil paths
Side-by-side top-down photos of soil with green plants; right panel denser and aligned

Phase II: Scaling Field Phenotyping for Crop Prediction

In GEMINI Phase II, the AI-enabled sensing team is scaling the Phase I platform to collect high-resolution measurements across key breeding populations and diverse growing environments. Combining ground-based systems with low-altitude drone flights, the team uses RGB and thermal imagery alongside environmental sensors to track how plants respond to changing conditions throughout the season.

Key priorities for Phase II include:

  • High-frequency tracking: Extracting agronomic traits, such as plant height, canopy development, flowering dynamics, and maturity, with intensified sampling during critical stages like flowering to capture subtle genetic differences
  • Cross-site validation: Evaluating AI trait extraction across multiple field sites using targeted ground-truth measurements to ensure reliability
  • 3D model calibration: Generating reliable field measurements to parameterize and validate the Helios 3D crop modeling framework, providing the data needed to predict crop performance in environments where varieties have not yet been tested