Two Robotics Data Case Studies That Challenge Conventional Thinking

What if you could improve physical AI performance without massive data augmentation?

Improving AI performance typically means adding more training data or refining the model. But what if there's another way?

• We cut one dataset by 98% and still retained 97.6% of its original performance.

• Adding just 8 data samples raised the success rate from 0% to 52%.

This post covers two cases. The first is an AI consulting project for Company L, which had requested data augmentation. The second is an AI robotics research project in which we diagnosed the data distribution, trimmed overrepresented regions, and filled in underrepresented ones.

Robotics Data Case Study 1: What Should You Check Before Augmenting AI Training Data?

Company L ran an AI system that recommended corrective actions for faulty equipment. However, the system struggled to accurately identify less common cases. Company L assumed that filling in the missing data would improve performance, so it requested data augmentation.

Company L's dataset contained approximately 8,000 samples, with nearly 5,300 features and 125 label classes. The data followed a long-tail distribution, with sample counts varying widely across labels. We also found duplicate records and data errors.

Company L's Data Composition and Issues

Pebblous first removed the duplicates and fixed the data errors. We then applied Feature Diet to cut the roughly 5,300 features by 46% and used an AI model suited to the long-tail distribution.

As a result, we achieved 83.2% Top-3 accuracy without any data synthesis. The number of classes with accuracy of 50% or higher also rose to 35.

Where Should You Start When Diagnosing Data Quality Issues?

To diagnose data issues more systematically, Pebblous applied ISO/IEC 5259-2, a quality standard for AI training data. The standard defines 23 quality characteristics and 66 detailed measures for assessing the quality of AI training data.

The figure below shows how ISO/IEC 5259 organizes its quality metrics.

Applying ISO/IEC 5259-2 to Real-World Datasets

When applying AI training data quality criteria to real-world data, you need to consider both the data type and its intended use. Because data differs across companies and projects, you also need to define specifically which quality criteria to measure and how.

Take label accuracy as an example.

• Image classification: The label to check is the class assigned to each image.

• Object detection: You need to check every object and its bounding box within each image.

Even under the same “label accuracy” criterion, what counts as a single label depends on the type of data. As a result, the way you measure it can differ as well.

• For object detection data, you need to check bounding boxes and annotations.

• Image classification may not require these checks at all.

• For structured data, you also need to check for missing values.

In other words, the same quality criterion may need to be applied differently depending on the data. Pebblous defined exactly what to check and how to calculate it based on the actual data. We also designed a way to identify which data needs to be re-examined when an issue is found.

Robotics Data Case Study 2: Where Should You Add Data to Improve Robot Performance?

Even large robot training datasets can be skewed toward certain areas. So rather than simply adding more data, you should first identify which areas are underrepresented.

Pebblous analyzed the data distribution to pinpoint areas that needed reinforcement, then measured how performance changed when data was added to those areas.

Data Diet: How Much Can You Cut Before Performance Drops?

We started with a preliminary experiment on the CIFAR-100 dataset. We compared 24 data selection methods across 9 data retention ratios, running each condition 3 times for a total of 648 runs.

The results showed that how much data was retained had a greater impact on performance than the selection method used. At retention rates of 30% or higher, the differences between selection methods were small.

However, reducing the data too aggressively caused some classes to disappear entirely. To address this, we applied a per-class minimum retention threshold to Graph Cut, which reduced performance degradation even at high reduction rates.

Up to 98% Less Data: How Much Performance Was Retained Across 18 Datasets? 

We applied Data Diet to 18 datasets. Beyond standard benchmarks, these included real-world industrial data such as wildfire monitoring, medical imaging, and industrial waste.

Reductions ranged from 21% to 98%, while performance retention ranged from 89.3% to 100% of the original.

• The largest reduction came from the wildfire monitoring data. We cut 98% of the dataset and still retained 97.6% of the original performance.

• For the industrial waste data, we cut 83% and retained 94.6% of performance. For the casting defect data, we retained 99.4% even after cutting 56%.

• Most of the 18 datasets retained more than 92% of their original performance. Even the lowest, the Korean food dataset, retained 89.3%.

For industrial data, the reduction rate and performance retention varied by dataset. The table below compares the key results.

Data Bulk-Up: Where and How Much Data Should You Add?

While some regions of a dataset are overrepresented, others may lack sufficient coverage. Pebblous used the density and coverage analysis from Data Diet to identify areas where data was sparse or missing. This was the preparatory step for generating synthetic data for AI robots.

• Generating synthetic data for robots comes with a unique requirement: for a robot to learn, the data must reflect situations in which the robot can actually move.

• In other words, images created with generative AI may not fully account for the laws of physics. The results can include scenes where objects overlap or where a robot physically cannot move.

That's why Pebblous used PebbloSim, a synthetic data generator that fills underrepresented regions of a dataset using a simulator that faithfully models the laws of physics.

Filling Data Gaps with Physics-Based Validation

Here's how we put PebbloSim to work. We generated 420 scenes across four robot manipulation datasets and one autonomous driving dataset.

Every generated scene went through physics checks. We checked whether robot movements were physically feasible and screened for unintended collisions or object overlaps. We also filtered out scenes where the robot left its workspace or a vehicle drifted out of its lane.

Of the 420 scenes, 409 passed the checks, a 97% pass rate. On LIBERO, data coverage rose from 14.3% to 44.6%, and all 200 added scenes passed the physics checks. Comparing the distributions before and after gap filling alongside the actual generated scenes makes the change in coverage easy to see.

(Insert actual video here)

For autonomous driving, we also generated scenes that didn't exist in the real-world data. We created and validated 34 rare or high-risk scenes, such as high-speed driving and heavy traffic.

Can Filling Data Gaps Improve Robot Performance?

The scenes PebbloSim generated passed the physics checks. But does training on this data actually improve robot performance?

Pebblous ran two robot manipulation experiments using a VLA (Vision-Language-Action) model. We used a pretrained 7B VLA model, fine-tuned it once per condition, and ran 50 test trials for each.

• Before: The first experiment was Cube Picking. The original 200 training samples were clustered in the center. When the robot had to pick up a cube near the edge, its success rate was 0%.

• After: Adding just 8 samples near the edge raised the success rate to 52%. Adding the same amount of data to the center produced no meaningful change beyond the margin of error.

In Block Stacking, Results Also Depended on Where Data Was Added

We also compared edge performance in Block Stacking. The original 200 training samples were also concentrated in the center, and the success rate at the edge was 8%.

Adding 64 samples near the edge raised the success rate to 50%. Once again, adding the same amount to the center produced no meaningful change beyond the margin of error.

Data Diagnose Your Data Before Reducing or Augmenting It

In the earlier Company L case, the data appeared to need augmentation. In reality, the duplicates, data errors, and unnecessary features had to be cleaned up first. That alone delivered 83.2% Top-3 accuracy without any data synthesis.

The robot case showed the opposite: performance improved when we identified underrepresented regions and filled them. In the Cube Picking experiment, adding 8 samples near the edge raised the success rate from 0% to 52%.

The two cases called for different solutions. That's why, rather than blindly adding or removing data, you should first diagnose the quality and distribution of your current data. Only then can you decide where to trim and where to fill.

Get Test Reports Based on ISO/IEC 5259-2

DataClinic and DataGreenhouse, Pebblous's data quality management solutions, can automatically generate test reports based on ISO/IEC 5259-2.

Planning to expand into global markets? Pebblous helps AI and robotics companies document their data quality assessments in accordance with international standards, providing valuable supporting evidence for global business opportunities. 

We document data quality results in accordance with international standards, so AI companies entering global markets can use them as supporting evidence.

If Your AI Isn't Performing as Expected, Start With Your Data

DataClinic analyzes the quality and distribution of your data to identify issues that affect AI performance. Beyond insufficient data, it also checks for duplicates and gaps in specific areas.

If you're considering collecting more data or applying data augmentation, first find out what issues exist in your current data.