A scalable pipeline for transforming large, noisy image datasets into high-quality training datasets using modern vision embeddings and vector similarity search.
This pipeline processes millions of images to remove duplicates and near-duplicates, then selects a diverse subset of images suitable for model training.
Beyond filtering and sampling, it also supports targeted image retrieval: users can retrieve images semantically similar to a candidate image or folder, which is useful for object-specific datasets, environment coverage, and failure-case mining.
Large image datasets often contain near-duplicates, poor-quality frames, redundant samples, and class imbalance.
In robotics-style capture pipelines, millions of frames can be collected while only a fraction are training-worthy. Unfiltered training data wastes compute and can increase overfitting risk.
Manual curation does not scale when datasets reach hundreds of thousands or millions of images, so an automated and cloud-ready pipeline is essential.
The core flow is: feature extraction with DINOv2 embeddings, duplicate filtering, diversity sampling, and semantic retrieval.
The output can either be a curated training dataset or a targeted set of similar images for analysis and downstream model iteration.
Without indexing, duplicate checks become O(N) comparisons per image. With HNSW-based vector indexing, nearest-neighbor lookup is approximately O(log N).
Using pgvector directly in Postgres keeps metadata and embeddings in one system, simplifies infrastructure, and is often more cost-effective than hosting a separate vector database.
Embedding extraction is the most expensive stage and is optimized with batch GPU inference.
Diversity sampling with k-center style selection introduces memory pressure because filtered embeddings must be loaded into memory. Future improvements include candidate-pool caps and reduced precision where possible.