Not All Datasets Are Equal
The hardest part of building physical AI isn’t the algorithms—it’s the data. High-quality, high-coverage datasets are almost impossible to acquire unless you’re willing to deploy immense sensor fleets across the globe. Even when these datasets exist, their flaws are often invisible until they show up as failures in real-world deployment. Their quality often varies dramatically—a detail that is difficult to assess from a surface-level evaluation. Poorly designed datasets don’t just make your performance metrics look bad—they set your system up to fail when it matters most.
Why Does It Matter?
We’ve seen firsthand how poorly designed datasets—with bad ontologies, limited diversity, or inconsistent labeling—can cause significant issues in real-world deployment and result in mishaps in production. By sharing this list of top open-source maritime datasets, we aim to provide a framework for evaluating and selecting datasets that meet the highest standards of quality.
Whether you're building a USV perception system or simply exploring maritime AI, this list will serve as a resource to help you find the best data for your needs.
How Do We Measure Dataset Quality?
At Bifrost, we employ a suite of tools and methodologies to rigorously quantify dataset quality. These include UMAP-based cluster analysis, which visualizes high-dimensional data distributions to detect redundancies, over-representations and gaps, ensuring datasets are well-balanced and representative. Our scenario coverage assessments systematically verify the inclusion of diverse operational contexts and edge cases, while distribution evaluations of object classes and environmental conditions ensure sufficient representation across all relevant operational conditions.
These analyses feed into a simplified scoring framework across three critical dimensions: ontology, diversity and label quality.
1. Ontology Quality
Ontology refers to how the authors have chosen to group specific maritime objects into classes. This is one of the trickiest steps to get right - we’ve personally seen poor ontology lead to a great deal of failure cases in the field. One of the primary reasons for this is the significant variation in object ontologies and inconsistent definitions across authors, regions, and maritime subject matter experts.
Good ontologies classify visually and dimensionally similar objects into distinct categories, covering the majority of objects the system is likely to encounter during deployment. For instance, distinguishing between fishing boats, container ships, and speedboats based on visual characteristics enables more precise learning of class boundaries, rather than lumping them all into a single, overly broad category like “boat.”
Poor ontologies often rely on overly broad or semantic-based groupings, such as classifying vessels as “recreational boats.” This approach lumps together vastly different shapes and sizes, from large cruise ships to small sailboats and mid-sized yachts, ignoring the crucial visual features that distinguish these classes. Such oversights compromise the dataset’s ability to train models effectively.
2. Diversity Coverage
A diverse and high-coverage dataset captures a high percentage of all possible operational conditions, ensuring that a physical AI system is equipped to handle the complexity and variability of real-world deployment. This includes:
Objects: A variety of vessel and buoy types, sizes, paint schemes, and configurations.
Scenarios: Variations in the spatial arrangement of ships and objects, including densely packed harbors, open sea formations, near-collision scenarios, and vessels partially obscured by other objects or environmental factors.
Environmental Conditions: Scenarios like fog, twilight operations, storms, high sea states, and diverse lighting conditions.
Sensor Conditions: Perturbations such as glare, motion blur, sea spray, and out-of-focus effects.
Diversity is the single most important factor in ensuring a model can generalize effectively to real-world scenarios—a crucial capability for the success of USV systems.
3. Label Quality
Label quality refers to the accuracy and consistency of annotations. Inconsistent, noisy, or inaccurate annotations, such as mislabeled classes or poorly drawn bounding boxes, can lead to significant performance degradation in deployed systems. We use both automated consistency checks and manual audits to verify precision and accuracy of real world data.
High-quality labels provide tight bounding boxes, precise segmentation masks, and consistent annotations across the dataset. Occluded parts of objects are labeled, and distant objects are annotated only if their resolution supports accurate identification.
Low-quality labels introduce noise and inconsistency, such as poorly drawn bounding boxes, mislabeled classes, or missing annotations. These issues can sabotage both training and evaluation.
We hope these categories outline a clearer framework for evaluating maritime datasets, ultimately enhancing their utility for maritime perception development.
1. WaterScenes Dataset
Task : USV Perception, Obstacle Detection and Sensor Anomaly Detection
Ontology Quality : 4/5
Data Diversity : 5/5
Label Quality : 4/5
Total Score : 13/15
WaterScenes is especially useful for training and evaluating models in tough conditions. We love that they’ve already tagged these images so it’s easy to retrieve images with a specific condition. Overall high quality dataset, with quite a few different label types.
Use Case: Can be applied to tasks such as object detection, instance segmentation, semantic segmentation, free-space segmentation, and waterline segmentation.
Contents: 54,120 sets of RGB images, radar point clouds, GPS and IMU data, covering over 200,000 objects.
Annotations: 2D box-level and pixel-level annotations for camera images, and 3D point-level annotations for radar point clouds.
Author: Yao Shanliang, Guan Runwei, Wu Zhaodong, Ni Yi, Huang Zile, Ryan Wen Liu, Yue Yong, Ding Weiping, Lim Eng Gee, Seo Hyungjoon, Man Ka Lok, Ma Jieming, Zhu Xiaohui, and Yue Yutao
Source: [WaterScenes Dataset](https://github.com/WaterScenes/WaterScenes), 2024
Publishing Institution: IEEE Transactions on Intelligent Transportation Systems
2. LARS Dataset
Task : USV Perception and Obstacle Detection
Ontology Quality : 4/5
Data Diversity : 4/5
Label Quality : 4/5
Total Score : 12/15
LARS is a maritime panoptic obstacle detection benchmark designed for real-time marine object detection, featuring scenes from lakes, rivers and seas. It boasts the largest diversity in recording locations, scene types, obstacle classes, and acquisition conditions among the related datasets.
Use Case: Suitable for maritime traffic monitoring via panoptic segmentation.
Contents: 4000+ per-pixel labeled key frames (with 9 preceding frames, total 40k+ frames) categorized into 3 stuff classes and 8 thing (dynamic obstacle) categories, with 20 scene-level attributes.
Annotations:Panoptic annotations of 3 stuff and 8 thing classes, and semantic annotations that are pixel-wise labels of 3 classes
Author: Lojze Zust, Janez Perš, Matej Kristan
Source: [LARS Dataset](https://lojzezust.github.io/lars-dataset/#download), 2023
Publishing Institution: International Conference on Computer Vision
3. MODD2 (Marine Object Detection Dataset)
Task : USV Perception and Obstacle Detection
Ontology Quality : 3/5
Data Diversity : 3/5
Label Quality : 4/5
Total Score : 10/15
A large and challenging multi-modal marine obstacle detection dataset captured by a real USV. The USV was manually guided and simulated realistic navigation scenarios in which an obstacle may present a danger to the USV, allowing it to capture diverse weather conditions, extreme situations, and various small obstacles.
Use Case: Useful for training and evaluating models for obstacle detection and segmentation in marine environments.
Contents: 28 video sequences of variable length, totalling 11675 stereo-frames at 1278x958 pixels resolution. Dataset contains frames across different times of day and various weather conditions, all time-synchronized with measurements of on-board sensors: IMU, GPS and compass.
Annotations: Bounding boxes for large and small obstacles, polygon annotation for water edges.
Author: Borja Bovcon, Jon Muhovič, Janez Perš, Matej Kristan
Source: MODD2, 2018
Publishing Institution: Robotics and Autonomous Systems
4. MASTR1325 Dataset
Task : USV Perception and Obstacle Detection
Ontology Quality : 4/5
Data Diversity : 3/5
Label Quality : 3/5
Total Score : 10/15
MASTR1325 is a large-scale marine dataset focused on semantic segmentation, tailored for development of obstacle detection methods in small-sized coastal USVs. Each image is labeled for various marine objects, critical for navigation systems. Do note however that the images in this dataset are segmentation only, no bounding boxes.
Use Case: Useful for training maritime object detection algorithms for small-sized coastal USVs.
Contents: 1325 high-quality, manually annotated images covering a range of realistic conditions encountered in coastal surveillance tasks, as well as obstacles such as cargo ships, sailboats and buoys. The images are time-synchronized with measurements of the on-board GPS and IMU.
Annotations:Semantic segmentation into 3 categories (sea, sky, environment) by expert in-house annotators.
Author: Borja Bovcon, Jon Muhovič, Janez Perš, Matej Kristen
Source: MASTR1325, 2019
Publishing Institution: IEEE International Conference on Intelligent Robots and Systems
5. SPSCD Maritime Dataset
Task : Port Monitoring, Ship Classification
Ontology Quality : 3/5
Data Diversity : 4/5
Label Quality : 3/5
Total Score : 10/15
The SPSCD dataset was built as a reference dataset for video surveillance and ship classification in real maritime zones. The dataset allows for estimating detection and classification performance, which provides versatile ship annotations and classifications for passenger ports with a large number of small-to-medium-sized ships that were not monitored by the automatic identification system AIS and/or the vessel traffic system (VTS).
Use Case: Ship detection, classification, port and port property surveillance.
Contents: 19,337 high-resolution images with 27,849 manually labeled ship instances classified into 12 categories such as sailing boat, small passenger ship, large ferry, etc.across different weather conditions, illumination effects from the sun and sea surface reflections, and different sea state conditions.
Annotations: Manually annotated bounding boxes that have been normalized to make it easier to work with after scaling or stretching the images.
Author: Petković Miro, Vujović Igor, Lušić Zvonimir, Šoda Joško
Source: [Maritime Dataset](https://labs.pfst.hr/maritime-dataset/), 2023
Publishing Institution: Journal of Marine Science and Engineering
6. KOLOMVERSE
Task - USV Perception and Object Detection
Ontology Quality : 2/5
Data Diversity : 4/5
Label Quality : 4/5
Total Score : 10/15
This dataset contains 100k+ 4K images with variations in illumination, viewpoint, occlusion, background, scale and proportion and is available by request. However the ontology is weak (all vessels are considered “boat”), but at least there’s rare wind farm and lighthouse labels if they’re objects you need to detect.
Use Case: Ideal for maritime object detection under diverse visual sea conditions.
Contents: 186,419 images, captured across 21 territorial waters of South Korea, categorized into five classes (ship, buoy, fishnet buoy, lighthouse, wind farm).
Annotations: Bounding boxes across the five different object classes.
Author: Abhilasha Nanda, Sung Won Cho, Hyeopwoo Lee, Jin Hyoung Park [Maritime Safety and Environmental Research Division, Korea Research Institute of Ships and Ocean Engineering (KRISO)]
Source: KOLOMVERSE Dataset, 2022
Publishing Institution: arXiv preprint
7. Pohang Canal Dataset
Task - Multi-Modal USV Perception, Navigation and Object Detection
Ontology Quality : 3/5
Data Diversity : 2/5
Label Quality : 5/5
Total Score : 10/15
A multimodal maritime dataset for autonomous ship navigation in restricted waters, acquired in restricted waters in Pohang, South Korea. The dataset focuses on small boats and vessels in canal environments, under varied weather and lighting conditions. High quality, clean dataset that has LiDAR but not many obstacles - useful for navigation purposes.
Use Case: Ideal for autonomous canal navigation and vessel detection.
Contents: 6 scenarios (2 acquired at night, 4 acquired at daytime) of a 7.5km long route, including a pier, narrow canal, inner and outer port, and near-coastal areas, across various weather and visual conditions.
Annotations: LiDAR, Radar, Infrared, Stereo and omnidirectional cameras.
Author: Dongha Chung, Jonghwi Kim, Changyu Lee, and Jinwhan Kim [Department of Mechanical Engineering, Korea Advanced Institute of Science and Technology (KAIST)]
Source: [Pohang Canal Dataset](https://sites.google.com/view/pohang-canal-dataset/home), 2023
Publishing Institution: The International Journal of Robotics Research
8. Singapore Maritime Dataset (SMD)
Task - Port Monitoring, USV Perception and Object Detection
Ontology Quality : 3/5
Data Diversity : 2/5
Label Quality : 4/5
Total Score : 9/15
Everyone’s favorite SMD is the ImageNet of maritime object detection - good for its time, dated, but everyone still uses it. Good for general object detection with decent classes and good label quality (there’s even an additional set of cleaned-up, third party labels, see SMD Plus). Lacking in diversity - only has clear, hazy and evening conditions in relatively calm waters. We recommend using SMD as baseline evaluation data for open ocean testing instead of training. Has updated labels created by a third party https://github.com/kjunhwa/SMD-Plus.
Use Case: Open ocean footage suitable for maritime surveillance: detection and tracking. Recommended for evaluation only.
Contents: 81 videos of annotated vessels split into on-shore, on-board, and near-infrared footage.
Annotations: Custom formatted bounding box labels that must be parsed into modern formats, unless you use the updated SMD-Plus labels.
Author: Dilip K. Prasad, Deepu Rajan, Lily Rachmawati, Eshan Rajabally, and Chai Quek
Source: Singapore Maritime Dataset (SMD), 2017
Publishing Institution: N.A.
9. SeaShips MCVWT Dataset
Task - Port Monitoring, USV Perception, Object Detection, Ship Classification
Ontology Quality : 3/5
Data Diversity : 2/5
Label Quality : 4/5
Total Score : 9/15
A dataset for detecting and classifying maritime vessels with around 7k maritime images containing 6 classes of vessels of varying sizes such as bulk cargo carriers, ore carriers, and fishing boats. Images lack diversity - only has clear, hazy conditions in relatively calm waters.
Use Case: Ideal for training object detection models focusing on maritime traffic and vessel tracking.
Contents: 6,979 maritime images at resolution of 1920x1080 across 6 classes of marine vessels (fishing boat, ore carrier, bulk cargo carrier, general cargo ship, container ship, passenger ship)
Annotations: Bounding boxes across 6 classes of marine vessels.
Author: YOLOv5Seaships
Source: [SeaShips MCVWT Dataset](https://universe.roboflow.com/yolov5seaships/seaships-mcvwt), 2022
Publishing Institution: Roboflow Universe
10. MassMIND Dataset
Task : USV Perception and Object Detection
Ontology Quality : 3/5
Data Diversity : 3/5
Label Quality : 3/5
Total Score : 9/15
An exhaustive dataset of real-life Long Wave Infrared (LWIR) images geared towards development of deep learning algorithms specifically for the Maritime domain. This dataset has seasonal and temporal diversity with various scene coverage and class distribution.
Use Case: Suitable for developing deep learning models in maritime object detection, especially for navigation and obstacle avoidance in autonomous marine systems.
Contents: 2,916 Long Wave Infrared (LWIR) images across 8 types of scene coverage such as night time, day time, foggy weather, rough waters. The dataset is also segmented into 7 different classes such as sky, water, bridge, obstacle.
Annotations: Ground truth semantic and ground truth instance segmentation labels.
Author: Shailesh Nirgudkar, Michael DeFilippo, Michael Sacarny, Michael Benjamin, Paul Robinette
Source: [MassMIND Dataset](https://github.com/uml-marine-robotics/MassMIND), 2023
Publishing Institution: The International Journal of Robotics Research
11. MUSSID Dataset
Task : Sea Horizon Line (SHL) Detection and Navigation
Ontology Quality : 3/5
Data Diversity : 3/5
Label Quality : 3/5
Total Score : 9/15
The MUSSID dataset serves to fill the gap for publicly available maritime image datasets that were developed under limited environments with slight-to-moderate variations in maritime features. This dataset accomplishes this by incorporating various geographical, seasonal, and maritime features. While the dataset offers a wider variety of features, its main use case is for sea horizon line (SHL) detection - a lot of empty ocean shots, unique angles, lots of glare - which would be good for training against glare and reflections if that’s what you’re looking for.
Use Case: Sea horizon line (SHL) detection, coastal surveillance and navigation
Contents: 2,673 high-definition (1920x1080 pixels) RGB images, offering 36 different features across day conditions, weather conditions, sea states, environmental conditions, false linear features, occlusion, presence of object, and artifacts.
Annotations: Images are annotated manually using SuperAnnotate to identify SHL and used as a reference for performance evaluation of an SHL detection algorithm.
Author: Manzoor Ahmed Hashmani, Muhammad Umair
Source: [MUSSID Dataset](https://www.kaggle.com/datasets/umairatwork/manzoorumair-sea-image-dataset-musid), 2022
Publishing Institution: Journal of Marine Science and Engineering
12. MARVEL 2016 Dataset
Task : Ship Classification
Ontology Quality : 2/5
Data Diversity : 2/5
Label Quality : 4/5
Total Score : 8/15
The Marvel dataset serves to help with ship identification and categorisation which makes it a very curated and easy dataset in the sense that there’s only one ship per image and they are fairly large ships. In terms of usefulness, it’s a good dataset to get started with but not really suitable and relevant for production.
Use Case: Vessel classification, verification, retrieval and recognition
Contents: Consists of 2 million user uploaded images and their attributes, categorized into 109 vessel type classes and 26 constructed superclasses.
Annotations: Ship vessels categorized and annotated using AlexNet
Author: Erhan Gundogdu, Berkan Solmaz, Veysel Yücesoy & Aykut Koç
Source: [MARVEL Dataset 2016](https://github.com/avaapm/marveldataset2016), 2016
Publishing Institution: Asian Conference on Computer Vision
13. MariShipSegHEU
Task : Ship Classification
Ontology Quality : 2/5
Data Diversity : 2/5
Label Quality : 4/5
Total Score : 8/15
The dataset mainly contains images used for ship classification and segmentation tasks, with a portion of its images from pre-existing public maritime datasets such as Singapore Maritime Dataset (SMD). As such, the images here are very curated and easy as well, similar to the MARVEL dataset. Would also recommend this as a good dataset to get started with, but not really suitable and relevant for production either.
Use Case: Best suited for ship segmentation and object tracking in AI based maritime monitoring.
Contents: 3560 images, of which, 10% are from Singapore Maritime Dataset (SMD), 25% are from the Maritime Detection, Classification, and Tracking Dataset (MarDCT), and the rest are from the internet.
Annotations: Manually labeled visible image dataset for ship classification and ship segmentation task.
Author: Wen Zhang, Xujie He, Wanyi Li, Zhi Zhang, Yongkang Luo, Li Su, Peng Wang
Source: [MariShipSegHEU](https://github.com/EddieEduardo/MariShipSeg-HEU/tree/master), 2020
Publishing Institution: Elsevier: Image and Vision Computing
14. VAIS Dataset
Task : Object Detection
Ontology Quality : 2/5
Data Diversity : 2/5
Label Quality : 2/5
Total Score : 6/15
The VAIS dataset contains 2014 trained images of maritime vessels. Vessels are annotated with bounding boxes but image quality is low - has only 1 large vessel per image and classifies all vessels as boats in general.
Use Case: Marine vessel identification.
Contents: 2014 maritime vessels train set images but only 1 class of categorization - boat.
Annotations: Bounding boxes
Author: Jerry Nyluw
Source: [VAIS Dataset](https://universe.roboflow.com/jerry-nyluw/vais-12/dataset/2), 2023
Publishing Institution: Roboflow Universe
Looking to create your own high-quality maritime datasets? We'd love to show you how! Reach out to us for a live demo of our data generation platform and see it in action.
