An investigation of benchmark image collections: how different from digital libraries?

This paper aims to introduce the construction methods, image organization, collection use and access of benchmark image collections to the digital library (DL) community. It aims to connect two distinct communities: the DL community and image processing researchers so that future image collections could be better constructed, organized and managed for both human and computer use.

Image collections are first identified through an extensive literature review of published journal articles and a web search. Then, a coding scheme focusing on image collections’ creation, organization, access and use is developed. Next, three major benchmark image collections are analysed based on the proposed coding scheme. Finally, the characteristics of benchmark image collections are summarized and compared to DLs.

Although most of the image collections in DLs are carefully curated and organized using various metadata schema based on an image’s external features to facilitate human use, the benchmark image collections created for promoting image processing algorithms are annotated on an image’s content to the pixel level, which makes each image collection a more fine-grained, organized database appropriate for developing automatic techniques on classification summarization, visualization and content-based retrieval.

This paper overviews image collections by their application fields. The three most representative natural image collections in general areas are analysed in detail based on a homemade coding scheme, which could be further extended. Also, domain-specific image collections, such as medical image collections or collections for scientific purposes, are not covered.

This paper helps DLs with image collections to understand how benchmark image collections used by current image processing research are created, organized and managed. It informs multiple parties pertinent to image collections to collaborate on building, sustaining, enriching and providing access to image collections.

This paper is the first attempt to review and summarize benchmark image collections for DL managers and developers. The collection creation process and image organization used in these benchmark image collections open a new perspective to digital librarians for their future DL collection development.