Medical image retrieval in mANY dimensions

Medical visual information retrieval has been an extremely active research domain but only very little retrieval has so far been attempted really taking into account 3D or 4D aspects of the medical data, although for a real integration of visual retrieval into PACS (Picture Archival and Communication Systems) this constitutes by far the largest amount of data to be treated. The MANY project will enlarge the current content–based medical image retrieval approaches from two dimensions to three and more dimensions as these datasets contain several challenges that need to be overcome such as the shear amount of data to be treated and the relatively small part of the data that actually contains relevant (or abnormal) information (region of interest). MANY combines knowledge of clinical partners in the emergency radiology with the process knowledge of the medical informatics service, the imaging experience of the CIBM and the image retrieval knowledge of the medGIFT research group.

The cornerstone of MANY is to use N-dimensional texture characterization of biomedical tissue to assess content-based inter-case distance. In a first step, the goal is to go beyond 3D surfaces and towards 3D texture, which was little attempted in the literature (see Figure 1).


Figure 1. Use of 3D texture analysis to characterize biomedical tissue.

In a second step, tools for 4D data analysis and retrieval will be explored. 4D data typically accounts for 3D videos (3D+time, see Figure 2a), but can also be any combination of physical dimensions (e.g. 3D + energy level with dual-energy CT, …). Fundamental questions such as “how to define a texture in 4 dimensions” will be addressed. Another goal is to allow retrieval on combination of medical tomographic modalities such as PET/CT and PET/MRI (see Figure 2b).

Figure 2. a) Screenshot of a video of a beating heart in 3D in 4D CT imaging. b) Fusion of modalities with PET-CT imaging.

Other contributions of this project include the development of a browser–based user interface for multi–dimensional data visualization to support the retrieval process of visually similar cases and the interaction with the user. A prototype for such an interface exists already and will mainly be extended and optimized. Such web interfaces will make the software available for a larger number of users, particularly within the Geneva University Hospitals and at the medical faculty of the University of Geneva but also to a larger community via the Internet. Both affiliations of the main project leader, the University and Hospitals of Geneva and the University of Applied Sciences Western Switzerland support the proposal and have already provided funding for preliminary studies as the topic is regarded as covering an important domain.


Henning Müller

Adrien Depeursinge (Postdoc)

Antonio Foncubierta (PhD candidate)

Alejandro Vargas

Oscar Jiménez


University Hospitals of Geneva
  • Pierre-Alexandre Poletti, Emergency Radiology
  • Antoine Geissbuhler, Medical Informatics
  • François Lazeyras, Center for Biomedical Imaging
  • Dimitri Van de Ville, Center for Biomedical Imaging and EPFL



  • A multimedia database of high-resolution computed tomography (HRCT) images from patients affected with interstitial lung diseases (ILD) is publicly available!