Posts Categorized:

By Stephen Aylward, Bram van Ginneken, Michael Liebling, and Bahram Parvin

During ISBI 2013, seven well-attended workshops were held in which participants offered competing solutions to specific Grand Challenges in Medical Image Analysis. This article describes the goals of the endeavor and gives summaries of the seven Challenges.

Grand Challenges

The goal of a Grand Challenge is to focus the efforts of researchers on a foundational problem within a field, to facilitate the thorough and quantitative comparison of the methods develop by the researchers, and to ultimately discover an effective solution to the driving problem.

The key components of a successful Grand Challenge are (1) data that is representative of the range of conditions that span the driving problem’s input space; (2) ground truth that represents the ideal solution for each input data; and (3) metrics that accurately predict a method’s performance for the driving problem. In that manner, Grand Challenges bridge the gap between theory and practice and accelerate the pace of research by distributing meaningful data to researchers and then quantitatively tracking their progress using real-world metrics.

Grand Challenges come in many forms, and an overview of those aimed at medical image analysis is provided at http://www.grand-challenge.org. They typically consist of making a portion of the problem’s data and associated ground truth publicly available for algorithm development and sequestering the remainder of the data for unbiased estimation of algorithms’ performance. Evaluation on the sequestered data is often done “live,” during workshops at conferences.

ISBI Grand Challenges

At ISBI 2013 there were seven outstanding Grand Challenges, and they were extremely well attended. Each Grand Challenges was hosted as a half-day workshop. Several of these workshops had over 50 registered attendees. Attendees were free to move between the Grand Challenges’ workshops. The grand challenges chosen for ISBI 2013 were the following:
1. HARDI Reconstruction Challenge
2. 3D Deconvolution Microscopy Challenge
3. Computer Aided Detection of Pulmonary Embolism (CADPE)
4. 3D Segmentation of Neurites in EM Images
5. Automated Segmentation of Prostate Structures
6. Localization Microscopy Challenge
7. Cell Tracking Challenge

One area of emphasis for the ISBI Grand Challenges was education. Many of the Grand Challenges included invited talks from users with vested interest in the challenge’s success, e.g., clinicians and biologists. Each Grand Challenge also included oral and/or poster presentations from the algorithm developers who were participating in the challenge.

Another area of emphasis was dissemination. That is, we wanted the lessons learned from the challenges to spread beyond the workshop attendees. This dissemination was supported in two ways. First, many of the Grand Challenges were fully automated, and those automated systems continue to run. For example, the prostate segmentation challenge used data hosted on NCI’s The Cancer Imaging Archive (TCIA, http://www.cancerimagingarchive.net/) and results from processing that data can be uploaded and automatically scored and ranked at http://challenge.kitware.com/midas/community/5. Second, each Grand Challenge organizer was given a chance to present their findings during the main conference and was encouraged to develop a companion journal article. Although those presentations were brief, they inspired further discussions with participants. Summaries of each of the ISBI 2013 Grand Challenges are given next.

1. HARDI Reconstruction Challenge

Alessandro Daducci (Signal Processing Lab, EPFL, Switzerland), Emmanuel Caruyer (Department of Radiology, University of Pennsylvania, USA), Maxime Descoteaux (Sherbrooke Connectivity Imaging Lab, Université de Sherbrooke, Canada), Jean-Philippe Thiran (Signal Processing, EPFL, Switzerland) http://hardi.epfl.ch/static/events/2013_ISBI/

In the last few years a multitude of new reconstruction approaches have been proposed to recover the local intra-voxel fiber structure: some of them aim at improving the quality of the reconstructions while others focus on reducing the acquisition time. However, when a new algorithm is proposed, the performances are normally assessed with ad hoc synthetic data and evaluation criteria, and comparing different approaches can be difficult.

The HARDI (high angular resolution diffusion imaging) reconstruction challenge was organized with the aim to provide all researchers in this field with a common framework to assess the performances of their reconstruction schemes and fairly compare their results under controlled conditions. The main goal of this contest was to investigate not only the accuracy in the local estimation of the intra-voxel fiber configuration of each algorithm, but also its impact on subsequent global connectivity analyses.

2. 3D Deconvolution Microscopy Challenge

Cedric Vonesch, (Ecole Polytechnique Federale de Lausanne, Switzerland), Stamatis Lefkimmiatis (Ecole Polytechnique Federale de Lausanne, Switzerland), Laure Blanc-Feraud (Laboratoir I3S, Sophia-Antipolis, France), Michael Unser (Ecole Polytechnique Federale de Lausanne, Switzerland), Rainer Heintzmann (Friedrich-Schiller-Universitat, Jena, Germany), Arne Seitz (Ecole Polytechnique Federale de Lausanne, Switzerland) http://bigwww.epfl.ch/deconvolution/challenge/

Deconvolution is one of the most common image-reconstruction tasks that arise in 3D fluorescence microscopy. The aim of this challenge was to benchmark existing deconvolution algorithms and to stimulate the community to look for novel, global and practical approaches to this problem.

The challenge was organized as a three-stage tournament (training, qualification and final). It was based on novel realistic-looking synthetic data sets representing various subcellular structures. In addition it relied on a number of common and advanced performance metrics to objectively assess the quality of the results.

3. Computer Aided Detection of Pulmonary Embolism (CADPE)

Germán González,(MIT-RLE Post-Doctoral Affiliate), María Jesús Ledesma, (Universidad Politécnica de Madrid), Raúl San José, (Radiology at Harvard Medical School), Eduardo Fraile (Unidad Central de Radiodiagnóstio. Madrid), Ara Kassarjian, MD. Corades S.L., Daniel Jiménez-Carretero (Universidad Politécnica de Madrid) http://www.cad-pe.org

A pulmonary (thrombo) embolism (PE) refers to the situation when a portion of a blood clot becomes lodged in a pulmonary artery. PE is diagnosed with computed tomography pulmonary angiography (CTPA). Delay of treatment or lack of treatment results in increased morbidity and mortality. In comparison to an expert panel, the average radiologist misses visible emboli. Computer aided detection (CAD) algorithms have been shown to increase radiologists’ sensitivity. However, CAD has not been adopted into clinical practice due to the large amount of false positive detections that those algorithms produce. This challenge was intended to enable new CAD algorithms with lower false positive rates than state-of-the-art and to make CAD for PE a reality.

4. 3D Segmentation of Neurites in EM Images

Ignacio Arganda-Carreras (Massachusetts Institute of Technology, Cambridge, MA), H. Sebastian Seung, Ashwin Vishwanathan, Daniel Berger http://brainiac.mit.edu/SNEMI3D

In this challenge, a full stack of electron microscopy (EM) slices was used to train machine-learning algorithms for the purpose of automatic segmentation of neurites in 3D. This imaging technique visualizes the resulting volumes in a highly anisotropic way, i.e., the x- and y-directions have a high resolution, whereas the z-direction has a low resolution, primarily dependent on the precision of serial cutting. EM produces the images as a projection of the whole section, so some of the neural membranes that are not orthogonal to a cutting plane can appear very blurred. None of these problems led to major difficulties in the manual labeling of each neurite in the image stack by an expert human neuro-anatomist. The aim of the challenge was to compare and rank the different competing methods based on their object classification accuracy in three dimensions.

5. Automated Segmentation of Prostate Structures

Keyvan Farahani (NCI), John Freymann (NCI-Fredrick), Justin Kirby (NCI-Fredrick), Carl Jaffe (Boston University), Nicholas Bloch (Boston University), Anant Madabhushi (Case Western Reserve University), Henkjan Huisman (Radboud University Nijmegen Medical Centre), and Andinet Enquobahrie (Kitware Inc.) https://wiki.cancerimagingarchive.net/x/8QRp

The Cancer Imaging Program of the National Cancer Institute (NCI) in collaboration with the International Society for Biomedical Imaging (ISBI) launched a grand challenge in segmentation of internal structures of the prostate gland based on magnetic resonance imaging data. The overall goal of this challenge was to promote the development of robust open and closed source codes that can automatically identify the peripheral zone and the central gland from volumetric T2-weighted MR image sets. Data for the prostate challenge was provided by Boston University and Radboud University, Nijmegen Medical Centre (the Netherlands). The collection, consisting of training and test data, is hosted on the Cancer Image Archive (TCIA), a publicly available NCI resource.

6. Localization Microscopy Challenge

Dr. Daniel Sage (Biomedical Imaging Group, EPFL, Lausanne, Switzerland), Dr. Thomas Pengo (Laboratory of Experimental Biophysics, EPFL, Lausanne, Switzerland), Dr. Hagai Kirshner (Biomedical Imaging Group, EPFL, Lausanne, Switzerland), Dr. Nico Stuurman (Vale Lab, UCSF, San Francisco) Jungong Min (Bio Imaging Signal Processing, KAIST, Republic of Korea), Prof. Suliana Manley (Laboratory of Experimental Biophysics, EPFL, Lausanne, Switzerland) http://bigwww.epfl.ch/smlm/challenge/

Super-resolution imaging is a recent emerging field of microscopy that improves the resolution of the conventional microscopy by an order of magnitude. It allows one to observe and study biological structures at the molecular scale. The most popular technique for super-resolution microscopy is the single molecule localization imaging, which localizes individual molecules that are spatially and temporally separated one form the other. Super-resolution data is reconstructed by image-analysis software, and the reconstruction quality is highly dependent on the algorithm used and the software implementation.

The goal of this challenge was to benchmark currently available localization software, in particular, comparing their performance in terms of detection rate, localization accuracy, and computation time. Accessibility and usability for the end-user were also evaluated. The benchmark data consisted of simulated images of various biological structures such as tubulins. The simulated data accounts for fluorophores activation and excitation, image formation, and known perturbations models. As the ground truth exact positions of the fluorophores are known, the benchmarking metrics used objective measures.

7. Cell Tracking Challenge

Carlos Ortiz de Solórzano (Center for Applied Medical Research, CIMA-University of Navarra), Michal Kozubek (Masaryk University, Brno, Czech Republic), Erik Meijering (Erasmus University, Rotterdam, The Netherlands), Arrate Muñoz Barrutia, CIMA-University of Navarra, Pamplona, Spain http://www.codesolorzano.com/celltrackingchallenge/Cell_Tracking_Challenge/Welcome.html

Tracking moving cells in time-lapse video sequences is a challenging task, required for many applications in both scientific and industrial settings. Properly characterizing how cells move as they interact with their surrounding environment is key to understanding the mechanobiology of cell migration and its multiple implications in both normal tissue development and many diseases.

In this challenge we called for the submission of existing and newly developed cell tracking algorithms. These algorithms were evaluated and compared in terms of cell segmentation and cell lineage accuracy, as well as the time required for the analysis of the time-lapse sequences.

The datasets consisted of fluorescence microscopy 2D and 3D time-lapse videos containing varying cell types and/or embedding environments, stained either nuclear or cytoplasmically. Furthermore, synthetic time-lapse sequences of 2D and 3D nuclei moving in realistic environments were used to further test the performance of the algorithms under different noise and cell density conditions.

Conclusion

Grand Challenges are effective in a wide variety of algorithm development domains; and the ISBI 2013 Grand Challenges were diverse, well attended, and productive. Visit the Grand Challenge’s website to learn more about its motivating problem, its data, and the algorithms that performed the best during the conference evaluations. Several of these Grand Challenges are ongoing, visit their websites to determine which challenges are tracking the progress of research, now and into the future.

Work is already underway to include Grand Challenges at ISBI 2014, and we are pursuing new ideas for advancing their education and dissemination missions.

Contributors

Stephen R. Aylward is the founder of and Senior Director of Operations at Kitware’s North Carolina office, and an adjunct professor in computer science at UNC. Stephen received a Ph.D. in medical image analysis from UNC. Dr. Aylward’s research, funded by multiple R01s and SBIRs, has recently focused on image registration in the presence of changing pathologies and on vascular network characterization for understanding cancer progression. Read more

Bram van Ginneken is Professor of Functional Image Analysis at Radboud University Nijmegen Medical Centre. He is co-chair of the Diagnostic Image Analysis Group within the Department of Radiology, together with Nico Karssemeijer. He obtained his Ph.D. at the Image Sciences Institute (ISI) on Computer-Aided Diagnosis in Chest Radiography. He is Associate Editor of IEEE Transactions on Medical Imaging and member of the Editorial Board of Medical Image Analysis. He has pioneered the concept of challenges in medical image analysis. He also works for Fraunhofer MEVIS in Bremen, Germany. Read more

Michael Liebling is an assistant professor in the Department of Electrical and Computer Engineering at the University of California, Santa Barbara. He received his PhD from the Ecole Polytechnique Federale de Lausanne (EPFL), Switzerland. His research interests include biological microscopy and image processing for the study of dynamic biological processes. Read more

Bahram Parvin is Staff Scientist in the Life Sciences Division of the Berkeley Lab of Lawrence Berkely National Laboratory and Adjunct Professor of Electrical Engineering at the University of California. He received his PhD in Electrical Engineering at the University of Southern California in 1991. His current research work focuses on Integrative Biology and development of novel imaging probes. Dr. Parvin is a Senior Member of the IEEE. Read more