MICCAI Hackathon

Come up with ideas and suggestions regarding reproducibility, diversity,
and selection of papers for future MICCAI editions

Sunday, 4 October 2020 from 10:30 UTC to 17:20 UTC

About the MICCAI Hackathon

(Slides available here)

The advance of machine learning has had a considerable impact on MICCAI, pushing the limits of algorithms, opening up completely new applications and ultimately, leading to an increased overall interest in MICCAI. With this success, however, come new challenges for the community. Complex, data-driven algorithms are more difficult to reproduce and the increasing number of paper submissions to the MICCAI conference poses new questions regarding the selection process and the diversity of topics covered.

As a platform to exchange, discuss, and possibly find creative and novel solutions to these challenges, we are organizing a hackathon as a full-day satellite event at the virtual MICCAI 2020 on Sunday, 4th October. The MICCAI Hackathon adheres to the typical format of a hackathon: participants gather together, receive input from keynote speakers, work in teams or individually to find solutions for the topic, and finally present their outcome at the end of the hackathon. We are convinced that the new satellitle event format of a hackathon is an opportunity for the MICCAI community to reflect and come up with ideas and suggestions regarding reproducibility, diversity, and selection of papers for future MICCAI editions.
Have a look at the topics!

Our white paper describing immediate and long-term measures to possibly improve MICCAI regarding reproducibility, diversity, and selection of papers is now available on arXiv. Check it out!


Balsiger, F., Jungo, A., Akash R J, N., Chen, J., Ezhov, I., Liu, S., Ma, J., Paetzold, J. C., Saravanan R, V., Sekuboyina, A., Shit, S., Suter, Y., Yekini, M., Zeng, G., & Rempfler, M. (2021). The MICCAI Hackathon on reproducibility, diversity, and selection of papers at the MICCAI conference. ArXiv Preprint. http://arxiv.org/abs/2103.05437

Topics

The MICCAI Hackathon addresses reproducibility, diversity, and selection of MICCAI papers. Five categories have been defined with questions listed below. For each category, a link will forward you to an Airtable with related material. As a participant, you will work, in a team or individually, on one or multiple categories and address the questions you think might be the most important. Write us if you would like to extend the list with questions or have found related material!

Reproducibility 1: What does it need for a MICCAI paper to be reproducible?

  • Create a machine learning reproducibility checklist specific for MICCAI.
  • Propose a machine learning code completeness checklist specific for MICCAI.
  • How to ensure reproducibility when the data cannot be shared?
(Resources available here)

Reproducibility 2: What could MICCAI do to encourage reproducibility?

  • How to integrate best practices for reproducibility into the review process?
  • Investigate how reproducible MICCAI papers are.
  • How to estimate the reproducibility of a paper without having to re-implement it?
  • Propose a challenge, prize, educational initiative, best practices or a related concept to encourage reproducibility.
  • Make it easier for authors to create reproducible contributions.
(Resources available here)

Diversity: Is MICCAI diverse enough?

  • Analyze the diversity of MICCAI.
  • How to encourage contributions from non-mainstream tasks or applications?
  • Propose ways to balance factors like methodological novelty, applicability and clinical relevance.
  • What defines a clinically relevant paper?
  • How can paper diversity be supported in the review process?
(Resources available here)

Selection 1: What could MICCAI do to improve the review process?

  • Analyze the influence of the paper matching on acceptance and propose potential improvements.
  • How to handle pre-print submission (e.g., a potential bias in review process)?
  • How to cope with reviewers who deviate from the reviewing guidelines?
  • Come up with methods to ensure review quality.
  • Propose solutions for rewarding good reviewers.
  • How to handle the growing need of reviewers?
  • Propose improvements to the rebuttal process.
(Resources available here)

Selection 2: What defines a good MICCAI paper?

  • How to promote papers with insights or high clinical relevance that are not new state-of-the art?
  • Propose best practices for good baselines.
  • Develop a method to estimate the significance and/or impact of a submission.
  • Propose ways to judge and promote clinical usefulness.
  • How to encourage diversity of contributions beyond "CVPR methods on a medical dataset"?
(Resources available here)

Keynote speakers

Prof. Dr. Anne L. Martel

Medical Biophysics, University of Toronto
&
Sunnybrook Research Institute
Toronto, Canada

Putting together a MICCAI scientific program: A guide to the reviewing process and program organization for MICCAI 2020

In this keynote, MICCAI 2020 co-program chair Anne L. Martel provides a concise overview of the reviewing process and program organization for this years' edition of the conference. The aim of the talk is to inform the hackathon participants on different aspects of the current state on reproducibility, diversity and selection of MICCAI papers. She not only describes the process, but also shares observations and experiences from this year, and points out areas with room for improvement. More specifically, the talk is structured into four parts. First, she describes the selection process of the program committee, area chairs and reviewers. Second, she moves on to the actual review phase, covering paper assignment as well as current guidelines and policies. Third, the statistics of the accepted papers of MICCAI 2020 are presented and discussed. Finally, the talk concludes with remarks on how MICCAI 2020 converged to a fully virtual venue.

(Slides available here)

Koustuv Sinha

School of Computer Science, McGill University (Mila)
&
Reproducibility Chair at NeurIPS 2020
Montreal, Canada

Reproducibility in Machine Learning: From Theory to Practice

A recurrent challenge in machine learning research is to ensure that the presented and published results are reliable, robust, and reproducible. Reproducibility, which is obtaining similar results as presented in a paper or talk, using the same code and data (when available), is a necessary step to verify the reliability of research findings. Reproducibility is also an important step to promote open and accessible research, thereby allowing the scientific community to quickly integrate new findings and convert ideas to practice. Reproducibility also promotes the use of robust experimental workflows, which potentially reduce unintentional errors. In this talk, I will first present some statistics on the need for reproducibility in machine learning research, and then cover the recent approaches taken by the community to promote reproducible science. Finally, I will talk in-depth about the experimental workflows that you can integrate with your research, to ensure and promote reproducible science.

(Slides available here)

Mentors

The mentors below will provide guidance, feedback, and inspiration to the participants during 20 minutes time slots.

Marleen de Bruijne

Marleen de Bruijne

Mattias P. Heinrich

Mattias P. Heinrich

Georg Langs

Georg Langs

İlkay Öksüz

İlkay Öksüz

Lena Maier-Hein

Lena Maier-Hein

Koustuv Sinha

Koustuv Sinha

Program

This is a preliminary program and might be subject to change.
All daytimes are indicated in UTC+0 (conference reference time zone).

  • Sunday, 27 September

    ● Data available

    The welcome video, the recorded keynotes, as well as all the material will be available online.

  • Sunday, 4 October  10.00 - 10.20am

    ● Welcome and FAQ

    The organizers open the hackathon, provide final information and answer questions.

  • 10.30 - 10.50am

    ● Mentoring session with İlkay Öksüz

    The participants may interact with the mentor available during this time slot.

  • 11.30 to 11.50am

    ● Mentoring session with Mattias Heinrich

    The participants may interact with the mentor available during this time slot.

  • 12.30 to 12.50pm

    ● Mentoring session with Georg Langs

    The participants may interact with the mentor available during this time slot.

  • 1.00 to 1.20pm

    ● Q&A keynote Anne Martel

    The Q&A session for the first keynote (video to be watched before).

  • 2.00 to 2.20pm

    ● Q&A keynote Koustuv Sinha

    The Q&A session for the second keynote (video to be watched before).

  • 3.00 to 3.20pm

    ● Mentoring session with Marleen de Bruijne

    The participants may interact with the mentor available during this time slot.

  • 4.00 to 4.20pm

    ● Mentoring session with Lena Maier-Hein

    The participants may interact with the mentor available during this time slot.

  • 5.00 to 5.20pm

    ● Mentoring session with Koustuv Sinha

    The participants may interact with the mentor available during this time slot.

  • Wednesday, 7 October  5.00pm

    ● Presentation deadline

    A video of the presentation needs to be delivered to the organizers.

  • Thursday, 8 October  3.00-5.00pm

    ● Discussion and Q&A with the participants

    Public discussion and Q&A with the participants about the outcomes.
    Closing remarks by the organizers.

Organizing committee

Fabian Balsiger

PhD student
University of Bern
Bern, Switzerland

Alain Jungo

PhD student
University of Bern
Bern, Switzerland

Dr. Markus Rempfler

Machine learning engineer
Friedrich Miescher Institute for Biomedical Research (FMI)
Basel, Switzerland

Sponsors

We are grateful for the support by our sponsors.
If you also want to support the MICCAI Hackathon, please contact us!

Contact

Do not hesitate to get in contact with us!