GithubHelp home page GithubHelp logo

mcgip's Introduction

[AAAI 2024] Mining Gaze for Contrastive Learning toward Computer-assisted Diagnosis

by Zihao Zhao*, Sheng Wang*, Qian Wang, Dinggang Shen

The illustration of our proposed McGIP. For contrastive pre-training, a positive pair is typically only constructed between a image and its augmented version. In our McGIP, the images with similar gaze patterns when diagnosed by a radiologist are also considered as positive pairs and be pulled closer in the latent space.

[Under Review] Early Alignment in Self-Supervised Medical Imaging Pre-Training Using Human Eye Gaze

by Sheng Wang*, Zihao Zhao*, Zhenrong Shen, Bin Wang, Qian Wang, Dinggang Shen

Illustration of our proposed framework. (a) Examples showing images of similar semantics often with similar gaze patterns. In the left, there are two knee X-rays, both showing joint narrowing and corresponding similar gaze. In the middle, there are four breast mammography images, among which two are benign masses (green boxes) and other two are malign (blue boxes). The distributions of gaze points are similar across two benign masses, and also similar across two malign masses. In the right, there are two dental X-ray images of different patients. The yellow and red boxes indicate wisdom teeth on the upper and lower jaws, respectively. Across two images, the teeth of the same location have similar gaze heatmaps, corresponding to their shared anatomical roles and common underlying image semantics. (b) Gaze collection with an eye-tracker placing below radiologist's monitor. (c) The proposed Temporal Image Moment Analysis (TIMA) algorithm for evaluating radiologist's gaze similarity. (d) Gaze Pre-training (GzPT), where images with similar gaze are regarded as positive pairs in contrastive learning and thus pulled together in the latent space.

Introduction

In this paper, we introduce a plug-and-play module to rectify the issue of false-negative in medical contrasitve learning. This module efficiently constructs positive sample pairs for contrastive learning in medical image analysis based on Gaze similarity.

  • We provide the core code for integrating gaze data into the contrastive learning framework, available at here and here.
  • Furthermore, we offer code to evaluate different schemes for comparing gaze similarity in medical images, available at here.

This integration enhances the performance of contrastive learning, leading to improved results.

Usage

This repository contains the following:

  1. core-code/Contrastive+McGIP(GzPT): You can find modified code of contrastive learning under this folder. These codes demonstrate how to incorporate gaze data into an existing contrastive learning framework to achieve superior performance. we conduct experiments under mmselfsup 0.x environments. The main difference is shown in function self._create_buffer(N, idx_list)

  2. core-code/GazeSimilarityEval: We provide code implementations to measure gaze similarity under different scenarios. We designed different schemes tailored to various gaze data formats (i.e., gaze sequence and gaze heatmap) and situations (i.e., unstructured and structured images) in medical image analysis. We also propose an advanced method named temporal image moment analysis in the journal extension.

  3. pretraining: This directory provides the pretraining code modified from mmselfsup 0.x, and the pre-trained ViT weights mentioned in our journal extension can be found here.

    ViT-Small ViT-Base ViT-Large
    here here here
  4. downstream: This directory offers code for downstream evaluation.

Citation

@article{zhao2023mining,
      title={Mining Gaze for Contrastive Learning toward Computer-Assisted Diagnosis},
      author={Zihao Zhao and Sheng Wang and Qian Wang and Dinggang Shen},
      journal={arXiv preprint arXiv:2312.06069},
      year={2023},
}

Acknowledgement

This experiments are conducted on the basis of mmselfsup 0.x, thanks for their contributors.

mcgip's People

Contributors

zhaozh10 avatar

Stargazers

Sheng Wang avatar 白马非马 avatar Nicholas Baird avatar Shilin Wang avatar  avatar Dainis Graveris avatar XOR Turnkey avatar Tianshu Xin avatar  avatar 安琪 avatar Karina Zhao avatar Hanwen Bi avatar Han Wu avatar  avatar

Watchers

Kostas Georgiou avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.