huy-ngo / skin-crawler

A web crawler for crawling dermoscopic images of skin cancer.

Crawler for skin cancer images

This project attempts to create a crawler for dermoscopic images of skin cancer. This includes both frontend and backend.

The backend consists of packages to crawl images from websites and a web server to collect them and serve the images as an API.

The frontend consumes the API and displays information to the user.

How to run

Backend

• Add Kaggle API token at ~/.kaggle/kaggle.json
• In backend/, run static_crawler.py. The crawling process takes some time.
• Run Flask server:

export FLASK_APP=app.py
export FLASK_ENV=development
flask run

Frontend

Open file frontend/index.html

skin-crawler's Issues

Add Yandex and Wikimedia crawler to the static crawler

It seems even for 1 search, the Wikimedia crawler is time taking. I suppose Yandex one would suffer the same problem. Maybe we should crawl them beforehand.

This also let us write code consistently with the other two crawlers.

Shared folder for downloaded image

Since two of our API sources (Kaggle and ISIC) require downloading to the backend, let's move those images to a common folder backend/images/ for ease of hosting. They should be named in a way to avoid conflict (e.g. prefixing with ISIC- and Kaggle-). As far as I'm aware ISIC images are already prefixed, not sure about Kaggle.

Tutorial report

Skin cancer image crawler

List of the members.

1 Introduction

Maybe paraphrase the task.

2 Crawler

Describe how and where we crawl the datasets
Describe how the crawled datasets are given to the users.
Describe how we link everything togethers

2.1 Crawler's backend

Paraphrase/copy the whole modules documentations.
Describe the file that link everything. app.py

2.2 Crawler's frontend

Describe how the page is implemented

3 Datasets

Provide descriptions of datasets.

3.1 Kaggle

Description

3.2 ISIC

3.3 Yandex

3.4 Wikimedia

4 Interface

What can user do with the web app
Some images here

Clean up the code

The current backend code was written in a rushed manner due to the deadlines so it is not very clean and of bad quality. Let's refactor it after #26 is merged.

Incorrect file path

As can be seen in the screenshot, from #19, the images are supposed to be stored in static/kaggle/dinhanhx/skin-cancer-mnist-ham10000/. However, in reality, they're stored in two subfolder: HAM10000_images_part_1 and HAM10000_images_part_2.

Please fix this.

Module ISIC downloaded failed: truncated image

As of writing, commit aa8165e, branch isic-pullrequest, pull request #3,
when I called ISIC_getdata(5, 5, False, False) and ISIC_getdata(), no images are downloaded.
CL ouput:

Expected output: a list.

Create route to serve images from Yandex image

Child issue of #5

Create a crawler to crawl from Yandex image

Child issue of #4

Create route to serve images from Wikimedia

Child issue of #5

Create route to serve images from Kaggle

Child issue of #5

Create a crawler to crawl from Wikimedia

Child issue of #4

Image URL should always return an absolute path

The frontend in this case is separated from the server. Therefore, the static path that is returned by the server ('static/...') can't be used. It needs to be made absolute.

Create a API for frontend

This API basically integrates images from all other sources and return to the frontend requests.

Frontend development

The frontend needs to show images and caption by requesting from the crawler API.

Creating modules that crawl images from websites and API

• Kaggle (#2 - Dinh Anh)
• ISIC (#3 - Vuong)
• Yandex (#26 - Quang)
• Wikimedia (#25 - Long)

The image URL field is not correct

Related: #28

Currently, the API use path for the URL of the image. This came up because I was dealing with the downloaded files, and within a computer we would say path to a file and not link or URL. However, since it's incorrect and unintuitive to say so, we should change the name of the field. I would like to change it to image_url.

Create route to serve images from ISIC

Child issue of #5