A novel soft attention-based multi-modal deep learning framework for multi-label skin lesion classification

Skin cancer is one of the fatal cancers worldwide. Early detection of this disease can significantly increase the survival rate. In this study, a multi-modal and soft attention based hybrid deep learning model is proposed for automated and accurate multi-label skin lesion classification. The proposed network includes a multi-branch structure that integrates feature maps from different modalities in a hybrid manner. These branches enable to extract features from multiple modalities separately and learn complex combinations between them. These branches include a modified Xception architecture, a new feature extraction method, as well as soft attention module that is proposed to make the network focus on discriminative parts of skin lesions. The final diagnosis is obtained by the fusion of the predictions from three branches. The proposed framework was evaluated on the publicly available seven-point criteria evaluation dataset, a well-established multi-modality multi-label skin disease dataset. It achieved an average accuracy of 83.04% for multi-label skin lesion classification. It is more accurate than the state-of-the-art methods and improves the average accuracy by more than 2.14% on the test set.