A Hybrid Vision Transformer with Intra-Attention Architecture for Enhanced Medical Image Retrieval

Abstract

The rapid growth in medical imaging techniques and the expansionof medical image repositories have created a strong need for accurate image retrieval techniques to efficiently retrieve relevant images. In this approach, a Hybrid Vision Transformer (ViT) Architecture with intra-attention mechanism for enhanced image retrieval. This approach integrates the Convolutional Block Attention Module (CBAM) directly with the multi-head self-attention of Vision Transformer (ViT), enabling more adaptive and fine-grained feature refinement. Unlike traditional fusion-based methods, this model dynamically reweights feature representations by leveraging spatial and channel-wise attention at multiple transformer stages. With spatial attention applied at each stage of MSA, ViT learns to focus more on medically significant image regions, while channel attention enables ViT to prioritize the most informative features and suppress irrelevant information. Experimental results demonstrated the significance of proposed method over standalone features of ViT and other existing methods in terms of improved efficiency, precision and recall. These findings suggest that embedding CBAM within ViT’s self-attention layers can enhance retrieval accuracy while maintaining interpretability, making it a promising solution for medical image analysis.