MedOpenSeg: Open-world medical segmentation with memory-augmented transformers

Open-world segmentation in medical imaging presents unique challenges, as models

must generalize to seen and unseen classes while retaining knowledge of previously

seen structures. We propose MedOpenSeg, a Memory-Augmented transformer framework

that dynamically stores and updates class prototypes to enhance segmentation accuracy,

improve adaptability to new anatomical structures, and detect novel regions during

inference. MedOpenSeg integrates a Swin-Transformer 3D backbone with a memory

bank module that retrieves class-specific feature embeddings and facilitates prototypebased novelty detection using cosine similarity and Euclidean Distance Sum (EDS). We benchmark MedOpenSeg on multiple datasets against state-of-the-art closed-set segmentation and foundation models, demonstrating its effectiveness in handling openset medical segmentation. Code is publicly available at https://github.com/robustmleurecom/MedOpenSeg.git.