Unsupervised hard Negative Augmentation for contrastive learning

Shu: UNA

Yuxuan Shu   Vasileios Lampos

preprint 2024

Code     Paper


We present Unsupervised hard Negative Augmentation (UNA), a method that generates synthetic negative instances based on the term frequency-inverse document frequency (TF-IDF) retrieval model. UNA uses TF-IDF scores to ascertain the perceived importance of terms in a sentence and then produces negative samples by replacing terms with respect to that. Our experiments demonstrate that models trained with UNA improve the overall performance in semantic textual similarity tasks. Additional performance gains are obtained when combining UNA with the paraphrasing augmentation. Further results show that our method is compatible with different backbone models. Ablation studies also support the choice of having a TF-IDF-driven control on negative augmentation.


We therefore propose Unsupervised hard Negative Augmentation (UNA), an augmentation strategy for generating negative samples in Self-supervised contrastive learning. The proposed method is driven by TF-IDF to generate hard negative pairs. Words with more substance have a greater probability of being swapped, and more common words do not.


We pretrained on an English Wikipedia corpus (containing 1 million sentences) proposed by SimCSE and evaluated on seven Semantic Textual Similarity tasks, which are STS 2012-2016, STS Benchmark, and SICK Relatedness. All data sets can be found in the official code base of SimCSE.


      title={Unsupervised hard Negative Augmentation for contrastive learning},
      author={Yuxuan Shu and Vasileios Lampos},
      journal={arXiv preprint arXiv:2401.02594}

© Yuxuan Shu 2024