Contrastive Learning for Document Understanding

Abstract

Contrastive learning is a self-supervised methodology that enables models to learn robust representations by bringing similar samples together in embedding space while pushing dissimilar ones apart. The origins of contrastive learning in NLP trace back to 2013, where Mikolov et al. introduced it via word embeddings using co-occurrence and negative sampling, significantly improving representation quality in a computationally efficient manner.Encyclopedia Pub Notably, Logeswaran and Lee (2018) extended contrastive ideas to sentence-level representation by casting context prediction as classification: distinguishing the true context sentence from contrastive alternatives. This enabled learning high-quality sentence embeddings from unlabeled text and led to superior performance on downstream tasks with remarkable training speed.arXiv Additionally, Bose et al. (2018) proposed Adversarial Contrastive Estimation, which enhances contrastive learning by incorporating an adversarially trained negative sampler, resulting in harder negative examples. This accelerated convergence and improved embedding representations for word embeddings, order embeddings, and knowledge graph embeddings.arXiv These foundational works highlight methodologies applicable to document understanding: contrastively learning representations through context vs. non-context sentences, using adversarial negatives to strengthen embedding quality, and scaling from words to sentences—forming a basis for document-level applications.