Domain-specific corpora for Word2vec model training - materials science, chemistry, and physics

Abstract

The project repository contains the materials science, chemistry, and physics-based domain corpora curated for the training purposes of the Word2vec model. All corpora were generated using the full-length manuscripts, as provided in the "Semantic Scholar Open Research Corpus" (S2ORC) database, and tagged by the corresponding research field tag – chemistry, physics, and materials science. Materials science (corpus-matsci980k), chemistry (corpus-chem980k), and physics (corpus-phys900k) corpora contain roughly 3.6 million tokens, with approximately 900k unique terms, while the mixed corpus (corpus-mixed1800k), generated as a mixture of documents in the following percentage, chemistry:matsci:physics = 23:32:45, contains about 7.7 million tokens with approximately 1800k unique terms. During the assembly process, the absolute number of articles from the given domain was assumed to be insignificant as long as the total number of generated tokens was aligned across corpora. Using the SciSpaCy tokenizer, each sampled training document was segmented into individual sentences that populated the corpus with one sentence per line of the txt file. Splitting raw text into sentences was followed by splitting each sentence into individual tokens using the ChemDataExtractor tokenizer, with all tokens lower-cased except chemical formulas and units of measurement. Tokens with a minimum count threshold of 5 were removed from the corpus.