compile multilingual document-aligned comparable corpora using the crosslanguage links.categories allow to compile domain-specific corpora.However, the rich annotations in the XML files facilitate many more applications: Main purpose of the Wikipedia Monolingual Corpora is to provide large text corpora in many languages which can be used for the routine tasks of corpus linguistics, like generating word frequency lists or collecting n-gram statistics. (It’s not a bug, it’s a feature!) Applications We also (try to) discard sections with weblinks and references at the end of articles, because they often contain foreign language material. In order to minimize noise in the corpus, we discard all articles with unmatched brackets, i.e. Because articles can be edited manually by anybody, unmatched brackets sometimes occur. Wiki markup uses all kinds of brackets to mark links, categories, etc. The corpora are based on the Wikipedia dumps which contain articles in Wiki markup format, packed in XML. Only articles are output that belong to one of the categories listed in FILE. Options:Īll content that is enclosed in math tags is deleted.Īll content that is enclosed in table tags is deleted.Īrticles that are marked as disambiguation page are deleted.Īll articles that belong to one of the categories listed in FILE are ignored. The encoding of the output file will be UTF-8. Where INPUT is an unzipped Wikipedia corpus XML file, and OUTPUT is the raw text file that will be produced. You can extract text only from the XML files using the Perl script. The sample contains one article from the English Wikipedia. To see an example of the XML format click here. The title of the target article is contained in the attribute target. Marks a textlink to another Wikipedia article. This element encloses the current article’s textual content. This elements marks the article as a disambiguation page. These links contain a colon that separates the namespace from the link target. There are also crosslanguage links to other namespaces, e.g. In this case, the link contains the article name followed by a ‚#‘ and the section’s anchor name. Important note: In Wikipedia, crosslanguage links may link an article to a section of a target article. The attribute language specifies the target language. The attribute name contains the title of a Wikipedia article in another language the current article is linked to. The attribute name contains the number of textlinks in the current article that refer to other Wikipedia articles. An article can have 0.n daughter elements of type category. The attribute name contains a Wikipedia category the current article is assigned to. The attribute freq contains the number of times this anchor text was used to refer to the current article. The attribute name contains the anchor text that is used in a textlink to refer to the current article. The attribute name contains the total number of textlinks in other articles that link to the current article. An article can have 0.n daughter elements of type redirect.
The attribute name contains an article title that redirects to the present article. However, the redirect information is contained in the redirect element(s) of the target article. The title is unique (in the current language version).Īrticles that are redirects to another article are not stored in the XML files. It has an attribute name which contains the article’s title. This element encloses each Wikipedia article. The XML files contain the following information: XML element Have a quick look at a sample XML file containing one English article.
#Monolingual download full#
Each XML file contains the full textual content of the individual language version of Wikipedia, extended with many annotations like article and paragraph boundaries, number of links referring to each article, crosslanguage links, categories and more.
#Monolingual download download#
Here you can download text corpora extracted from the Wikipedia dumps in 30 languages, amounting to nearly 10 billion tokens altogether.
0 kommentar(er)
