Tài liệu Báo cáo khoa học: "Large linguistically-processed Web corpora for multiple languages" doc

Large linguistically-processed Web corpora for multiple languages
Marco Baroni
SSLMIT
University of Bologna
Italy

Adam Kilgarriff
Lexical Computing Ltd. and
University of Sussex
Brighton, UK

Abstract
The Web contains vast amounts of linguis-
tic data. One key issue for linguists and
language technologists is how to access
it. Commercial search engines give highly
compromised access. An alternative is to
crawl the Web ourselves, which also al-
lows us to remove duplicates and near-
duplicates, navigational material, and a
range of other kinds of non-linguistic mat-
ter. We can also tokenize, lemmatise and
part-of-speech tag the corpus, and load the
data into a corpus query tool which sup-
ports sophisticated linguistic queries. We
have now done this for German and Ital-
ian, with corpus sizes of over 1 billion
words in each case. We provide Web ac-
cess to the corpora in our query tool, the
Sketch Engine.
1 Introduction

The Web contains vast amounts of linguistic data
for many languages (Kilgarriff and Grefenstette,
2003). One key issue for linguists and language
technologists is how to access it. The drawbacks
of using commercial search engines are presented
in Kilgarriff (2003). An alternative is to crawl the
Web ourselves.
1
We have done this for two lan-
guages, German and Italian, and here w e report on
the pipeline of processes which give us reasonably
well-behaved, ‘clean’ corpora for each language.
1
Another Web access option is Alexa (http://pages.
alexa.com/company/index.html), who allow the
user (for a modest fee) to access their cached Web directly.
Using Alexa would mean one did not need to crawl; however
in our experience, crawling, given free software like Heritrix,
is not the bottleneck. The point at which input is required is
the filtering out of non-linguistic material.
We use the German corpus (which was developed
first) as our example throughout. The procedure
was carried on a server running RH Fedora Core 3
with 4 GB RAM, Dual Xeon 4.3 GHz CPUs and
about 2.5 TB hard disk space. We are making the
tools we develop as part of the project freely avail-
able,
2
in the hope of stimulating public sharing of
resources and know-how.

2 Crawl seeding and crawling
We would like a “balanced” resource, containing
a range of types of text corresponding, to some
degree, to the mix of texts we find in designed lin-
guistic corpora (Atkins et al., 1992), though also
including text types found on the Web which w ere
not anticipated in linguists’ corpus design discus-
sions. We do not want a “blind” sample dominated
by product listings, catalogues and computer sci-
entists’ bulletin boards. O ur pragmatic solution is
to query Google through its API service for ran-
dom pairs of randomly selected content words in
the target language. In preliminary experimenta-
tion, we found that single word queries yielded
many inappropriate pages (dictionary definitions
of the word, top pages of companies with the word
in their name), whereas combining more than two
words retrieved pages with lists of words, rather
than collected text.
Ueyama (2006) showed how queries for words
sampled from traditional written sources such as
newspaper text and published essays tend to yield
“public sphere” pages (online newspaper, govern-
ment and academic sites), whereas basic vocabu-
lary/everyday life words tend to yield “personal”
pages (blogs, bulletin boards). Since we wanted
both types, we obtained seed URLs with queries
2
bo.
it/wac/wac.php

87
for words from both kinds of sources. For Ger-
man, we sampled 2000 mid-frequency words from
a corpus of the S
¨
uddeutsche Zeitung newspaper
and paired them randomly. Then, we found a ba-
sic vocabulary list for German learners,
3
removed
function words and particles and built 653 random
pairs. We queried Google via its API retrieving
maximally 10 pages for each pair. We then col-
lapsed the URL list, insuring maximal sparseness
by keeping only one (randomly selected) URL for
each domain, leaving a list of 8626 seed URLs.
They were fed to the crawler.
The crawls are performed using the Her-
itrix crawler,
4
with a multi-threaded breadth-first
crawling strategy. The crawl is limited to pages
whose URL does not end in one of several suffixes
that cue non-html data (.pdf, .jpeg, etc.)
5
For
German, the crawl is limited to sites from the .de
and .at domains. Heritrix default crawling op-
tions are not modified in any other respect. We
let the German crawl run for ten days, retrieving

gzipped archives (the Heritrix output format) of
about 85GB.
3 Filtering
We undertake some post-processing on the ba-
sis of the Heritrix logs. We identify documents
of mime type text/html and size between 5
and 200KB. As observed by Fletcher (2004) very
small documents tend to contain little genuine text
(5KB counts as “very small” because of the html
code overhead) and very large documents tend to
be lists of various sorts, such as library indices,
store catalogues, etc. The logs also contain sha-
1 fingerprints, allowing us to identify perfect du-
plicates. After inspecting some of the duplicated
documents (about 50 pairs), we decided for a dras-
tic policy: if a document has at least one dupli-
cate, we discard not only the duplicate(s) but also
the document itself. We observed that, typically,
such documents came from the same site and were
warning messages, copyright statements and sim-
ilar, of limited or no linguistic interest. While the
strategy may lose some content, one of our gen-
eral principles is that, given how vast the Web is,
we can afford to privilege precision over recall.
All the documents that passed the pre-filtering
3
/>cusipage/
4

5

Further work should evaluate pros and cons of retrieving
documents in other formats, e.g., Adobe pdf.
stage are run through a perl program that performs
1) boilerplate stripping 2) function word filtering
3) porn fi ltering.
Boilerplate stripping
By “boilerplate” we mean all those components
of Web pages which are the same across many
pages. We include stripping out HTML markup,
javascript and other non-linguistic material in this
phase. We aimed to identify and remove sections
of a document that contain link lists, navigational
information, fixed notices, and other sections poor
in human-produced connected text. For purposes
of corpus construction, boilerplate removal is crit-
ical as it will distort statistics collected from the
corpus.
6
We adopted the heuristic used in the Hyp-
pia project BTE tool,
7
: content-rich sections of a
page will have a low html tag density, whereas
boilerplate is accompanied by a wealth of html
(because of special formatting, newlines, links,
etc.) The method is based on general properties
of Web documents, so is relatively independent of
language and crawling strategy.
Function word and pornography fi ltering
Connected text in sentences reliably contains a

high proportion of function words (Baroni, to ap-
pear), so, if a page does not meet this criterion
we reject it. The German function word list con-
tains 124 terms. We require that a minimum of 10
types and 30 tokens appear in a page, with a ra-
tio of function words to total words of at least one
quarter. The filter also works as a simple language
identifier.
8
Finally, we use a stop list of words likely to oc-
cur in pornographic Web pages, not out of prudery,
but because they tend to contain randomly gener-
ated text, long keyword lists and other linguisti-
cally problematic elements. We filter out docu-
ments that have at least three types or ten tokens
from a list of words highly used in pornography.
The list was derived from the analysis of porno-
graphic pages harvested in a previous crawl. This
is not entirely satisfactory, since some of the words
6
We note that this phase currently removes the links from
the text, so we can no l onger explore the graph structure of
the dataset. In future we may retain link structure, to support
research into the relation between it and linguistic character-
istics.
7
/>8
Of course, these simple methods will not filter out all
machine-generated text (typically produced as part of search
engine ranking scams or for other shady purposes); some-

times this appears to have been generated with a bigram lan-
guage model, and thus identifying it with automated tech-
niques is far from trivial.
88
in the list, taken in isolation, are wholly innocent
(fat, girls, tongue, etc.) We shall revisit the strat-
egy in due course.
This filtering took 5 days and resulted in a ver-
sion of the corpus containing 4.86M documents
for a total of 20GB of uncompressed data.
4 Near-duplicate detection
We use a simplified version of the “shingling” al-
gorithm (Broder et al., 1997). For each document,
after removing all function words, we take finger-
prints of a fixed number s of randomly selected n-
grams; then, for each pair of documents, we count
the number of shared n-grams, which can be seen
as an unbiased estimate of the overlap between the
two documents (Broder et al., 1997; Chakrabarti,
2002). We look for pairs of documents sharing
more than t n-grams, and we discard one of the
two.
After preliminary experimentation, we chose to
extract 25 5-grams from each document, and to
treat as near-duplicates documents that shared at
least two of these 5-grams. Near-duplicate spot-
ting on the German corpus took about 4 days.
2,466,271 near-duplicates were removed. The cor-
pus size decreased to 13GB. Most of the process-
ing time was spent in extracting the n-grams and

adding the corresponding fingerprints to the data-
base (which could be parallelized).
5 Part-of-speech tagging/lemmatization
and post-annotation cleaning
We performed German part-of-speech tagging and
lemmatization with TreeTagger.
9
Annotation took
5 days. The resulting corpus contains 2.13B
words, or 34GB of data including annotation.
After inspecting various documents from the
annotated corpus, we decided to perform a further
round of cleaning. There are two reasons for this:
first, we can exploit the annotation to find other
anomalous documents, through observing where
the distribution of parts-of-speech tags is very un-
usual and thus not likely to contain connected text.
Second, the TreeTagger was not trained on Web
data, and thus its performance on texts that are
heavy on Web-like usage (e.g., texts all in lower-
case, colloquial forms of inflected verbs, etc.) is
dismal. While a better solution to this second
problem would be to re-train the tagger on Web
9
/>projekte/corplex/TreeTagger
data (ultimately, the documents displaying the sec-
ond problem might be among the most interest-
ing ones to have in the corpus!), for now we try to
identify the most problematic documents through
automated criteria and discard them. The cues we

used included the number of words not recognised
by the lemmatizer; the proportion of words with
upper-case initial letters; proportion of nouns, and
proportion of sentence markers.
After this further processing step, the corpus
contains 1,870,259 documents from 10818 differ-
ent domains, and its final size is 1.71 billion to-
kens (26GB of data, with annotation). The final
size of the Italian corpus is 1,875,337 documents
and about 1.9 billion tokens.
6 Indexing and Web user interface
We believe that matters of efficient indexing and
user friendly interfacing will be crucial to the suc-
cess of our initiative, both because many linguists
will lack the relevant technical skills to write their
own corpus-access routines, and because w e shall
not publicly distribute the corpora for copyright
reasons; an advanced interface that allows lin-
guists to do actual research on the corpus (includ-
ing the possibility of saving settings and results
across sessions) will allow us to make the corpus
widely available while keeping it on our servers.
10
We are using the S ketch Engine,
11
a corpus query
tool which has been widely used in lexicography
and which supports queries combining regular ex-
pressions and boolean operators over words, lem-
mas and part-of-speech tags.

7 Comparison with other corpora
We would like to compare the German Web cor-
pus to an existing “balanced” corpus of G erman
attempting to represent a broad range of genres
and topics. Unfortunately, as far as we know no
resource of this sort is publicly available (which
is one of the reasons why we are interested in de-
veloping the German Web corpus in the first in-
stance.) Instead, w e use a corpus of newswire
articles from the Austria Presse Agentur (APA,
kindly provided to us by
¨
OFAI) as our reference
10
The legal situation is of course complex. We consider
that our case is equivalent to that of other search engines,
and that offering linguistically-encoded snippets of pages to
researchers does not go beyond the “fair use” terms routinely
invoked by search engine companies in relation to Web page
caching.
11
/>89
WEB APA
ich hier APA NATO
dass wir Schluß EU
und man Prozent Forts
sie nicht Mill AFP
ist das MRD Dollar
oder sind Wien Reuters
kann so Kosovo Dienstag

du mir DPA Mittwoch
wenn ein US Donnerstag
was da am sei
Table 1: Typical Web and APA words
point. This corpus contains 28M tokens, and,
despite its uniformity in terms of genre and re-
stricted thematic range, it has been successfully
employed as a general-purpose German corpus in
many projects. After basic regular-expression-
based normalization and filtering, the APA con-
tains about 500K word types, the Web corpus
about 7.4M. There is a large overlap among the 30
most frequent words in both corpora: 24 out of 30
words are shared. The non-overlapping words oc-
curring in the Web top 30 only are function words:
sie ‘she’, ich ‘I’, werden ‘become/be’, oder ‘or’,
sind ‘are’, er ‘he’. The words only in the APA
list show a bias towards newswire-specific vocab-
ulary (APA, Prozent ’percent’, Schluß ’closure’)
and temporal expressions that are also typical of
newswires (am ’at’, um ’on the’, nach ’after’).
Of the 232,322 hapaxes (words occurring only
once) in the APA corpus, 170,328 (73%) occur in
the Web corpus as well.
12
89% of these APA ha-
paxes occur more than once in the Web corpus,
suggesting how the Web data will help address
data sparseness issues.
Adopting the methodology of Sharoff (2006),

we then extracted the 20 words most characteris-
tics of the Web corpus vs. APA and vice versa,
based on the log-likelihood ratio association mea-
sure. Results are presented in Table 1. The APA
corpus has a strong bias towards newswire par-
lance (acronyms and named entities, temporal ex-
pressions, financial terms, toponyms), whereas the
terms that come out as most typical of the Web
corpus are function words that are not strongly
connected with any particular topic or genre. Sev-
eral of these top-ranked function words mark first
and second person forms (ich, du, wir, mir).
This preliminary comparison both functioned as
a “sanity check”, showing that there is consider-
12
Less than 1% of the Web corpus hapaxes are attested in
the APA corpus.
able overlap between our corpus and a smaller cor-
pus used in previous research, and suggested that
the Web corpus has more a higher proportion of
interpersonal material.
8 Conclusion
We have developed very large corpora from the
Web for German and Italian (with other languages
to follow). We have filtered and cleaned the text so
that the obvious problems with using the Web as a
corpus for linguistic research do not hold. Prelim-
inary evidence suggests the ’balance’ of our Ger-
man corpus compares favourably with that of a
newswire corpus (though of course any such claim

begs a number of open research questions about
corpus comparability). We have lemmatised and
part-of-speech-tagged the data and loaded it into
a corpus query tool supporting sophisticated lin-
guistic queries, and made it available to all.
References
B. Atkins, J. Clear, and N. Ostler. 1992. Corpus design
criteria. Literary and Linguistic Computing, 7:1–16.
M. Baroni. to appear. Distributions in text. In
A. L¨udeling and M. Kyt¨o, editors, Corpus lin-
guistics: An international handbook. Mouton de
Gruyter, Berlin .
A. Broder, S. Glassman, M. Manasse, an d G. Zw e ig.
1997. Syntactic clustering of the Web. In Proc.
Sixth International World-Wide Web Conference.
S. Chakrabarti. 2002. Mining the Web: Discovering
knowledge from hypertext data. Morgan Kaufmann,
San Francisco.
W. Fletcher. 2004. Making the web mor e useful as
a source for linguistic corpora. In U. Connor a nd
T. Upton, editors, Corpus Linguistics in North Amer-
ica 2002.
A. Kilgarriff and G. Grefenstette. 2003. Introd uction
to the spec ia l issue on the Web as corpus. Compu-
tational Linguistics, 29(3):3 33–34 7.
A. Kilgarriff. 2003. L inguistic search en gine. In
K. Simov, editor, Proc. SPROLAC Workshop, Lan-
caster.
S. Sharoff. 2006. Creating general-purpose corpor a
using automated search engine queries. In M. Ba-

roni and S. Bernardini, editors, WaCky! Working pa-
pers on the Web as Corpus. Ged it, Bologna.
M. Ueyama. 2006. Creation of general-purpose
Japanese Web corpora with different search engine
query strategies. In M. Baroni and S. Bernardini,
editors, WaCky! Working papers on the Web as Cor-
pus. Gedit, Bologna.
90