Báo cáo khoa học: "A Discriminative Hierarchical Model for Fast Coreference at Large Scale" ppt

Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, pages 379–388,
Jeju, Republic of Korea, 8-14 July 2012.
c
2012 Association for Computational Linguistics
A Discriminative Hierarchical Model for Fast Coreference at Large Scale
Michael Wick
University of Massachsetts
140 Governor’s Drive
Amherst, MA

Sameer Singh
University of Massachusetts
140 Governor’s Drive
Amherst, MA

Andrew McCallum
University of Massachusetts
140 Governor’s Drive
Amherst, MA

Abstract
Methods that measure compatibility between
mention pairs are currently the dominant ap-
proach to coreference. However, they suffer
from a number of drawbacks including diffi-
culties scaling to large numbers of mentions
and limited representational power. As these
drawbacks become increasingly restrictive,
the need to replace the pairwise approaches
with a more expressive, highly scalable al-
ternative is becoming urgent. In this paper

we propose a novel discriminative hierarchical
model that recursively partitions entities into
trees of latent sub-entities. These trees suc-
cinctly summarize the mentions providing a
highly compact, information-rich structure for
reasoning about entities and coreference un-
certainty at massive scales. We demonstrate
that the hierarchical model is several orders
of magnitude faster than pairwise, allowing us
to perform coreference on six million author
mentions in under four hours on a single CPU.
1 Introduction
Coreference resolution, the task of clustering men-
tions into partitions representing their underlying
real-world entities, is fundamental for high-level in-
formation extraction and data integration, including
semantic search, question answering, and knowl-
edge base construction. For example, coreference
is vital for determining author publication lists in
bibliographic knowledge bases such as CiteSeer and
Google Scholar, where the repository must know
if the “R. Hamming” who authored “Error detect-
ing and error correcting codes” is the same” “R.
Hamming” who authored “The unreasonable effec-
tiveness of mathematics.” Features of the mentions
(e.g., bags-of-words in titles, contextual snippets
and co-author lists) provide evidence for resolving
such entities.
Over the years, various machine learning tech-
niques have been applied to different variations of

the coreference problem. A commonality in many
of these approaches is that they model the prob-
lem of entity coreference as a collection of deci-
sions between mention pairs (Bagga and Baldwin,
1999; Soon et al., 2001; McCallum and Wellner,
2004; Singla and Domingos, 2005; Bengston and
Roth, 2008). That is, coreference is solved by an-
swering a quadratic number of questions of the form
“does mention A refer to the same entity as mention
B?” with a compatibility function that indicates how
likely A and B are coreferent. While these models
have been successful in some domains, they also ex-
hibit several undesirable characteristics. The first is
that pairwise models lack the expressivity required
to represent aggregate properties of the entities. Re-
cent work has shown that these entity-level prop-
erties allow systems to correct coreference errors
made from myopic pairwise decisions (Ng, 2005;
Culotta et al., 2007; Yang et al., 2008; Rahman and
Ng, 2009; Wick et al., 2009), and can even provide
a strong signal for unsupervised coreference (Bhat-
tacharya and Getoor, 2006; Haghighi and Klein,
2007; Haghighi and Klein, 2010).
A second problem, that has received significantly
less attention in the literature, is that the pair-
wise coreference models scale poorly to large col-
lections of mentions especially when the expected
379
Name!"#$%&'"($))$*"
Ins(tu(ons:-(+, /01"

Topics:2333-"04-"506047"
Name!#$%&'"($))$*"
Ins(tu(ons:"
Topics:-04"
Name!"#1"($))$*"
Ins(tu(ons:-(+, /0"
Topics:-333"
Name!"#1"($))$*"
Ins(tu(ons: /0"
Topics:-333"
Name!"#$%'8"($))$*"
Ins(tu(ons:-(+,"
Topics:2333-"04-")$9:';8<$)'7"
Coref?-
#$%&'"($))$*"
Topics:-04"
#1"($))$*"
Inst: /0"
#1"($))$*"
Topic:-333"
#1"($))$*"
Inst:-(+,"
#$%&'"($))$*"
Topics:-04"
#1"($))$*"
Inst:-(+,"
#$%'8"($))$*"
Topics:-333"
0*8=!(+,"
#1"($))$*"

Topics:-04"
Inst:-(+,"
#1"($))$*"
Topics: ;5"
Figure 1: Discriminative hierarchical factor graph for coreference: Latent entity nodes (white boxes)
summarize subtrees. Pairwise factors (black squares) measure compatibilities between child and parent
nodes, avoiding quadratic blow-up. Corresponding decision variables (open circles) indicate whether one
node is the child of another. Mentions (gray boxes) are leaves. Deciding whether to merge these two entities
requires evaluating just a single factor (red square), corresponding to the new child-parent relationship.
number of mentions in each entity cluster is also
large. Current systems cope with this by either
dividing the data into blocks to reduce the search
space (Hern
´
andez and Stolfo, 1995; McCallum et
al., 2000; Bilenko et al., 2006), using fixed heuris-
tics to greedily compress the mentions (Ravin and
Kazi, 1999; Rao et al., 2010), employing special-
ized Markov chain Monte Carlo procedures (Milch
et al., 2006; Richardson and Domingos, 2006; Singh
et al., 2010), or introducing shallow hierarchies of
sub-entities for MCMC block moves and super-
entities for adaptive distributed inference (Singh et
al., 2011). However, while these methods help man-
age the search space for medium-scale data, eval-
uating each coreference decision in many of these
systems still scales linearly with the number of men-
tions in an entity, resulting in prohibitive computa-
tional costs associated with large datasets. This scal-
ing with the number of mentions per entity seems

particularly wasteful because although it is common
for an entity to be referenced by a large number
of mentions, many of these coreferent mentions are
highly similar to each other. For example, in author
coreference the two most common strings that refer
to Richard Hamming might have the form “R. Ham-
ming” and “Richard Hamming.” In newswire coref-
erence, a prominent entity like Barack Obama may
have millions of “Obama” mentions (many occur-
ring in similar semantic contexts). Deciding whether
a mention belongs to this entity need not involve
comparisons to all contextually similar “Obama”
mentions; rather we prefer a more compact repre-
sentation in order to efficiently reason about them.
In this paper we propose a novel hierarchical dis-
criminative factor graph for coreference resolution
that recursively structures each entity as a tree of la-
tent sub-entities with mentions at the leaves. Our
hierarchical model avoids the aforementioned prob-
lems of the pairwise approach: not only can it jointly
reason about attributes of entire entities (using the
power of discriminative conditional random fields),
but it is also able to scale to datasets with enor-
mous numbers of mentions because scoring enti-
ties does not require computing a quadratic number
of compatibility functions. The key insight is that
each node in the tree functions as a highly compact
information-rich summary of its children. Thus, a
small handful of upper-level nodes may summarize
millions of mentions (for example, a single node

may summarize all contextually similar “R. Ham-
ming” mentions). Although inferring the structure
of the entities requires reasoning over a larger state-
space, the latent trees are actually beneficial to in-
ference (as shown for shallow trees in Singh et
al. (2011)), resulting in rapid progress toward high
probability regions, and mirroring known benefits
of auxiliary variable methods in statistical physics
(such as Swendsen and Wang (1987)). Moreover,
380
each step of inference is computationally efficient
because evaluating the cost of attaching (or detach-
ing) sub-trees requires computing just a single com-
patibility function (as seen in Figure 1). Further,
our hierarchical approach provides a number of ad-
ditional advantages. First, the recursive nature of the
tree (arbitrary depth and width) allows the model to
adapt to different types of data and effectively com-
press entities of different scales (e.g., entities with
more mentions may require a deeper hierarchy to
compress). Second, the model contains compatibil-
ity functions at all levels of the tree enabling it to si-
multaneously reason at multiple granularities of en-
tity compression. Third, the trees can provide split
points for finer-grained entities by placing contex-
tually similar mentions under the same subtree. Fi-
nally, if memory is limited, redundant mentions can
be pruned by replacing subtrees with their roots.
Empirically, we demonstrate that our model is
several orders of magnitude faster than a pairwise

model, allowing us to perform efficient coreference
on nearly six million author mentions in under four
hours using a single CPU.
2 Background: Pairwise Coreference
Coreference is the problem of clustering mentions
such that mentions in the same set refer to the same
real-world entity; it is also known as entity disam-
biguation, record linkage, and de-duplication. For
example, in author coreference, each mention might
be represented as a record extracted from the author
field of a textual citation or BibTeX record. The
mention record may contain attributes for the first,
middle, and last name of the author, as well as con-
textual information occurring in the citation string,
co-authors, titles, topics, and institutions. The goal
is to cluster these mention records into sets, each
containing all the mentions of the author to which
they refer; we use this task as a running pedagogical
example.
Let M be the space of observed mention records;
then the traditional pairwise coreference approach
scores candidate coreference solutions with a com-
patibility function ψ : M × M →  that mea-
sures how likely it is that the two mentions re-
fer to the same entity.
1
In discriminative log-
1
We can also include an incompatibility function for when
linear models, the function ψ takes the form of

weights θ on features φ(m
i
, m
j
), i.e., ψ(m
i
, m
j
) =
exp (θ · φ(m
i
, m
j
)). For example, in author coref-
erence, the feature functions φ might test whether
the name fields for two author mentions are string
identical, or compute cosine similarity between the
two mentions’ bags-of-words, each representing a
mention’s context. The corresponding real-valued
weights θ determine the impact of these features on
the overall pairwise score.
Coreference can be solved by introducing a set of
binary coreference decision variables for each men-
tion pair and predicting a setting to their values that
maximizes the sum of pairwise compatibility func-
tions. While it is possible to independently make
pairwise decisions and enforce transitivity post hoc,
this can lead to poor accuracy because the decisions
are tightly coupled. For higher accuracy, a graphi-
cal model such as a conditional random field (CRF)

is constructed from the compatibility functions to
jointly reason about the pairwise decisions (McCal-
lum and Wellner, 2004). We now describe the pair-
wise CRF for coreference as a factor graph.
2.1 Pairwise Conditional Random Field
Each mention m
i
∈ M is an observed variable, and
for each mention pair (m
i
, m
j
) we have a binary
coreference decision variable y
ij
whose value de-
termines whether m
i
and m
j
refer to the same en-
tity (i.e., 1 means they are coreferent and 0 means
they are not coreferent). The pairwise compatibility
functions become the factors in the graphical model.
Each factor examines the properties of its mention
pair as well as the setting to the coreference decision
variable and outputs a score indicating how likely
the setting of that coreference variable is. The joint
probability distribution over all possible settings to
the coreference decision variables (y) is given as a

product of all the pairwise compatibility factors:
P r(y|m) ∝
n

i=1
n

j=1
ψ(m
i
, m
j
, y
ij
) (1)
Given the pairwise CRF, the problem of coreference
is then solved by searching for the setting of the
coreference decision variables that has the highest
probability according to Equation 1 subject to the
the mentions are not coreferent, e.g., ψ : M × M × {0, 1} → 
381
Jamie,Callan,
Jamie,Callan,
J.,Callan,
J.,Callan,
J.,Callan,
J.,Callan,
Jamie,Callan, Jamie,Callan,
v,
Jamie,Callan,

J.,Callan,
v,v,v,
J.,Callan,
J.,Callan,
J.,Callan,
J.,Callan,
Jamie,Callan,
Figure 2: Pairwise model on six mentions: Open
circles are the binary coreference decision variables,
shaded circles are the observed mentions, and the
black boxes are the factors of the graphical model
that encode the pairwise compatibility functions.
constraint that the setting to the coreference vari-
ables obey transitivity;
2
this is the maximum proba-
bility estimate (MPE) setting. However, the solution
to this problem is intractable, and even approximate
inference methods such as loopy belief propagation
can be difficult due to the cubic number of determin-
istic transitivity constraints.
2.2 Approximate Inference
An approximate inference framework that has suc-
cessfully been used for coreference models is
Metropolis-Hastings (MH) (Milch et al. (2006), Cu-
lotta and McCallum (2006), Poon and Domingos
(2007), amongst others), a Markov chain Monte
Carlo algorithm traditionally used for marginal in-
ference, but which can also be tuned for MPE in-
ference. MH is a flexible framework for specify-

ing customized local-search transition functions and
provides a principled way of deciding which local
search moves to accept. A proposal function q takes
the current coreference hypothesis and proposes a
new hypothesis by modifying a subset of the de-
cision variables. The proposed change is accepted
with probability α:
α = min

1,
P r(y

)
P r(y)
q(y|y

)
q(y

|y)

(2)
2
We say that a full assignment to the coreference variables
y obeys transitivity if ∀ ijk y
ij
= 1 ∧ y
jk
= 1 =⇒ y
ik

= 1
When using MH for MPE inference, the second term
q(y|y

)/q(y

|y) is optional, and usually omitted.
Moves that reduce model score may be accepted and
an optional temperature can be used for annealing.
The primary advantages of MH for coreference are
(1) only the compatibility functions of the changed
decision variables need to be evaluated to accept a
move, and (2) the proposal function can enforce the
transitivity constraint by exploring only variable set-
tings that result in valid coreference partitionings.
A commonly used proposal distribution for coref-
erence is the following: (1) randomly select two
mentions (m
i
, m
j
), (2) if the mentions (m
i
, m
j
) are
in the same entity cluster according to y then move
one mention into a singleton cluster (by setting the
necessary decision variables to 0), otherwise, move
mention m

i
so it is in the same cluster as m
j
(by
setting the necessary decision variables). Typically,
MH is employed by first initializing to a singleton
configuration (all entities have one mention), and
then executing the MH for a certain number of steps
(or until the predicted coreference hypothesis stops
changing).
This proposal distribution always moves a sin-
gle mention m from some entity e
i
to another en-
tity e
j
and thus the configuration y and y

only dif-
fer by the setting of decision variables governing to
which entity m refers. In order to guarantee transi-
tivity and a valid coreference equivalence relation,
we must properly remove m from e
i
by untethering
m from each mention in e
i
(this requires computing
|e
i

| − 1 pairwise factors). Similarly—again, for the
sake of transitivity—in order to complete the move
into e
j
we must coref m to each mention in e
j
(this
requires computing |e
j
| pairwise factors). Clearly,
all the other coreference decision variables are in-
dependent and so their corresponding factors can-
cel because they yield the same scores under y and
y

. Thus, evaluating each proposal for the pairwise
model scales linearly with the number of mentions
assigned to the entities, requiring the evaluation of
2(|e
i
| + |e
j
| − 1) compatibility functions (factors).
3 Hierarchical Coreference
Instead of only capturing a single coreference clus-
tering between mention pairs, we can imagine mul-
tiple levels of coreference decisions over different
382
granularities. For example, mentions of an author
may be further partitioned into semantically similar

sets, such that mentions from each set have topically
similar papers. This partitioning can be recursive,
i.e., each of these sets can be further partitioned, cap-
turing candidate splits for an entity that can facilitate
inference. In this section, we describe a model that
captures arbitrarily deep hierarchies over such lay-
ers of coreference decisions, enabling efficient in-
ference and rich entity representations.
3.1 Discriminative Hierarchical Model
In contrast to the pairwise model, where each en-
tity is a flat cluster of mentions, our proposed model
structures each entity recursively as a tree. The
leaves of the tree are the observed mentions with
a set of attribute values. Each internal node of the
tree is latent and contains a set of unobserved at-
tributes; recursively, these node records summarize
the attributes of their child nodes (see Figure 1), for
example, they may aggregate the bags of context
words of the children. The root of each tree repre-
sents the entire entity, with the leaves containing its
mentions. Formally, the coreference decision vari-
ables in the hierarchical model no longer represent
pairwise decisions directly. Instead, a decision vari-
able y
r
i
,r
j
= 1 indicates that node-record r
j

is the
parent of node-record r
i
. We say a node-record ex-
ists if either it is a mention, has a parent, or has at
least one child. Let R be the set of all existing node
records, let r
p
denote the parent for node r, that is
y
r,r
p
= 1, and ∀r

= r
p
, y
r,r

= 0. As we describe
in more detail later, the structure of the tree and the
values of the unobserved attributes are determined
during inference.
In order to represent our recursive model of coref-
erence, we include two types of factors: pairwise
factors ψ
pw
that measure compatibility between a
child node-record and its parent, and unit-wise fac-
tors ψ

rw
that measure compatibilities of the node-
records themselves. For efficiency we enforce that
parent-child factors only produce a non-zero score
when the corresponding decision variable is 1. The
unit-wise factors can examine compatibility of set-
tings to the attribute variables for a particular node
(for example, the set of topics may be too diverse
to represent just a single entity), as well as enforce
priors over the tree’s breadth and depth. Our recur-
sive hierarchical model defines the probability of a
configuration as:
P r(y, R|m) ∝

r∈R
ψ
rw
(r)ψ
pw
(r, r
p
) (3)
3.2 MCMC Inference for Hierarchical models
The state space of our hierarchical model is substan-
tially larger (theoretically infinite) than the pairwise
model due to the arbitrarily deep (and wide) latent
structure of the cluster trees. Inference must simul-
taneously determine the structure of the tree, the la-
tent node-record values, as well as the coreference
decisions themselves.

While this may seem daunting, the structures be-
ing inferred are actually beneficial to inference. In-
deed, despite the enlarged state space, inference
in the hierarchical model is substantially faster
than a pairwise model with a smaller state space.
One explanatory intuition comes from the statisti-
cal physics community: we can view the latent tree
as auxiliary variables in a data-augmentation sam-
pling scheme that guide MCMC through the state
space more efficiently. There is a large body of lit-
erature in the statistics community describing how
these auxiliary variables can lead to faster conver-
gence despite the enlarged state space (classic exam-
ples include Swendsen and Wang (1987) and slice
samplers (Neal, 2000)).
Further, evaluating each proposal during infer-
ence in the hierarchical model is substantially faster
than in the pairwise model. Indeed, we can replace
the linear number of factor evaluations (as in the
pairwise model) with a constant number of factor
evaluations for most proposals (for example, adding
a subtree requires re-evaluating only a single parent-
child factor between the subtree and the attachment
point, and a single node-wise factor).
Since inference must determine the structure of
the entity trees in addition to coreference, it is ad-
vantageous to consider multiple MH proposals per
sample. Therefore, we employ a modified variant
of MH that is similar to multi-try Metropolis (Liu
et al., 2000). Our modified MH algorithm makes k

proposals and samples one according to its model
ratio score (the first term in Equation 2) normalized
across all k. More specificaly, for each MH step, we
first randomly select two subtrees headed by node-
383
records r
i
and r
j
from the current coreference hy-
pothesis. If r
i
and r
j
are in different clusters, we
propose several alternate merge operations: (also in
Figure 3):
• Merge Left - merges the entire subtree of r
j
into
node r
i
by making r
j
a child of r
i
• Merge Entity Left - merges r
j
with r
i

’s root
• Merge Left and Collapse - merges r
j
into r
i
then
performs a collapse on r
j
(see below).
• Merge Up - merges node r
i
with node r
j
by cre-
ating a new parent node-record variable r
p
with r
i
and r
j
as the children. The attribute fields of r
p
are
selected from r
i
and r
j
.
Otherwise r
i

and r
j
are subtrees in the same entity
tree, then the following proposals are used instead:
• Split Right - Make the subtree r
j
the root of a new
entity by detaching it from its parent
• Collapse - If r
i
has a parent, then move r
i
’s chil-
dren to r
i
’s parent and then delete r
i
.
• Sample attribute - Pick a new value for an at-
tribute of r
i
from its children.
Computing the model ratio for all of coreference
proposals requires only a constant number of com-
patibility functions. On the other hand, evaluating
proposals in the pairwise model requires evaluat-
ing a number of compatibility functions equal to the
number of mentions in the clusters being modified.
Note that changes to the attribute values of the
node-record and collapsing still require evaluating

a linear number of factors, but this is only linear in
the number of child nodes, not linear in the number
of mentions referring to the entity. Further, attribute
values rarely change once the entities stabilize. Fi-
nally, we incrementally update bags during corefer-
ence to reflect the aggregates of their children.
4 Experiments: Author Coreference
Author coreference is a tremendously important
task, enabling improved search and mining of sci-
entific papers by researchers, funding agencies, and
governments. The problem is extremely difficult due
to the wide variations of names, limited contextual
evidence, misspellings, people with common names,
lack of standard citation formats, and large numbers
of mentions.
For this task we use a publicly available collec-
tion of 4,394 BibTeX files containing 817,193 en-
tries.
3
We extract 1,322,985 author mentions, each
containing first, middle, last names, bags-of-words
of paper titles, topics in paper titles (by running la-
tent Dirichlet allocation (Blei et al., 2003)), and last
names of co-authors. In addition we include 2,833
mentions from the REXA dataset
4
labeled for coref-
erence, in order to assess accuracy. We also include
∼5 million mentions from DBLP.
4.1 Models and Inference

Due to the paucity of labeled training data, we did
not estimate parameters from data, but rather set
the compatibility functions manually by specifying
their log scores. The pairwise compatibility func-
tions punish a string difference in first, middle, and
last name, (−8); reward a match (+2); and reward
matching initials (+1). Additionally, we use the co-
sine similarity (shifted and scaled between −4 and
4) between the bags-of-words containing title to-
kens, topics, and co-author last names. These com-
patibility functions define the scores of the factors
in the pairwise model and the parent-child factors
in the hierarchical model. Additionally, we include
priors over the model structure. We encourage each
node to have eight children using a per node factor
having score 1/(|number of children−8|+1), manage
tree depth by placing a cost on the creation of inter-
mediate tree nodes −8 and encourage clustering by
placing a cost on the creation of root-level entities
−7. These weights were determined by just a few
hours of tuning on a development set.
We initialize the MCMC procedures to the single-
ton configuration (each entity consists of one men-
tion) for each model, and run the MH algorithm de-
scribed in Section 2.2 for the pairwise model and
multi-try MH (described in Section 3.2) for the hi-
erarchical model. We augment these samplers us-
ing canopies constructed by concatenating the first
initial and last name: that is, mentions are only
selected from within the same canopy (or block)

to reduce the search space (Bilenko et al., 2006).
During the course of MCMC inference, we record
the pairwise F1 scores of the labeled subset. The
source code for our model is available as part of the
FACTORIE package (McCallum et al., 2009, http:
3
/>4
/>aculotta/data/rexa.html
384
!
"#
!
$#
!
%#
!
"#
!
$#
!
"#
!
$#
!
$#
&
"#
!
"#
!

$#
&
"#
&
"#
!"#$%&'()%)*' +*,-*'.*/' +*,-*'0"$)1'.*/' +*,-*'23' +*,-*'.*/'%"4'56&&%3(*'
!
"#
!"#$%&'7)%)*'
&
"#
&
$#
!'
"#
&
"#
&
$#
!'
"#
73&#)8#-9)'
&
$#
!'
"#
56&&%3(*'
Figure 3: Example coreference proposals for the case where r
i
and r

j
are initially in different clusters.
//factorie.cs.umass.edu/).
4.2 Comparison to Pairwise Model
In Figure 4a we plot the number of samples com-
pleted over time for a 145k subset of the data. Re-
call that we initialized to the singleton configuration
and that as the size of the entities grows, the cost of
evaluating the entities in MCMC becomes more ex-
pensive. The pairwise model struggles with the large
cluster sizes while the hierarchical model is hardly
affected. Even though the hierarchical model is eval-
uating up to four proposals for each sample, it is still
able to sample much faster than the pairwise model;
this is expected because the cost of evaluating a pro-
posal requires evaluating fewer factors. Next, we
plot coreference F1 accuracy over time and show in
Figure 5a that the prolific sampling rate of the hierar-
chical model results in faster coreference. Using the
plot, we can compare running times for any desired
level of accuracy. For example, on the 145k men-
tion dataset, at a 60% accuracy level the hierarchical
model is 19 times faster and at 90% accuracy it is
31 times faster. These performance improvements
are even more profound on larger datasets: the hi-
erarchical model achieves a 60% level of accuracy
72 times faster than the pairwise model on the 1.3
million mention dataset, reaching 90% in just 2,350
seconds. Note, however, that the hierarchical model
requires more samples to reach a similar level of ac-

curacy due to the larger state space (Figure 4b).
4.3 Large Scale Experiments
In order to demonstrate the scalability of the hierar-
chical model, we run it on nearly 5 million author
mentions from DBLP. In under two hours (6,700
seconds), we achieve an accuracy of 80%, and in
under three hours (10,600 seconds), we achieve an
accuracy of over 90%. Finally, we combine DBLP
with BibTeX data to produce a dataset with almost 6
million mentions (5,803,811). Our performance on
this dataset is similar to DBLP, taking just 13,500
seconds to reach a 90% accuracy.
5 Related Work
Singh et al. (2011) introduce a hierarchical model
for coreference that treats entities as a two-tiered
structure, by introducing the concept of sub-entities
and super-entities. Super-entities reduce the search
space in order to propose fruitful jumps. Sub-
entities provide a tighter granularity of coreference
and can be used to perform larger block moves dur-
ing MCMC. However, the hierarchy is fixed and
shallow. In contrast, our model can be arbitrarily
deep and wide. Even more importantly, their model
has pairwise factors and suffers from the quadratic
curse, which they address by distributing inference.
The work of Rao et al. (2010) uses streaming
clustering for large-scale coreference. However, the
greedy nature of the approach does not allow errors
to be revisited. Further, they compress entities by
averaging their mentions’ features. We are able to

provide richer entity compression, the ability to re-
visit errors, and scale to larger data.
Our hierarchical model provides the advantages
of recently proposed entity-based coreference sys-
tems that are known to provide higher accuracy
(Haghighi and Klein, 2007; Culotta et al., 2007;
Yang et al., 2008; Wick et al., 2009; Haghighi and
Klein, 2010). However, these systems reason over a
single layer of entities and do not scale well.
Techniques such as lifted inference (Singla and
Domingos, 2008) for graphical models exploit re-
dundancy in the data, but typically do not achieve
any significant compression on coreference data be-
385
Samples versus Time
0 500 1,000 1,500 2,000
Running time (s)
0
50,000
100,000
150,000
200,000
250,000
300,000
350,000
400,000
Number of Samples
Hierar Pairwise
(a) Sampling Performance
Accuracy versus Samples

0 50,000 100,000 150,000 200,000
Number of Samples
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
F1 Accuracy
Hierar Pairwise
(b) Accuracy vs. samples (convergence accuracy as dashes)
Figure 4: Sampling Performance Plots for 145k mentions
Accuracy versus Time
0 250 500 750 1,000 1,250 1,500 1,750 2,000
Running time (s)
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9

1.0
F1 Accuracy
Hierar Pairwise
(a) Accuracy vs. time (145k mentions)
Accuracy versus Time
0 10,000 20,000 30,000 40,000 50,000 60,000
Running time (s)
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
F1 Accuracy
Hierar Pairwise
(b) Accuracy vs. time (1.3 million mentions)
Figure 5: Runtime performance on two datasets
cause the observations usually violate any symmetry
assumptions. On the other hand, our model is able
to compress similar (but potentially different) obser-
vations together in order to make inference fast even
in the presence of asymmetric observed data.
6 Conclusion
In this paper we present a new hierarchical model
for large scale coreference and demonstrate it on
the problem of author disambiguation. Our model

recursively defines an entity as a summary of its
children nodes, allowing succinct representations of
millions of mentions. Indeed, inference in the hier-
archy is orders of magnitude faster than a pairwise
CRF, allowing us to infer accurate coreference on
six million mentions on one CPU in just 4 hours.
7 Acknowledgments
We would like to thank Veselin Stoyanov for his feed-
back. This work was supported in part by the CIIR, in
part by ARFL under prime contract #FA8650-10-C-7059,
in part by DARPA under AFRL prime contract #FA8750-
09-C-0181, and in part by IARPA via DoI/NBC contract
#D11PC20152. The U.S. Government is authorized to
reproduce and distribute reprints for Governmental pur-
poses notwithstanding any copyright annotation thereon.
Any opinions, findings and conclusions or recommenda-
tions expressed in this material are those of the authors
and do not necessarily reflect those of the sponsor.
386
References
Amit Bagga and Breck Baldwin. 1999. Cross-document
event coreference: annotations, experiments, and ob-
servations. In Proceedings of the Workshop on Coref-
erence and its Applications, CorefApp ’99, pages 1–8,
Stroudsburg, PA, USA. Association for Computational
Linguistics.
Eric Bengston and Dan Roth. 2008. Understanding
the value of features for coreference resolution. In
Empirical Methods in Natural Language Processing
(EMNLP).

Indrajit Bhattacharya and Lise Getoor. 2006. A latent
Dirichlet model for unsupervised entity resolution. In
SDM.
Mikhail Bilenko, Beena Kamath, and Raymond J.
Mooney. 2006. Adaptive blocking: Learning to scale
up record linkage. In Proceedings of the Sixth Interna-
tional Conference on Data Mining, ICDM ’06, pages
87–96, Washington, DC, USA. IEEE Computer Soci-
ety.
David M. Blei, Andrew Y. Ng, and Michael I. Jordan.
2003. Latent Dirichlet allocation. Journal on Machine
Learning Research, 3:993–1022.
Aron Culotta and Andrew McCallum. 2006. Prac-
tical Markov logic containing first-order quantifiers
with application to identity uncertainty. In Human
Language Technology Workshop on Computationally
Hard Problems and Joint Inference in Speech and Lan-
guage Processing (HLT/NAACL), June.
Aron Culotta, Michael Wick, and Andrew McCallum.
2007. First-order probabilistic models for coreference
resolution. In North American Chapter of the Associa-
tion for Computational Linguistics - Human Language
Technologies (NAACL HLT).
Aria Haghighi and Dan Klein. 2007. Unsupervised
coreference resolution in a nonparametric bayesian
model. In Annual Meeting of the Association for Com-
putational Linguistics (ACL), pages 848–855.
Aria Haghighi and Dan Klein. 2010. Coreference reso-
lution in a modular, entity-centered model. In North
American Chapter of the Association for Computa-

tional Linguistics - Human Language Technologies
(NAACL HLT), pages 385–393.
Mauricio A. Hern
´
andez and Salvatore J. Stolfo. 1995.
The merge/purge problem for large databases. In Pro-
ceedings of the 1995 ACM SIGMOD international
conference on Management of data, SIGMOD ’95,
pages 127–138, New York, NY, USA. ACM.
Jun S. Liu, Faming Liang, and Wing Hung Wong. 2000.
The multiple-try method and local optimization in
metropolis sampling. Journal of the American Statis-
tical Association, 96(449):121–134.
Andrew McCallum and Ben Wellner. 2004. Conditional
models of identity uncertainty with application to noun
coreference. In Neural Information Processing Sys-
tems (NIPS).
Andrew McCallum, Kamal Nigam, and Lyle Ungar.
2000. Efficient clustering of high-dimensional data
sets with application to reference matching. In In-
ternational Conference on Knowledge Discovery and
Data Mining (KDD), pages 169–178.
Andrew McCallum, Karl Schultz, and Sameer Singh.
2009. FACTORIE: Probabilistic programming via im-
peratively defined factor graphs. In Neural Informa-
tion Processing Systems (NIPS).
Brian Milch, Bhaskara Marthi, and Stuart Russell. 2006.
BLOG: Relational Modeling with Unknown Objects.
Ph.D. thesis, University of California, Berkeley.
Radford Neal. 2000. Slice sampling. Annals of Statis-

tics, 31:705–767.
Vincent Ng. 2005. Machine learning for coreference res-
olution: From local classification to global ranking. In
Annual Meeting of the Association for Computational
Linguistics (ACL).
Hoifung Poon and Pedro Domingos. 2007. Joint infer-
ence in information extraction. In AAAI Conference
on Artificial Intelligence, pages 913–918.
Altaf Rahman and Vincent Ng. 2009. Supervised mod-
els for coreference resolution. In Proceedings of the
2009 Conference on Empirical Methods in Natural
Language Processing: Volume 2 - Volume 2, EMNLP
’09, pages 968–977, Stroudsburg, PA, USA. Associa-
tion for Computational Linguistics.
Delip Rao, Paul McNamee, and Mark Dredze. 2010.
Streaming cross document entity coreference reso-
lution. In International Conference on Computa-
tional Linguistics (COLING), pages 1050–1058, Bei-
jing, China, August. Coling 2010 Organizing Commit-
tee.
Yael Ravin and Zunaid Kazi. 1999. Is Hillary Rodham
Clinton the president? disambiguating names across
documents. In Annual Meeting of the Association for
Computational Linguistics (ACL), pages 9–16.
Matthew Richardson and Pedro Domingos. 2006.
Markov logic networks. Machine Learning, 62(1-
2):107–136.
Sameer Singh, Michael L. Wick, and Andrew McCallum.
2010. Distantly labeling data for large scale cross-
document coreference. Computing Research Reposi-

tory (CoRR), abs/1005.4298.
Sameer Singh, Amarnag Subramanya, Fernando Pereira,
and Andrew McCallum. 2011. Large-scale cross-
document coreference using distributed inference and
hierarchical models. In Association for Computa-
tional Linguistics: Human Language Technologies
(ACL HLT).
387
Parag Singla and Pedro Domingos. 2005. Discrimina-
tive training of Markov logic networks. In AAAI, Pitts-
burgh, PA.
Parag Singla and Pedro Domingos. 2008. Lifted first-
order belief propagation. In Proceedings of the 23rd
national conference on Artificial intelligence - Volume
2, AAAI’08, pages 1094–1099. AAAI Press.
Wee Meng Soon, Hwee Tou Ng, and Daniel Chung Yong
Lim. 2001. A machine learning approach to coref-
erence resolution of noun phrases. Comput. Linguist.,
27(4):521–544.
R.H. Swendsen and J.S. Wang. 1987. Nonuniversal crit-
ical dynamics in MC simulations. Phys. Rev. Lett.,
58(2):68–88.
Michael Wick, Aron Culotta, Khashayar Rohanimanesh,
and Andrew McCallum. 2009. An entity-based model
for coreference resolution. In SIAM International
Conference on Data Mining (SDM).
Xiaofeng Yang, Jian Su, Jun Lang, Chew Lim Tan, Ting
Liu, and Sheng Li. 2008. An entity-mention model for
coreference resolution with inductive logic program-
ming. In Association for Computational Linguistics,

pages 843–851.
388