Guide-job-2p40s

Explanation of the task
When a user starts typing a query in a search engine, the system can often
guess what they were trying to type and fill in the rest of the query, saving the
user some work. The part of the query the user has typed so far is called
the prefix. The system’s guess about what the entire query would be if the
user had finished typing it is called the search completion, or just completion for
short.
You will be provided with a prefix along with the context information (such as geo-location and
search date) and two lists of completions presented side by side. After understanding the prefix,
you will be asked to answer your preference as one side over the other as well as helping us
flagging any issues you see for the completion(s).

Important Concepts
We will often use an arrow to indicate a prefix getting a particular completion.
For example, “[fac] → [facebook]” means the user typed the prefix “fac” and
the system suggested the completion “facebook.”


prefix — the part of a search query that the user has typed so far.
o A prefix will often end in the middle of a word or phrase.
o A prefix may also contain spelling and/or typographical errors.
o Example: A few possible prefixes you might see for users who intended to
type the query "facebook" are [fac], [face], [fcaebo], [facebook].



intended query — the entire query the user had in mind when they started typing.
o Sometimes most users typing the same prefix will have the same intended
query, such as [fac] → [facebook]. We call this a dominant intent.
o Other times different users will have different intended queries even
though they’ve typed the same prefix. For example, if the prefix is [wal],

some users might have [walgreens] in mind, while others were thinking of
[walmart]. We call this a split intent.



completion — the system’s prediction about the intended query.


o Occasionally the completion will be the same as the prefix. This means the
system’s best guess is that the user was done typing the query. e.g.
[instagram] → [instagram].


matching — how much the text of the prefix overlaps with the text of the
completion. Although users typically prefer completions that start with and include
the entire prefix (e.g. [ama] → [amazon]), there are some special cases where a
good completion may change the text, matching conceptually but not letter-forletter:
o Abbreviations: A good completion may expand or collapse initials: [uc
davis] → [university of california davis]
o Corrections: A good completion may correct spelling or typographical
errors: [fcaebook] → [facebook].
o Special Character Equivalents: In some languages, a user may type a
combination of letters instead of a single accented character. A good
completion may replace the combination with the correct character. For
example, in German, “oe” is another way to write the character ö, so a
good completion in locale de-DE might be: [moe] → [möllersdorf].



context — everything we know about the query besides the prefix itself, such as

the date of the query, the user’s location when the query was issued, and so on.
When deciding how good a completion is, you’ll need to take the context info
account.
o Example: The completion “[cafe in cam] → [cafe in campbell]“ might be
good if the searcher is in San Jose, CA, but not if they are in Boston, MA
(where ”cafe in cambridge“ would be a more likely completion).

General Guideline For Likelihood Rating
Before you begin the task, make sure your language and country settings align with the
task’s designated locale. Then, for each prefix and completion, you’ll do the following:
1. Try to determine the intended query for the prefix.
2. Mark any potential problems you see with the given prefix.
3. Decide how likely you think it is that the completion shown is the intended
query.See the sections below for details about how to do each step.

1. Determining the Intended Query


To determine the intended query, you’ll need to get a sense of what some
possible completions are. Click on the links to the web search engines and see
what some of their suggestions are. This will give you a sense of
some possible queries the user might have had in mind. However, remember
that the user may have been thinking of query that is not shown.
Remember to consider the context when thinking about the intended query. In
particular:


Part of the context is the user’s location. Think about how that might make a
certain completion more or less likely.



Part of the context is the date of the query. Consider what happened on or just
before that date. If there was a major news event, that might change which
completion is most likely.

2. Identifying Completion Problems
Indicate whether the completion has any of the following problems by checking
as many choices as apply:


Spelling Errors: Spelling, typographical, or grammatical errors.



Inappropriate: Inappropriate language, including profanity, explicit sexual
references, and hate speech.



Wrong language: Completion is in the wrong language, i.e. a language that is not
the locale language, AND not the name of an entity (person, place, title, etc.) that
the user was looking for.



Incomplete: Completion ends in the middle of a word, phrase, or name.



Intention too specific: Overly specific completion for broad prefix.



Gibberish: Completion is gibberish.

3. Deciding the Likelihood of a Completion
Grade each completion using the following scale, which answers the question
“How likely do you think it is that a user who typed this prefix had this completion in
mind?” (Or, in some cases where different users may have different intents,
“How likely do you think it is that many users who typed this prefix had this
completion in mind?”)




Highly Likely: It means search completion almost fully satisfies the
user's typing intent for the given prefix. Usually a completion is
highly likely with respect to a prefix when there are no other
completions you can think about or the it is the dominant
intent. e.g. [ama] --> [amazon], [fac] --> [facebook], [earring] -->
[earrings], [google.c]-->[google.com], [Vans] --> [Vans shoes]



Likely: It means the search completion satisfies partial intent of the
given prefix but such prefix probably does not has a best
possibility. Usually a completion is likely with repsect to a prefix
when there are only very few other completions you can think
about at first glance e.g. [wal] --> [walmart],[walgreen], [fetal

alcohol] --> [fetal alchohol syndrome], [fetal alchohol definition]



Somewhat Likely: The search completion matches with the prefix.
However, the semantic meaning of the completion deviates too much
from prefix, which makes it only satisfy the search intent for a few
people. Usually a completion is somewhat likely with repsect to
a prefix when there are more than a few other completions you
can think of e.g. [in] --> [instgram], [indeed], [internet], [india],
[integer], [how old is o] --> [how old is obama],[how old is oprah], [how
old is olivia], [how old is ozzy], [how old is oj], [1122] --> [1122 Brook
St.], [1122 Ocean Rd], [1122 angel number], [1122 Certral Ave]



Unlikely: Generally if the prefix is very broad, while the search
completion is too specific; e.g. [thu] --> [thursday night football]
(too specific); [dese] --> [deseandate frankie ruiz lyrics]); [what does it
mean w] -->[what does it mean when you left eye jumps], [what]->[what is dogecoin]



Highly Unlikely: If the search completion is gibberish,
incomplete, has severe spelling error (missing space in website
domain name, profane, wrong language). e.g. [how to] --> [how
what] (non sense completion), [esdfdofdfowngf] -->
[esdfdofdfowngf] (gibberish), [fucking big a] --> [fucking big
a] (inappropriate), [top 30 college coahce] --> [top 30 college
coahce] (spelling error), [bridge health home and ] --> [bridge health

home and ] (Incomplete), [fel] --> [felicidad] (happiness in Spanish)
[Wrong language], [gl] --> [glück] (happiness in German) [Wrong
language]

How to Handle Errors
Remember that a completion is trying to predict what the user intended their search
query to be. Because people don’t intend to make spelling/typographical/grammatical


errors, a completion containing an error is Highly Unlikely to be what the user wanted.
However, there are a few exceptions where an error is so minor that you do not need to
grade it Highly Unlikely. Instead, simply assign a grade one level lower than you would
have if the error were not present. These exceptions are:


When a word has multiple valid spellings, even though only one is technically
correct in the given context.
o Example: [huntington theater c] → [huntington theater company]




The actual name is “Huntington Theatre Company,” but since
“theater” and “theatre” are both valid spellings of the word in US
English, this counts as only a minor error.

When a word has regional/national variations in spelling across countries sharing
that language.
o Example: [driver’s lic] → [driver’s licence]




This completion uses a British/Canadian spelling of the word,
instead of the American “license.”

When the completion and the intended query differ only by a grammatical article
like “the” or “a”.
o Example: [andy weir ma] → [andy weir martian]




The actual name of the book is “The Martian,” but this completion
left out “the.”

When the completion and the intended query differ only by punctuation.
o Example: [olympics womens s] → [olympics womens soccer]


The correct punctuation would be “women’s”.[Note: All the above

examples assume locale is en-US.]
Nonstandard/Dialect Usage
A deliberate nonstandard, informal, slang or dialect usage is not an error.
Mistakenly “correcting” such usage is an error.


Example: [50 cent in d] → [50 cent in da club]

o A valid completion uses the intended song title, even though “da”
is not standard spelling/usage.

Task Starts!
The search prefix is [target deodant pack womens]
The search was completed in au/tasmania/launceston__pt_b; and
the search date is 2022-01-28

Please use the checkbox to select all issue(s) that apply for each
suggestion (if any of issues applies, the intent rating for the auto-completion
should not be better than "Unlikely")
Left
Only in this list

1
Prefix: [target deodant pack womens] --> Search Completion: [target deodant pack womens]
Inappropriate
Wrong Language
Too specific
Incomplete
Gibberish
Spelling error

How likely would the auto-completion satisfy the intent of the prefix? (Rule of thumb is the more the meaning
of auto-completion deviates from the given prefix, the less likely the auto-completion would satisfy the prefix's
intent. )
N/AHighly UnlikelyUnlikelySomewhat LikelyLikelyHighly Likely

First use following instrustions to rate the likelihood of

each side separately:
Option 1: If the completion is obviously (Highly) Likely (you
think there is a dominant intent for the
prefix) or (Highly) Unlikely (reasons can be too specific,
incomplete, non-sense, spelling errors, gibberish, inappropriate,
wrong language) to you --> ignore Option 2/3 and give a
likelihood rating accordingly.


Option 2: If you are not sure about the answer --> please click
here: 'Third party search link' to further look into this. Once landed,
click on the search box, type or copy the prefix into the box (DON'T click
search button or press 'enter') and you should see multiple suggestions in the
dropdown list.


If all of suggestions contain the completion, it's almost certain
that the completion is 'Highly Likely'



If at least half of suggestions contain the completion, it's safe
to rate the completion as 'Likely'



If less than half of suggestions contain the completion, the
completion is 'Somewhat Likely'



If no suggestions contain the completion, the completion
should be 'Unlikely'



If there are applicable issue flags other than 'Too specific',
the completion should 'Highly Unlikely'

Option 3 (Exception): If you don't think Option 1-2 apply to the case, use your
best judgement and explain your thought in the comment section.

Second compare the difference in ratings that you gave for
both sides and give a side-by-sde preference rating:
SBS rating scale instructions:
1. When the degree of likelihood from one side is two degree

higher then "Much better"

2. When the degree of likelihood from one side is one degree

higher then "Better"

3. When the degree of likelihood from one side fall into the same

bucket, most of the time, you can choose "About the same".
However, even if they are in the same bucket of likelihood,
but you still have your preference, then "Slightly better"

4. When one side is non-completion (empty), use the following

rule:




Non-completion side is 'Slightly Better' than a 'Somewhat
Likely' completion side



Non-completion side is 'Better' than
an 'Unlikely' completion side



Non-completion side is 'Much better' than a 'Highly
Unlikely' completion side



A 'Likely' completion is 'Better' than non-completion side



A 'Highly Likely' completion is 'Much Better' than non-completion side

Considering the overall search experience, which side would you prefer?
Much better
Better

Slightly better
About the same
Slightly better
Better
Much better
