bài viết hay 3

The interpretation of the query [iphone], English (US) has changed over time as new iPhone models are released. The first iPhone was introduced in 2007.  Users searching for [iphone], English (US) at that time were looking for the new (at the time) first iPhone model. Most users now are looking for the most recent or upcoming iPhone model. In the future, new models will come out and the dominant interpretation will change again.

12.1          Understanding User Intent

It can be helpful to think of queries as having one or more of the following intents.

·         Know query, some of which are Know Simple queries

·         Do query, some of which are Device Action queries

·         Website query, when the user is looking for a specific website or webpage

·         Visit-in-person query, some of which are looking for a specific business or organization, some of which are looking for a category of businesses

12.1.1       Know and Know Simple Queries

The intent of a Know query is to find information on a topic. Users want to Know more about something.

Know Simple queries are a special type of Know query. Know Simple queries seek a very specific answer, like a fact, diagram, etc. This answer has to be correct and complete, and can be displayed in a relatively small amount of space: the size of a mobile phone screen. As a rule of thumb, if most people would agree on a correct answer, and it would fit in 1-2 sentences or a short list of items, the query can be called a Know Simple query.

Know Simple queries may be questions such as [how tall is barack obama]. Frequently, Know Simple queries do not have question words. For example, [barack obama height] has the same user intent as [how tall is barack obama], but is not in a question format.

Most queries are not Know Simple queries, such as:

·         Broad, complex, and/or in-depth informational queries that do not have a short answer

·         Ambiguous or unclear informational queries

·         Informational queries on controversial topics

·         Informational queries with no definitive “right answer”

·         Queries where different users may want different types of information, or different sources of information

Here are some examples where the Know Simple query asks for a simple fact, which can be answered correctly and completely in a small amount of space, and the Know query answer is more complex.

Know Simple Query	Know Query	Explanation
[barack obama height] [how tall is obama]	[barack obama]	The Know query is a broad information query and different users may be looking for different things (e.g., biography, books, social media posts, etc.).
[new york city population 2013]	[new york city]	The Know query is a broad information query and different users may be looking for different things (e.g., tourist and trip planning information, facts, photographs, history).
[who is graves disease named after]	[graves disease]	The Know query is a broad query for medical information and different users may have different needs.  There is no single “answer” for this query.
[macy’s store hours]	[macy’s gift wrap options]	The Know query is a broad query for a particular service offered by a department store, and does not have a short answer.
[what is the symbol for the element nickel]	[what nickel is used for]	The Know query is a broad query and there is no short, complete answer.
[who won the 2014 bcs national championship game]	[who is going to win the bcs national championship game]	The Know query asks for an opinion and there is no definitive answer.
[what is starbucks stock price]	[should i invest in starbucks stock]	Even though the Know query is theoretically a yes/no question, there is not a single answer that everyone would agree on.

Raters must think about mobile users when deciding if queries are Know Simple. Use your judgment here.

Important Rating Example: The query [weather] may seem like a broad information query, but most mobile users likely have a fairly simple informational need: to find the current or upcoming temperature, and the chance of local weather events, such as rain or snow. Therefore, we will consider queries like [weather], [weather today], [weather tomorrow], [weather this week] to be Know Simple queries for mobile phone users.

12.1.2       Do and Device Action Queries

The intent of a Do query is to accomplish a goal or engage in an activity on a phone. The goal or activity may be to download, to buy, to obtain, to be entertained by, or to interact with a website or app.  Users want to Do something.

Here are some examples.

Query	Likely User Intent
[get candy crush game]	Install the Candy Crush game.
[online personality test]	Take an online personality test.
[what is my bmi?]	Calculate BMI (body mass index).
[buy citizen kane dvd]	Purchase this DVD.
[bubble spinner]	Play Bubble Spinner online or download/open the app.

Device Action queries are a special kind of Do query. Users are asking their phone to do something for them. Users giving Device Action queries may be using phones in the hands-free mode, for example, while in a car. It’s very important for mobile phones to accommodate Device Action queries, and we have a high standard for rating these results.

A Device Action query usually has a clear action word and intent. The verb or action word is often at the beginning of the query, but a query might start with “OK Google” or “Google” or “Siri” or “I want to.” Use your judgment.

Here are some examples of Device Action queries.

Type of Action	Device Action Query Examples (verb or action shown in bold)
Call someone	[call mom’s mobile phone], [place a call to anne jones], [ok google, could you please call my mom]
Send a message	[send a text to john smith], [text john smith], [sms john smith]
Get map information (e.g., directions, traffic, etc.)	[navigate to central park], [show me traffic], [show me a map]
Open an app or webpage, or install an app	[go to amazon.com], [open facebook app], [open facebook.com], [open the business section of the new york times], [siri, would you open facebook for me], [get candy crush app]
Play or view media	[play me songs from the white album], [watch the life of pi], [show me cute kittens], [i want to look at cute kittens], [i want to listen to imagine dragons radioactive]
Schedule a meeting	[schedule a meeting for 9am]
Set an alarm or timer	[set an alarm for 6am], [alarm for 6am], [countdown timer for 30 secs]

Here are some examples of queries that are not Device Action queries.

Query	Query is NOT a Device Action Query: Explanation
[phone number empire state building]	There is no action word. The user may be looking for the phone number without the intention of calling the number.  We’ll consider this a Know Simple query.
[facebook.com]	There is no action word, such as “open.”  We will consider this a Website query.
[amazon.com]	There is no action word, such as “open.”  We will consider this a Website query.
[anne jones]	There is no action word, such as “call” or “text.” We will consider this a Know query for information.

12.1.3       Website Queries

The intent of a Website query is to locate a specific website or webpage which users have requested. This single webpage is called the target of the query.

One type of Website query is a URL Query, which can be:

·         Exact, perfectly-formed, working URLs, such as [http://www.ibm.com] or [www.ibm.com] or [ibm.com].

·         Imperfect URL queries: Queries that look like URL queries, but are not “working URL” queries. These URLs do not load if you type or paste them into your browser address bar. Even so, we believe users have a specific page in mind.

Here are some examples.

Query	Likely User Intent
[kayak], English (US)	View the Kayak website.
[youtube], English (US)	View the YouTube website.
[ebay], Italian (IT)	View the Italian eBay website.
[new york time health section], English (US)	View the Health section of the New York Times website.
[canon.com eos digital camera], English (US)	View the EOS digital cameras page on the Canon website.

12.1.4       Visit-in-Person Queries and User Location

Users carry mobile phones with them throughout the day, for example, at work, to school, to restaurants, or running errands. One reason to carry a mobile phone is to have help with Visit-in-Person queries, such as finding coffee shops, gas stations, ATMs, restaurants, etc. Because mobile phones are often used for Visit-in-Person queries, make sure to consider visit-in-person intent as a possibility for mobile phone users.

Some queries clearly “ask” for nearby information or nearby results (e.g., businesses, organizations, other nearby places). Some queries are not asking for nearby information or nearby results. Here are some  examples.

Queries with Visit-in-Person Intent	Query with Non-Visit-in-Person Intent
More examples: [pizza], [yoga class], [coffee shops], [movie showtimes], [car repair], [dentists], [bank of america atm locations], [starbucks near me]	More examples: [boston red sox], [washington post], [scrabble cheat], [definition of sedentary], [aapl], [beyonce], [angry birds], [small dog breeds], [dance videos], [oscars 2012], [pick up lines], [bank of america login]

And some queries could go either way.  Some users may want nearby results and others may not.

Here are some examples of queries with both visit-in-person and non-visit-in-person intent:

·         [hotels]

·         [post office]

·         [apple store]

·         [citibank]

·         [best buy]

·         [office depot]

·         [target]

·         [library]

·         [bank of america]

·         [the gap]

Sometimes, the user location can change our understanding of the query. For users close to Sunnyvale, California, the query [turmeric] could have two different interpretations: a popular restaurant named Turmeric or the spice turmeric.

In most other user locations, there is no restaurant (or anything else) named Turmeric and there is just one interpretation of the query [turmeric]: the spice. The Sunnyvale restaurant is not well known outside of Sunnyvale, California.

Use your common sense when thinking about queries and whether they have possible visit-in-person intent.

12.1.5       Queries with Multiple User Intents

Many queries have more than one likely user intent. Please use your judgment when trying to decide if one intent is more likely than another intent.

Query	Likely User Intent
[harvard]	Depending on the user need and location, users may want to visit the official homepage (Website), get directions (Visit-in-Person), or learn more about the school (Know).
[walmart]	Most users want to go to a nearby Walmart (Visit-in-Person) or view the homepage to shop online (Website). Some or few users may want to learn more information about the company (Know).

12.8          Understanding Result Blocks

The following sections contain examples of different types of queries and results. In these guidelines, please assume that the result blocks and pages are easy to use on the phone, unless otherwise noted.

12.8.1       Web Search Result Block Examples

Web Search Result Blocks typically have a title link, a URL and a “snippet” of text describing the page. For many queries, Web Search Result Blocks are the most helpful type of result.

Query, User Location, User Intent	Web Search Result Block
Query: [cuisinart food processor reviews] User Location: Oklahoma City, Oklahoma User Intent: This is a Know query. The user wants to find recent reviews of Cuisinart food processors. Result: This is a Web Search Result Block which has a link to a landing page with Cuisinart reviews.
Query: [broadway tickets] User Location: New York City, New York User Intent: This is a Know query or Do query. The user wants to search prices and/or purchase tickets to a Broadway show in New York City.

12.8.2       Special Content Result Block Examples

Special Content Result Blocks (SCRBs) appear in the search results page, along with Web Search Result Blocks. They are frequently, but not always, the first result on the search results page.

Special Content Result Blocks are designed to show content directly to users on the search results page. From working calculators to playable videos to interactive weather information, these results help users immediately get information or content, and may not require additional “clicks” or page loads.

Because mobile phones can be difficult to use, SCRBs can help mobile phone users accomplish their tasks very quickly, especially for certain Know Simple, Visit-in-Person, and Do queries.

Important: Please assume that any interactive features work and function properly. Some notes:

·         All result blocks are “screenshots” or images of search results with prominent links enabled. Unfortunately, a screenshot or image of an interactive result block will not function as it would for a real user. For the purpose of rating, please assume that interactive result blocks do function as intended, even though you are unable to use the buttons, tabs, or other features in the result block.

·         There may be a delay between when the rating task is created and when you actually rate the block, causing some information in special content result blocks to be a few hours or even days out of date. Stock price or weather informational blocks are designed to give users extremely current and timely information. However, due to a delay in rating time, the information may no longer be accurate. Don’t penalize a special content result block for being out of date. Assume that the blocks show current information for users, unless instructed otherwise.

Query, User Location, User Intent	Special Content Result Block
Query: [weather] User Location: Chicago, Illinois User Intent: This is a Know Simple query since mobile users have a fairly simple informational need: find the current temperature and chance of rain or snow. The user wants to know the weather for the User Location. Note: Assume the block shows current information for users.

Query, User Location, User Intent	Special Content Result Block
Query: [emma stone movies] User Intent: This is a Know query. The user wants to get information on movies with Emma Stone. User Location: Des Moines, Iowa Result: In this result block, users can immediately see some popular movies starring Emma Stone, with an option to click on the links to learn more about each movie. Users can also swipe to see a list of more movies.
Query: [calories in a banana] User Location: Oakland, California User Intent: This is a Know Simple query. The user wants to find out how many calories are in a banana.
Query: [baseball scores] User Location: Wichita, Kansas User Intent: This is a Know query. The user wants to find the most recent Major League Baseball scores.

Query, User Location, User Intent	Special Content Result Block
Query: [coldplay fix you] User Location: New Orleans, Louisiana User Intent: This is a Do query. The user probably wants to listen to the song by Coldplay called “Fix You.” Note: In this result block, users can immediately play the song on their phone, or click the links to learn more about the artist or album.
Query: [what is the tallest tree] User Location: Boise, Idaho User Intent: This is a Know Simple query. The user wants to know what type of tree is the tallest.

12.8.3       Device Action Result Block Examples

A mobile phone should respond to a Device Action query and do what the user is asking. If the block is a Device Action query to open an app, please assume that the user has the app installed on their phone. If the query is to download the app, please assume that the user does not have the app installed on their phone.

Query, User Location, User Intent, Result	Device Action Result Block
Query: [open angry birds] User Location: Little Rock, Arkansas User Intent: This is a Device Action query. The user wants the device to open the Angry Birds app so he/she can play the game. Result: In this result block, users can click the link to open the app.

Query, User Location, User Intent, Result	Device Action Result Block
Query: [go to amazon.com] User Location: Glendale California User Intent: This is a Device Action query. The user wants to go to amazon.com to interact with the website. Result: In this result block, users can click the links to go directly to the website.
Query: [set alarm for 5 o’clock am] User Location: Tacoma, Washington User Intent: This is a Device Action query. The user wants to set the device’s alarm to go off at 5:00 AM. Result: In this result block, the query has initiated the alarm on the phone to be set for 5:00 AM.
Query: [send text to mom] User Location: Tacoma, Washington User Intent: This is a Device Action query. The user wants to send a text message to a contact nicknamed “Mom.” Result: In this Device Action Result Block, the query has initiated a text to the contact nicknamed “Mom,” with an option to text either the work or mobile number.

Query, User Location, User Intent, Result	Device Action Result Block
Query: [call best buy] User Location: Mountain View, California User Intent: This is a Device Action query. The user wants to call the nearest Best Buy store. Result: In this Device Action Result Block, the query has initiated a call to the nearest Best Buy location.

12.8.4       How Device Action Results are Displayed in Rating Tasks

Important note: Users issuing queries actually experience the phone’s response to the query, while raters are given a description of it. If an iPhone user says “Siri, call Mom,” the iPhone displays a visual prompt to show that it’s dialing a phone number.  In these rating tasks, you will see a description of this action.

Please also assume that the phone successfully performs the action in a fully satisfying way.

Below are some additional examples of Device Action queries, along with the corresponding “action text” displayed in the rating task.

This is what the user sees on his or her phone for the Device Action query	This is what you (the rater) might see in the rating task
The left side shows what the user sees on the phone for the query [set alarm for 30 mins], while the right side shows the “action text” that you (the rater) might see in the rating task. The text on the right indicates that the user issued the query at 1:48 PM (13:48:00), and the device has set its alarm to go off 30 minutes later at 2:18 PM (14:18:00).

This is what the user sees on his or her phone for the Device Action query	This is what you (the rater) might see in the rating task
The left side shows what the user sees on the phone for the query [open facebook app]. In this case, the phone shows an “Opening app” message and then opens the app. The right side shows the “action text” that you (the rater) might see in the rating task. The text indicates that the device responded by opening the Facebook app on the user’s phone.
The left side shows what the user sees on the phone for the query [call dan], while the right side shows the “action text” that you (the rater) might see in the rating task. The user wants to dial the number stored for the contact “Dan” on the device. This text indicates that the device responded by displaying the contact’s phone number and dialing the number.

This is what the user sees on his or her phone for the Device Action query	This is what you (the rater) might see in the rating task
For some Device Action queries, the device displays a list of options to choose from before it can respond to the Device Action query. Here is an example for the query [call target]. The user is shown a list of Target stores in the area to choose from. This example shows that the Device Action result taken by the device will be to call the desired Target store after the user makes his/her selection.
Here is an example for the query [play adele]. The user is shown a "Play media" type of result. In this case, since the user wants to play music, the Device Action result shown is a "Play Music" button to click. Clicking this button will play the specified song if the user has the song on his/her device. If not, clicking the result box will give the user options to hear the specified song online.

12.9          Rating on Your Phone Issues

We understand that raters using different phones, operating systems, and browsers may have different experiences. In general, do what you would do naturally, and rate based on your experience. Here are some additional points to keep in mind when you are rating on your phone:

·         If you see one of these messages when you open a landing page on your phone:

o    Asks whether you want to open the page in the browser or the website’s app: select and evaluate the webpage.  However, if the result automatically opens an app by default based on your phone’s settings (e.g. some users have set all YouTube pages to automatically open the result in the YouTube app), you should rate your natural app experience—you do not have to change your phone’s default settings.

o    Asks whether you want to visit the mobile page or desktop page: it is fine to select the mobile page as long as the landing page is the same. Sometimes, the mobile option will bring you to the mobile homepage instead of the specific URL in the task. You may need to check that the mobile page is in fact the same URL as the desktop page.

·         Please open and look at PDF files. Your experience may be easier or harder than other users depending on your phone and browser, but you should still open the PDF file and look at it.

·         Occasionally, you may be assigned some of these rating tasks on a desktop computer, but please rate from the perspective of a mobile user unless otherwise instructed.

·         When rating Device Action queries and Device Action Result Blocks, including queries for installing or opening apps, please assume that queries were issued on an Android device unless explicitly stated otherwise in the instructions.

Part 3: Needs Met Rating Guideline

13.1             Rating Using the Needs Met Scale

There are many different kinds of queries and results, but the process of rating is the same: Needs Met rating tasks ask you to focus on mobile user needs and think about how helpful and satisfying the result is for the mobile users.

This is what the Needs Met rating slider looks like:

Rating	Description
Fully Meets (FullyM)	A special rating category, which only applies to certain queries and results. All or almost all mobile users would be immediately and fully satisfied by the result and would not need to view other results to satisfy their need.
Highly Meets (HM)	Very helpful for many or most mobile users.  Some users may wish to see additional results.
Moderately Meets (MM)	Helpful for many users OR very helpful for some mobile users. Some or many users may wish to see additional results.
Slightly Meets (SM)	Helpful for fewer mobile users. There is a connection between the query and the result, but not a strong or satisfying connection. Many or most users would wish to see additional results.
Fails to Meet (FailsM)	Completely fails to meet the needs of the mobile users. All users would wish to see additional results.

Please note that you may assign in-between ratings. Use in-between ratings if you think the rating of a result falls between two labels. You can either drag the slider or click on the point that you want the slider to land on.

13.2          Rating Result Blocks: Block Content and Landing Pages

For Needs Met rating, you will assign a rating to each result. Each result includes the content inside the result block and landing pages associated with the result.

Which part of the result do you rate?  It depends on both the query and the result block:

Type of Block	What to Rate
Special Content Result Block (SCRB) Note: assume that interactive result blocks function as intended, even though you are unable to use the buttons, tab, or other features in your rating task.	The content inside this type of block should always play a large role in your rating. In addition, think about whether a user would click on the link(s) if available, in order to satisfy their user need. ·         If most users would not click, rate the Special Content Result Block based on the block content alone. ·         If some or many users would click, you may consider the helpfulness of the landing page(s) in addition to the content in the block. In this case, both need to be helpful to justify a high rating. If either the content in the block or on the landing page is unhelpful, give a low rating.
Web Search Result Block	In most cases, a click is required and you should evaluate the landing page.
Device Action Result Block	Base your rating on the helpfulness of the action itself.

For example, think about the query [what does love mean].

Result Block	Query: [what does love mean]	Rating
Special Content Result Block		Most users would probably not click on the Special Content Result Block, because the block contains a large amount of helpful content and has no obvious landing page link associated with it. Therefore, base your rating on the content inside the block itself.
Web Search Result Block		In this case, users would have to click on the web search result in order to get an answer to the question. Therefore, base your rating on the content of the landing page.
Special Content Result Block		While the block content may be helpful, this question may require a more in depth answer for some users. Some or many users might click on the link, in addition to looking at the content in the block itself. Therefore, base your rating on both the block and the landing page.

Here are some examples of Special Content Result Blocks where the block should be rated primarily on the content inside the block itself.   As always, please use your judgment.

Query and Special Content Result Block
Query: [utopia animal hospital]	Query: [chicago weather]	Query: [calories in a banana]
Query: [how to find security code on visa]	Query: [how did Lincoln die]	Query: [movies san francisco]

13.3          Fully Meets (FullyM)

Fully Meets is a special rating category, which can be used in the following situations:

·         The query and user need must be specific, clear, and unambiguous.

·         The result must be fully satisfying for mobile users, requiring minimal effort for users to immediately get or use what they are looking for.

·         All or almost all users would be completely satisfied by the result—users issuing that query would not need additional results to fully satisfy the user intent.

In other words, the Fully Meets rating should be reserved for results that are the “complete and perfect response or answer” so that no other results are necessary for all or almost all users to be fully satisfied.

You will need to use your judgment to decide whether a result block can Fully Meet the user need. Here are some scenarios when the Fully Meets rating is appropriate:

·         The user is clearly looking for a specific webpage or website and the result block is the specific result the user is looking for.

·         The user is trying to complete a device action and the result block accomplishes the desired request.

·         The user is looking for a very specific fact or piece of information and the result block provides the information immediately, thoroughly, and clearly.  No other results would be needed.

Fully Meets may apply in other situations as well. Ask yourself whether the result block alone is the perfect and complete result that would fully satisfy all or almost all users.  Be conservative when using the Fully Meets rating.  When in doubt, consider a lower rating.

Note: If a result block is very close to being fully satisfying, but the block alone may not be sufficient in order to fully satisfy all or almost all users, a rating of Highly Meets+ may be appropriate.

Sometimes you have to think about how specific the user need is and will need to make a judgment call on whether it is specific enough to possibly have a Fully Meets rating.

13.3.1       Examples of Fully Meets (FullyM) Result Blocks

Query, User Location, User Intent	Result	Rating	Fully Meets Explanation
Query: [amazon.com] User Location: Austin, Texas User Intent: Go to amazon.com.		What you might see in the rating task: Action: Open a specific webpage URL: http://www.amazon.com/ Website Name: Amazon.com: Online Shopping for Electronics, Apparel, Computers ...	The query has clear intent to go to amazon.com and the phone is opening the specified website.

Query, User Location, User Intent	Result	Rating	Fully Meets Explanation
Query: [amazon] User Location: Austin, Texas User Intent: Go to the Amazon website.			The query has clear intent to go to the amazon.com website. While there may be other interpretations for the query, the dominant one is by far the website.
Query: [target website] User Location: Jacksonville, Florida User Intent: Go to the Target website.			The query has clear intent to go to target.com.
Query: [shop nordstrom online] User Location: Monterey, California User Intent: Go to the Nordstrom website for online shopping.			The query has clear intent to go to the Nordstrom website.
Query: [titanic imdb] User Location: Lexington, Kentucky User Intent: Go to the IMDb page for Titanic.			The query has clear intent to go to the IMDb page for Titanic.
Query: [www.yahoo.c0m] User Location: Denver, Colorado User Intent: Go to the Yahoo website.			Even though this is an imperfect URL query, it’s clear the user wants to go to the Yahoo website.
Query: [cnn health] User Location: Annapolis, Maryland User Intent: Go to the Health section of cnn.com.			The query has clear intent to go to the Health section of the cnn.com webpage.

Query, User Location, User Intent	Result	Rating	Fully Meets Explanation
Query: [cnn] User Location: Annapolis, Maryland User Intent: Go to the CNN website.			The query has clear intent to access CNN news and content. It is not clear whether the user has the app installed, but the website has the content the user is searching for, and fully satisfies the user intent.
Query: [yelp] User Location: Raleigh, North Carolina User Intent: Open the Yelp app or go to the website at yelp.com.			This result Fully Meets the user need whether they want the Yelp app or the Yelp website, which are both popular. Note: If you see this kind of result, assume that the user has the app installed on their phone. Clicking on the Yelp link above the logo opens the app directly, and clicking on the “Open on yelp.com” link gives the option of opening the app or website.
Query: [lebron james stats basketball- reference.com] User Location: Miami, Florida User Intent: Go to the player statistics page for LeBron James on a specific website.			The query has clear intent to go to the player statistics page for LeBron James on a specific website, basketball- reference.com.
Query: [open instagram] User Location: Laguna Beach, California User Intent: Open the Instagram app on the user’s device.	What the user sees:	What you might see in the rating task: Action: Open an app App: Instagram	This Device Action query is clear and specific: the user wants to open the Instagram app. The result block shows the phone in the process of opening the specified app—it Fully Meets the user need.

Query, User Location, User Intent	Result	Rating	Fully Meets Explanation
Query: [open bing search app] User Location: Dallas, Texas User Intent: Open the Bing Search app on the user’s device.			The result block shows that the phone has detected that the app is not installed and gives the option of downloading the app. This result Fully Meets the user need. Note: Remember that you should assume that queries were issued on an Android device when rating Device Action Result Blocks, unless explicitly stated otherwise in the instructions. Therefore, the Google Play result for this query is appropriate.
Query: [open facebook app] User Location: Richmond, Virginia User Intent: Open the Facebook app on the user’s device.	What the user sees:	What you might see in the rating task: Action: Open an app App: Facebook	This Device Action query is clear and specific: the user wants to open the Facebook app. The result block shows the phone’s confirmation that it is opening the app—it Fully Meets the user need.
Query: [set alarm for 5 o’clock am] User Location: Tacoma, Washington User Intent: Set the device’s alarm to go off at 5:00 AM.	What the user sees:	What you might see in the rating task: Action: Set alarm Alarm Time: 5:00 AM	This Device Action query is clear and specific: the user wants to set the phone’s alarm clock for 5:00 AM. The result block shows the phone in the process of setting the alarm at the specified time—it Fully Meets the user need.
Query: [chef chu phone number] User Location: Los Altos, California User Intent: Find the phone number for the restaurant called Chef Chu’s.			Chef Chu’s is a Chinese restaurant located in the user location. This result block immediately shows the phone number with the option to call the number directly on the device—it Fully Meets the user intent.

Query, User Location, User Intent	Result	Rating	Fully Meets Explanation
Query: [call best buy] User Location: Mountain View, California User Intent: Call the nearest Best Buy store. Note that there is only one Best Buy store located in Mountain View.	What the user sees:	What you might see in the rating task: Action: Make a call Business Name: Best Buy Location: 2460 E Charleston Rd, Mountain View, CA 94043 Phone number: (650) 903-0591	This Device Action query is clear and specific: the user wants to call the nearest Best Buy store.  The result block shows the phone in the process of calling the store—it Fully Meets the user need.
Query: [open map of italy] User Location: Baltimore, Maryland User Intent: Open a map of Italy on the user’s device.	What the user sees:	What you might see in the rating task: Action: Open map Location: Italy	This Device Action query is clear and specific: the user wants to see a map of Italy. The result block shows a map with a link to open it in Google Maps—it Fully Meets the user need.
Query: [navigate to yosemite national park] User Location: Seattle, Washington User Intent: Navigate to Yosemite National Park from the user’s location.	What the user sees:	What you might see in the rating task: Action: Navigation Destination: Yosemite National Park Destination Address: Yosemite Village, CA 95389	This Device Action query is clear and specific: the user wants to navigate to Yosemite National Park. The result block shows the phone in the process of opening up the Navigation app, which will give the user turn-by-turn directions from the user’s location—it Fully Meets the user need.

Query, User Location, User Intent	Result	Rating	Fully Meets Explanation
Query: [how to find security code on visa] User Location: Chicago, Illinois User Intent: Find out how to locate the security code on a Visa card.			The result block displays the complete and correct answer prominently in the result block, in a way that is easy for users on mobile devices to read.  The block also includes an image showing users exactly where to find the code, and the block Fully Meets the user need.
Query: [starbucks stock price] User Location: Alexandria, Virginia User Intent: Find the current stock price for Starbucks.			The result block displays the complete and correct answer prominently in the result block, in a way that is easy for users on mobile devices to read—it Fully Meets the user need. Note: Assume that the result shows current information for users.
Query: [new york city population 2012] User Location: Pasadena, California User Intent: Find the population of New York City back in 2012.			The result block displays the complete and correct answer prominently in the result block, in a way that is easy for users on mobile devices to read—it Fully Meets the user need.
Query: [what is the weather forecast for today] User Location: Mountain View, California User Intent: Find weather forecast information for the current day in Mountain View, California.			The result block displays the complete and correct answer prominently in the result block, in a way that is easy for users on mobile devices to read—it Fully Meets the user need. Remember that we consider weather queries, like this one, to be a Know Simple query for mobile phone users. Note: Assume the result shows current information for users.

Query, User Location, User Intent	Result	Rating	Fully Meets Explanation
Query: [london time] User Location: New York, New York User Intent: Find the current time in London.			The result block displays the complete and correct answer prominently in the result block, in a way that is easy for users on mobile devices to read—it Fully Meets the user need. Note: Assume that the result shows current information for users.
Query: [who is the chancellor of germany] User Location: New York, New York User Intent: Find the name of the current Chancellor of Germany.			The result block displays the complete and correct answer prominently in the result block, in a way that is easy for users on mobile devices to read—it Fully Meets the user need.
Query: [argo trailer] User Location: Madison, Wisconsin User Intent: Watch the trailer for Argo.			This query can be considered a Know query with a specific intent or a Do query—regardless of the query type, the user is asking for a very specific thing: the trailer for Argo. The result has exactly what the user wants, displaying the trailer clearly and prominently inside the result block.  The video is immediately presented and the user can click on the video to show the exact information that was requested.
Query: [chevron at shoreline and middlefield] User Location: Mountain View, California User Intent: Find the specific gas station specified by the user in order to visit the location in person.			The intent is to visit the specific gas station specified by the user. This result block is for the specified Chevron location, with information to visit this location in person.

Query, User Location, User Intent	Result	Rating	Fully Meets Explanation
Query: [decemberists crane wife 3 lyrics] User Location: Long Beach, California User Intent: See the lyrics to The Decemberists song called “Crane Wife 3.”			The result has exactly what the user wants, displaying the lyrics to the specified song clearly and prominently inside the result block. Note: The lyrics posted on Google Play are licensed.
Query: [gas stations near me] User Location: Mountain View, California User Intent: Find gas stations near the user location. Note that this may be an urgent query.			The intent is to find gas stations near the specific user location, represented by the blue dot on the map. This result block has a very satisfying list of nearby, popular, and prominent options within one mile of the specific location. This kind of block is especially helpful for users who want to visit the business in person. This result would be very convenient for mobile users who need gas immediately, especially for those who are driving and cannot easily use their phone. Note: A range is included because some users would be fully satisfied with these selections, while some users would want more information (e.g., gas prices, hours, distance from location, longer list of options).

Query, User Location, User Intent	Result	Rating	Fully Meets Explanation
Query: [nearby coffee shops] User Location: Mountain View, California User Intent: Find coffee shops near the user location.			The intent is to find coffee shops near the specific user location, represented by the blue dot on the map. This result block has a very satisfying list of nearby, popular, and prominent options within a few miles of the specific location. This kind of block is especially helpful for users who want to visit the business in person.  Note that these blocks are interactive—clicking on a coffee shop will give options to call the business, get directions, read reviews, etc. Note: A range is included because some users would be fully satisfied with these selections, while some users would want more information (e.g., hours, distance from location, longer list of options).
Query: [1600 pennsylvania ave washington dc] User Location: Bakersville, California User Intent: Find a map, directions, information about what is located at this address, etc.			This block contains a map with the queried address, link to get directions, and specifies the landmark at the well-known address on the map (“The White House”). It’s not unusual to search for an address (or business) that is far from the user location. Here, the user has explicitly asked for a specific address in a far away place, and this result is very satisfying. For a query this specific, the user location does not change the rating. Note: A range is included because some users would be fully satisfied with this result, while some users would want more information (e.g., photos, a brief description).

13.3.2       Examples of Result Blocks that Cannot be Fully Meets

There are some queries which cannot have a Fully Meets result. Here are some examples.

Type of Query	Example	No Fully Meets Result: Explanation
Broad queries where no single result could fully satisfy all users	[knitting]	This is a broad informational query. Knitting is an activity anyone can do and that anyone can create a website for. Different users may want different types of content: videos, instructions, patterns, etc. There is no one official source for knitting information and no one result could satisfy most users. Therefore, no Fully Meets result is possible for this query.
	Famous names e.g., [barack obama]	This is a broad informational query and it is impossible to know exactly what the user is looking for.
Ambiguous queries without a clear user intent or dominant interpretation	[ada]	There is no dominant interpretation for this query. The following entities are all common interpretations: Americans with Disabilities Act, American Dental Association, and American Diabetes Association. While each interpretation has an official homepage, none is Fully Meets since there is no dominant interpretation.
	Non-famous people names e.g., [sam wen]	Queries for people’s names can be tricky.  Many or most people queries do not have a dominant interpretation. Even unusual sounding name queries may not have a dominant interpretation. For example, the queries [sam wen], [tran nguyen], and [david mease] can have no Fully Meets result because there are multiple people with each of these names, and it is not clear that most users are looking for any one particular individual.

13.4          Highly Meets (HM)

A rating of Highly Meets is assigned to results that meet the need of many or most users. Highly Meets results are highly satisfying and a good “fit” for the query. In addition, they often have some or all of the following characteristics: high quality, authoritative, entertaining, and/or recent (e.g., breaking news on a topic).

A query can have many Highly Meets results.

Have high standards for using the Highly Meets rating. This is especially important for queries with many, many on- topic results.

13.4.1       Examples of Highly Meets (HM) Result Blocks

Query, User Location, User Intent	Result	Rating	Explanation
Query: [trader joes] User Location: Charlotte, North Carolina User Intent: There are two possible user intents: most users probably want to visit a nearby location or go to the website.			This result shows a complete list of all three locations in the Charlotte area, with information that is especially helpful for users who want to visit the store. Note: This result block is not Fully Meets because users who want to go to the website to see coupons, promotions, etc. would have to see additional results.
Query: [trader joes] User Location: Charlotte, North Carolina User Intent: There are two possible user intents: most users probably want to visit a nearby location or go to the website.			This is the official website for Trader Joe’s, which has highly authoritative information on the businesses’ stores, promotions, recipes, news, etc. Note: This result block is not Fully Meets because users who want information to visit the store have to do a bit of work to find that information— it is not immediately available.
Query: [museum of modern art] User Location: Manhattan, New York User Intent: There are two possible user intents: most users probably want to visit the museum (note that the user is located in NYC where the museum is located) or go to the website.			The result block shows information about the museum, and is especially helpful for users who want to visit the museum. Note: This result block is not Fully Meets— even though there is a website link, it is not very satisfying for users who want to go to the website because it is not as informative as a web search result block with a title, snippet, and URL.

Query, User Location, User Intent	Result	Rating	Explanation
Query: [museum of modern art] User Location: Manhattan, New York User Intent: There are two possible user intents: most users probably want to visit the museum (note that the user is located in NYC where the museum is located) or go to the website.			This is the museum’s official website and has very helpful information about exhibits, membership, the store, etc. Note: This result block is not Fully Meets because users who want information to visit the museum have to do a bit of work to find that information—it is not immediately available.
Query: [prestige bmw] User Location: Ramsey, New Jersey User Intent: There are two possible user intents: most users probably want to visit the car dealership or go to the website.			The query is the name of a car dealership in New Jersey, which is the only dealership with that name in the user location (Ramsey, New Jersey). The result block shows information about the dealership, and is especially helpful for users who want to visit the business. Note: This result block is not Fully Meets— even though there is a website link, it is not very satisfying for users who want to go to the website because it is not as informative as a web search result block with a title, snippet, and URL.
Query: [prestige bmw] User Location: Ramsey, New Jersey User Intent: There are two possible user intents: most users probably want to visit the car dealership or go to the website.			This is the car dealership’s official website and has very helpful information about inventory, financing, etc. Note: This result block is not Fully Meets because users who want information to visit the dealership have to do a bit of work to find that information—it is not immediately available.