Bài viết hay 5

16.1           Rating Queries with Multiple Interpretations and Intents

Some queries really only have one meaning. Consider the query [iphone], English (US). There may be different user intents for this query (research iPhones, buy an iPhone, go to the iPhone page on Apple’s website), but all users are basically referring to the same thing: the phone made by Apple, Inc.

Some queries truly have different possible meanings. Consider the query [apple], English (US). Some users may want to find more information on the computer brand or the fruit. We refer to these different meanings as query interpretations.

When giving Needs Met ratings for results involving different query interpretations, think about how likely the query interpretation is and how helpful the result is.

·         A very helpful result for a dominant interpretation should be rated Highly Meets, because it is very helpful for many or most users.  Some queries with a dominant interpretation have a FullyM result.

·         A very helpful result for a common interpretation may be Highly Meets or Moderately Meets, depending on how likely the interpretation is.

·         A very helpful result for a very minor interpretation may be Slightly Meets or lower because few users may be interested in that interpretation.

·         There are some interpretations which are so unlikely that results should be rated FailsM. We call these “no chance” interpretations.

16.2          Rating Queries with Both Website and Visit-in-Person Intent

Some queries have two possible strong intents:

1.     Go to the website intent: in order to, for example, find out information, buy something online, make a reservation, schedule an appointment, interact with customer support, or fulfill some other need that can be satisfied online

2.     Visit-in-person intent: user wants to visit the store, business, etc. in person

We know the user intent is to accomplish one or the other, but it is unclear which one the user wants. For these queries, result blocks that only satisfy one intent should NOT get a Fully Meets rating.

Query, User Location, User Intent	Result Block and LP	Needs Met Rating and Explanation
Query: [target] User Location: Jacksonville, Florida User Intent: There are two possible strong user intents: most users probably want to visit a nearby Target location or go to the website to shop online, research products, find prices, etc.		The result block shows three popular Target locations in Jacksonville, with information that is especially helpful for users who want to visit the store. This fulfills the user intent to find a nearby Target location.
		The result is very satisfying and fulfills the user intent to shop online or otherwise use the website.

Query, User Location, User Intent	Result Block and LP	Needs Met Rating and Explanation
Query: [dmv] User Location: Belmont, California User Intent: There are two possible strong user intents: most users probably want to visit a nearby DMV location or go to the DMW website to renew a license, pay a fee, find some information, etc.		The result block shows two nearby DMV locations, with information that is especially helpful for users who want to visit the locations. This fulfills the user intent to find a nearby DMV location.
		The result is very satisfying and fulfills the user intent to find information or otherwise use the website (e.g., to review a license, pay a fee).
Query: [citibank] User Location: Palo Alto, California User Intent: There are two possible strong user intents: most users probably want to visit a nearby Citibank location or go to the website to bank online.		The result block shows three nearby Citibank locations in the user location of Palo Alto. The information is especially helpful for users who want to visit the bank. This fulfills the user intent to find a nearby Citibank location.
		The result is very satisfying and fulfills the user intent to do online banking or otherwise use the website.

17.0     Specificity of Queries and Landing Pages

Some queries are very general and some queries are specific. Here are some examples that compare levels of specificity of English (US) queries:

Query	More Specific Query	Even More Specific Query
[chair]	[dining room chair]	[ikea “henriksdal” highback upholstered chair]
[library]	[harvard library]	[harvard anthropology library]
[interview questions]	[interview questions for teachers]	[practice interview questions used for teach for america]
[restaurants]	[chinese restaurants]	[takeout chinese restaurants in downtown Austin]
[coffee shops]	[starbucks]	[red rock coffee mountain view]

Results for specific queries are easier to rate on the Needs Met scale because we know more about what the user is looking for. Giving a Needs Met rating for results for general queries can be difficult. As always, your rating is based on how helpful the result is for the query, not the specificity fit.

When the query is a broad category, such as [cafes] [restaurants] [hotels] [books] [tourist attractions in paris] etc., popular and prominent examples may be considered very helpful. Please do web research to help you understand what is popular and prominent in different locations.

Query, User Location, User Intent	Result Block and LP, E-A-T Rating, Needs Met Rating		Explanation
Query: [credit cards] User Location: Phoenix, Arizona User Intent: Users are probably looking to sign up for a credit card online, or want to research credit cards before signing up.			Even though there are many options, a page for an individual company that offers different cards could be very helpful. E-A-T: Visa is a well-known credit card company with a good reputation. Needs Met: This LP is more specific than the query, but it would still be helpful for many or most users because Visa is a popular credit card company.
			Even though there are many options, a page for an individual company that offers different cards could be very helpful. E-A-T: Discover is a well-known company that offers services for credit cards, banking, and loans, and has a good reputation. Needs Met: This LP is more specific than the query, but it would still be helpful for many or most users because Discover is a popular credit card company.
			This page offers a list of top credit cards in a variety of categories. E-A-T: This website helps consumers make money decisions. It has a good reputation and has been recommended by CNNMoney and The New York Times. Needs Met: This LP fits the query. It would be helpful for most users.
			This page describes a credit card that requires union membership. E-A-T: This company is an expert on its own credit card, which is issued by HSBC Bank, a well-known financial services organization. Needs Met: Since the credit card requires union membership, the page would be helpful for some users.

Query, User Location, User Intent	Result Block and LP, E-A-T Rating, Needs Met Rating		Explanation
Query: [hotels] User Location: Jacksonville, Florida User Intent: Users are probably planning a trip, but this query is very general and vague.			This is a popular travel aggregator website, and the hotel page on the site can help users find hotels in the U.S. Users can read reviews, compare hotels, or make a reservation. E-A-T: Orbitz is a popular website with a good reputation. Needs Met: This LP fits the query. It would be very helpful for most users.
			This is a very popular hotel chain with hotels available in the majority of the U.S. at many different price points. Even though the list of possible hotel chains is long, the homepage of an individual chain that offers different prices, features, and location options could be very helpful. E-A-T: The Marriott website gives information on Marriott hotels, a popular chain of hotels. Marriott is an expert on Marriott hotels and the information is highly authoritative. Needs Met: This LP is more specific than the query, but it would still be helpful for many or most users because the Marriott is a popular chain of hotels.
			This is the webpage of the Marriott Courtyard hotel in Emeryville, California. E-A-T: The Marriott website gives information on Marriott hotels, a popular chain of hotels. This page is professional, well designed, and functions well. Marriott has a good reputation and is an expert on Marriott hotels, making the information on this page highly authoritative. Needs Met: The LP is too specific for the query, but this is a well-known brand and users can navigate to other Marriott hotels from this page. Few users would find this page helpful.

Query, User Location, User Intent	Result Block and LP, E-A-T Rating, Needs Met Rating		Explanation
Query: [target] User Location: Jacksonville, Florida User Intent: Go to target.com or find a nearby Target store.			Needs Met: The result block shows three popular Target locations in Jacksonville, with information that is especially helpful for users who want to visit the store.
	No E-A-T rating required for this block
			The LP is the Target website. E-A-T: The Target website offers many different categories of products to buy online, as well as coupons, store information, and more topics that they are the experts on. Needs Met: The result is very satisfying for mobile users that want to go to the website.
			The LP is the “store locator” page on the Target website. E-A-T: Target is the expert about where its stores are located. This page is the most authoritative page on this topic. Needs Met: The LP is more specific than the query, but many or most users would be interested in this page.
			The LP is the “electronics” page on the Target website. E-A-T: Target has a good reputation, but is not an expert or authority on much of what it sells. Needs Met: The LP is more specific than the query, but many or most users would be interested in this page.

Query, User Location, User Intent	Result Block and LP, E-A-T Rating, Needs Met Rating		Explanation
Query: [target] User Location: Jacksonville, Florida User Intent: Go to target.com or find a nearby Target store.			Needs Met: While this result block has some helpful content, most people searching for Target are not looking for the stock price or the names of the CEO and Founder. A rating of Slightly Meets to Slightly Meets+ is appropriate.
	No E-A-T rating required for this block
			The LP on the Target website is for a specific type of women’s pants. E-A-T: Target has a good reputation, but is not an expert or authority on much of what it sells. Needs Met: The LP is much more specific than the query and would be helpful for very few users.

Query, User Location, User Intent	Result Block and LP, E-A-T Rating, Needs Met Rating		Explanation
Query: [chicken recipes] User Location: Austin, Texas User Intent: Users probably want to make a chicken dish and are looking for some recipes to choose from. Users probably expect and want a list of recipes.			This LP has many chicken recipes (with reviews) on a popular recipe website. E-A-T: Food Network is a popular website with a good reputation for providing high quality recipes and cooking information. Needs Met: This LP fits the query. It would be very helpful for most users.
			This LP has a single recipe for chicken parmesan. E-A-T: Food Network is a popular website with a good reputation for providing high quality recipes and cooking information. Needs Met: This is a popular type of chicken recipe on a popular website, but the LP is more specific than the query. Some or few users would find this page helpful.
			This LP has over 25 recipes for fried chicken, a popular chicken dish. E-A-T: Allrecipes.com is a popular website with a good reputation for providing high quality recipes with lots of reviews. Needs Met: Even though there are over 25 different recipes on the LP, they are all for the same basic dish.  Therefore, this LP is also more specific than the query. Some or few users would find this page helpful.
			This is a fake search page for chicken recipes. E-A-T: Ads should never disguise themselves as the MC of the page. Pages with Ads that are designed to look like MC should be considered deceptive, making this fake search untrustworthy. Needs Met: It should be rated FailsM as it would be helpful for no users.

Query, User Location, User Intent	Result Block and LP, E-A-T Rating, Needs Met Rating		Explanation
Query: [smithsonian air and space museum store] User Location: Washington, DC User Intent: Go to the store website or find information, such as opening hours, about the museum’s store in Washington, DC.			This LP is the official website for the museum store. E-A-T: The museum is the expert on information about the store, including hours, phone numbers, membership information, etc. Needs Met: The result is very satisfying for mobile users that want to go to the website.
			Needs Met: While the result is about the museum, the hours, address, and directions are the same as the museum store. This information is helpful for users searching for the store.
	No E-A-T rating required for this block
			This LP has information about how to volunteer at the museum. E-A-T: The museum is the expert on their own volunteer opportunities. This page is the most authoritative page on this topic. Needs Met: The LP is more specific than the query and would be helpful for few users.
			Needs Met: This is not the museum requested in the query so the result Fails to Meet the user intent.
	No E-A-T rating required for this block

18.0     Needs Met Rating and Freshness

Some queries demand very recent or “fresh” information. Users may be looking for “breaking news,” such as an important event or natural disaster happening right now. Here are different types of queries demanding current/recent results.

Type of Query	Example Queries	Explanation
“Breaking news” queries	[tornado], [tsunami]	Assume users need the information right away. Imagine someone who needs immediate weather information because a big storm is coming. Information about last year's weather would not be helpful.
Recurring event queries, such as elections, sports events, TV shows, conferences, etc.	[olympics], [american idol], [redsox schedule], [tax forms], [elections]	Assume users are looking for the most recent or current information about the event. For example, if the Olympics are happening right now, users searching [olympics] want information about the current Olympics, not results from years ago. If the next Olympics are a few months away, users are probably interested in the upcoming Olympics.
Current information queries	[population of paris], [amount of u.s. debt], [airfare from ny to sfo], [next federal holiday]	Assume users are looking for the most current information, such as prices or airfare.
Product queries	[iphone], [toyota camry], [windows operating system]	Assume users are looking for information about the most recent model/version for these product queries.

When a query demands recent content, only pages with current, recent, or updated content should get high Needs Met ratings. For these queries, pages about past events, old product models and prices, outdated information, etc. are not helpful. They should be considered “stale” and given low Needs Met ratings. In some cases, stale results are useless and should be rated FailsM.

For some queries, there may be “newsy” or recent information user intent, as well as more “timeless” information user intent. Users issuing queries for celebrities or politicians may be interested in biographical information, or users may be looking for the latest news or gossip.

Freshness is generally less of a concern for E-A-T rating. “Stale” pages can have high E-A-T ratings. For example, some highly reputable news websites maintain “archival” content. However, unmaintained/abandoned “old” websites or unmaintained and inaccurate/misleading content is a reason for a low E-A-T rating.

Note: The date the page was created may be different from when the content was last updated or modified. When content is updated, the page will sometimes show the date of the update, not the date the page was created. Some websites always show the current date, regardless of when the content was last updated. If you are curious about the content of a page, see here to try the “Wayback Machine” on the Internet Archive. Not all pages are available, but this tool may help you understand how some pages were created and how their content has changed over time.

Query, User Location, User Intent	Result Block and LP, E-A-T Rating, Needs Met Rating		Explanation
Query: [nasa mars landing] User Location: San Francisco, California User Intent: On the date this example was written, NASA had just landed the “Curiosity” rover on Mars. Most users are probably looking for breaking news and pictures about the event.			This LP has an article about the “Curiosity” rover landing, published on the same day as the landing, which is the day this example was written.
			This LP has an article about the launching of the “Curiosity” rover, published nine months before its landing on Mars. The content on this page would be considered stale for the query.
Query: [los angeles traffic] User Location: Los Angeles, California User Intent: Users are looking for current information about traffic conditions in Los Angeles. Even pages about traffic conditions the day before would be considered stale for the query.			This LP shows traffic conditions in Los Angeles on August 7, 2012, which is the day this example was written.
			This LP shows traffic conditions in Los Angeles on August 6, 2012. It is stale and useless for the query even though it is only one day old.
Query: [nobel peace prize winner] User Location: San Diego, California User Intent: Find the name of the most recent winner of this prize. This is a Know Simple query. At the time this example was originally written, 2011 was the most recent year that the Noble Peace Prize was awarded.			This LP on the New York Times website has information about the most recent winner of this prize, based on the date when this example was written.
			This LP on the BBC website has information about the 2006 winner of this prize. This page is stale for the query.

Query, User Location, User Intent	Result Block and LP, E-A-T Rating, Needs Met Rating		Explanation
Query: [arthritis] User Location: Philadelphia, Pennsylvania User Intent: Users are looking for information about this disease. Because there is ongoing research done on treatments and cures for most diseases, users are probably looking for authoritative, up-to-date information on arthritis.			Medline Plus is an authoritative website that regularly updates its content.
			This 2006 article would be considered stale for this medical query.
Query: [when was franklin d roosevelt born] User Location: Chicago, Illinois User Intent: Users want to find this well understood fact.			This LP has the information users are looking for.
	We couldn’t find a stale page for this query.

19.1           Misspelled and Mistyped Queries and Results

19.2        Misspelled and Mistyped Queries

You will notice that some queries are misspelled or mistyped. Here are some examples of queries that are obviously misspelled:

Misspelled Query	Query Interpretation
[federal expres], English (US)	The only reasonable query interpretation is the company named Federal Express.
[new england patroits], English US)	The only reasonable interpretation is the NFL football team.
[byonce knowles], English (US)	The only reasonable interpretation is the famous singer/actress, Beyonce Knowles.

Some misspelled or mistyped queries are more difficult to interpret. Use your judgment and do query research.

For obviously misspelled or mistyped queries, you should base your rating on user intent, not necessarily on exactly how the query has been spelled or typed by the user.

For queries that are not obviously misspelled or mistyped, you should respect the query as written, and assume users are looking for results for the query as it is spelled.

19.3        Name Queries

Consider the query [john stuart], English (US). There is a very famous Jon Stewart, the comedian and host of a popular U.S. television show. However, we should not assume that the query [john stuart] has been misspelled. There are many people named John Stuart. We will respect the query as written and assume the user is looking for someone named “John Stuart.”

Query, User Location, Possible User Intents	Result Block and LP, E-A-T Rating, Needs Met Rating		Explanation
Query: [micheal jordan], English (US) User Location: San Francisco, California User Intent: There are many people named “Micheal Jordan.” Helpful results for more prominent people named Micheal Jordan should receive higher ratings than helpful results for less prominent people named Micheal Jordan. Because Michael Jordan is such a famous athlete/celebrity, we will consider Michael Jordan to be a minor interpretation.			The page lists all members of the LinkedIn site named “Micheal Jordan.” LinkedIn is a very popular social networking website for professionals in the U.S. This result would be very helpful for many or most users trying to find information about someone named “Micheal Jordan.”
			LinkedIn page for Micheal Jordan, a facilities services professional in Atlanta, Georgia.
			Michael Jordan’s player profile page on the NBA basketball website.

20.0    Non-Fully Meets Results for URL Queries

Raters sometimes ask the question, “For a well-formed working URL query, are the only acceptable Needs Met ratings for a result either Fully Meets or Fails to Meet?”  The answer is no.  There can be other helpful results for URL queries.

Some users issue URL queries to find information about a website, such as reviews or recent news. We recommended this to you as one method of reputation research in the PQ guidelines. Real users do this too. Results which give reviews and reputation information can be very helpful for a URL query.

However, websites that offer usage statistics about a website are not usually helpful results for URL queries. Most users aren’t interested in this kind of information.

Query, User Location, User Intent	Result Block and LP, E-A-T Rating, Needs Met Rating		Explanation
Query: [coldwatercreek.com] User Location: Palo Alto, California User Intent: Coldwater Creek is a well-known U.S. company that sells women’s clothing and accessories online and in retail stores. Users may want to go to the website, do research, or find more information on the company.			This LP is the target of the query.
			Online coupon codes are very popular in the U.S. and this LP (from a well-known coupon site) displays codes for online purchases on the Coldwater Creek website. Users may be interested in coupon codes and this would be very helpful for those shopping online at coldwatercreek.com, especially frequent shoppers.
			This LP (from a well-known review site) has reviews of the coldwatercreek.com website. Users may be interested in reading reviews when considering whether to make purchases at coldwatercreek.com, especially new shoppers.

Query, User Location, User Intent	Result Block and LP, E-A-T Rating, Needs Met Rating		Explanation
Query: [ratemyprofessors.com] User Location: Los Angeles, California User Intent: Go to www.ratemyprofessors.co m, a website where students can rate their college professors.			This LP is the target of the query.
			This LP is a Huffington Post article dated August 31, 2012 about ratemyprofessors.com. The article was newly published when it was added to the guidelines.  Some users might be interested in this article.
			This LP is an article dated April 17, 2006 about the ratemyprofessors.com website. Few or no users would be interested in this outdated information.

21.0    Product Queries: Action (Do) vs. Information (Know) Intent

Some product queries, such as [ipad reviews], have a clear information-seeking (Know) intent. Other product queries, such as [buy ipad], have a clear purchase (Do) intent. And some product queries, such as [ipad store.apple.com], have a clear navigation (Website) intent. However, most product queries don’t obviously specify one type of intent.

Even though the ultimate goal may be to purchase a product, many other activities may take place first: researching the product (reviews, technical specifications), understanding the options that are available (brands, models, pricing), viewing and considering various options (browsing), etc.

Give high Needs Met ratings to results that allow users to research, browse, and decide what to purchase.

Important: E-A-T ratings for product results need extra care and attention.

Often, the results for product queries are YMYL pages. Users need high quality information from authoritative sources when researching products, especially when products are expensive or represent a major investment/important life event (e.g., cars, washing machines, computers, wedding gifts, baby products). When buying products, users need websites they can trust: good reputation, extensive customer service support, etc. Results for product queries may be important for both your money and your life (YMYL)!

Query, User Location, User Intent	Result Block and LP, E-A-T Rating, Needs Met Rating		Explanation
Query: [high definition tv] User Location: Los Angeles, California User Intent: This query doesn’t specify a size or brand of TV. Users are probably doing research before making a purchase.			This LP has comprehensive information about different types of high definition TVs (plasma, LCD, LED), as well as reviews about many specific models. The website is known for having helpful reviews and information about electronics.
Query: [girl toddler jeans size 3T] User Location: Los Angeles, California User Intent: The query is very specific. The user wants to browse and find an item to buy.			This is a helpful page for browsing and shopping for toddler girls jeans. There are many options for jeans in size 3T.  This is a high quality store known for selling jeans.
Query: [cufflinks] User Location: Los Angeles, California User Intent: Most users probably want to find different cufflink styles and options to browse or purchase.			This LP has a large collection of cufflinks from a very reputable retailer. There are many options and styles. This is a good page for browsing. There are helpful features at the top to choose styles, price ranges, etc.

22.1         Rating Visit-in-Person Intent Queries

When there is a user location for a visit-in-person intent query and a location has not been specified in the query itself, such as [chinese restaurants] with a user location of Boston, MA, results in or near the user location are the most helpful.

How close is “near”? The type of business and/or entity should be taken into consideration when deciding if the distance of the visit-in-person result is too far. For example, most people are not willing to travel very far for a gas station, coffee shop, supermarket, etc. Those are types of businesses that most users expect to find very nearby.

However, users might be willing to travel a little farther for certain kinds of visit-in-person results: doctors’ offices, libraries, specific types of restaurants, public facilities like swimming pools, hiking trails in open spaces,

etc.  Sometimes users may accept results that are even farther away, such as a very specialized medical clinic.

In other words, when we say users are looking for results “nearby,” the word “nearby” can mean different distances for different queries.  As always, please use your judgment.

22.2        Examples Where User Location Does (and Does Not) Matter

As previously mentioned, the user location may not always change our understanding of the query and user

intent. Below is one example where user location plays an important role, and two examples where user location does not matter.

Query, User Location, User Intent	Result Block and LP, Needs Met Rating	Explanation
Query: [minimum wage rate] User Location: Wichita, Kansas User Intent: Find out what the minimum wage is.  Different states in the U.S. have different minimum wage rates. Users might be interested in the minimum wage in their state and how it compares to other places.		This Special Content Result Block displays the complete and correct answer prominently in the result block, in a way that is easy for users on mobile devices to read. Note: Assume that the result shows current information for users.
		This is the official Kansas state website page on minimum wage for Kansas.
		This LP has the minimum wage rates for all states. It is on the official U.S. Department of Labor website, which has high E-A-T on the topic, but it takes some work to find the information specific to Kansas. On a mobile phone, you may want to select the Hard to Use flag. Please use your judgment.
		This is the official California state website page on minimum wage for California.

Query, User Location, User Intent	Result Block and LP, Needs Met Rating	Explanation
Query: [pictures of kittens] User Location: Pittsburgh, Pennsylvania User Intent: Users are looking for pictures of kittens. This is a non-visit- in-person intent query. There is no obvious user intent to find pictures of kittens in Pittsburgh. The user location plays no role in the Needs Met rating.		The LP is a page full of kitten pictures. The result block has a nice assortment of pictures as well that would highly satisfy users.
		This is a local listing of pets needing homes in the Pittsburgh area.  There are no pictures of any pets directly on this page and few pictures on the individual listings.
Query: [google jobs] User Location: Miami, Florida User Intent: Users are looking for job opportunities at Google.		The user is looking for a job at Google and there is no information in this result block that will help the user explore job opportunities—it Fails to Meet the user need.
		This is Google’s official page on careers and jobs at Google, where users can search for currently open opportunities. This page is exactly what the user is looking for—it Fully Meets the user need.

Part 4:  Using the Evaluation Platform

23.0   Introduction

Welcome to the Evaluation Platform! The Evaluation Platform is the system you will use to acquire and rate tasks.

24.0   Accessing the Evaluation Platform (EP)

Go to this link to access the Evaluation Platform: http://www.raterhub.com/evaluation/rater.

25.0   Evaluation Platform Screenshot

Here is a screenshot of the task acquisition page on the Evaluation Platform. Please note that changes are sometimes made to the appearance of pages on the EP, so it may not look exactly as it appears here.

The red numbers represent the following:

1.     Tasks

This text shows that you are at the task acquisition page on the Evaluation Platform.

2.     rater@gmail.com

Your Gmail account.

3.     Recent tasks

Click this link to revisit tasks completed in the last few minutes.

4.     Sign out

Click this link to end your rating session.

5.     Acquire if available

Click this link to acquire a new task.

6.     Experimental

This text indicates that there is an Experimental task available. Please note there are other types of rating tasks (e.g., Result Review, Side-by-Side). Sometimes there will be only one task type available, but other times you may see more than one type displayed. When there is more than one type displayed, you may choose the type you want to acquire.

26.0   Needs Met Task Page Screenshot

The red numbers represent the following:

1.     rater@gmail.com

Your Gmail account.

2.     Recent tasks

Click this link to revisit tasks completed in the last few minutes.

3.     Sign out

Click this link to end your rating session.

4.     Mobile

This text indicates that the current task belongs to the Mobile property type. Other properties you may rate include Web, Video, Image, etc.

5.     Experimental

This text indicates that the current task belongs to the Experimental project type. Other project types you may rate include Result Review, Side-by-Side, etc.

6.     Average Estimated Time

This text indicates the average estimated time for completion of the task.

7.     Instructions

Click this tab to view the project-specific instructions. Some tasks display project-specific instructions on the task page instead of behind the Instructions tab.

8.     Query

You should understand the query before rating the task.

9.     Locale

This refers to the task location and task language associated with the query.

10.  Report a Problem / Release this Task

Click this link to report a problem and/or release the task.

11.  User Location

This refers to where the user is located when the query is issued. Note that if there is no user location listed, there will be no user location-related map displayed in the rating task. In this case, the task could still show a map if one of the results has a response pin.

12.  Response Pin

This is a marker for a visit-in-person place.

13.  Approximate User Location

This area represents the approximate location of the user when the query is issued.

14.  Needs Met Rating Slider

Use this rating slider to assign a Needs Met rating.

15.  Porn Flag

Assign this flag if it applies to the result.

16.  Foreign Language Flag

Assign this flag if it applies to the result.

17.  Didn’t Load Flag

Assign this flag if it applies to the result.

18.  Hard to Use Flag

Assign this flag if it applies to the result.

19.  Comment

Use the result block comment box to write comments that will be helpful to you when assigning ratings. Note: All comments must be written in English.

20.  Special Content Result Block

This result block shows content directly in the block.

21.  Result with Response Pin

The response pin in this result indicates the location of the corresponding response pin on the map.

22.  Web Search Result Block

Click the block to visit the landing page.

23.  Dupe Confirmation Checkbox

Check this box to confirm that you are done flagging duplicate results.

24.  Submit

Use this button to submit the task and continue rating.

25.  Submit and Stop Rating

Use this button to submit the task and stop rating.

26.  Cancel

Use this button to cancel any ratings you have assigned in the task.

26.1   Understanding the User Location on the Task Page

While most tasks have a user location listed below the query, the map can also help clarify where the user was physically located when the query was issued and is more precise than the user location listed below the query.

On the map, the user location can be precise (blue dot) or approximated (blue rectangle). Here are some examples of what you might see at the top of the task:

A precise user location is represented by a blue dot ).	Sometimes you will see a shaded blue circle around the blue dot. The blue dot is user’s most likely location, but it is possible they are located anywhere within the larger shaded blue region.
The approximate user location is shown as a blue rectangle on the map.	The map includes a response pin, which is a marker for a visit-in- person result block in the task. If there are three visit-in-person result blocks in the task, there should be three response pins on the map.

27.0   Notes about Using the Needs Met Rating Interface

Result blocks are sometimes slow to load: Before assigning any ratings, please scan the task page to make sure that all result blocks with a Needs Met rating slider have loaded. Please note that result blocks may be slow to load and/or that the task page may need reloading if there are empty result blocks. If the result blocks fail to load after refreshing a few times, please release the task.

Sometimes a side will not have any result blocks: If one side displays the message “This side did not generate any results,” please do not release the task.

Do not refresh the task page after assigning ratings: If you refresh the page after assigning ratings, you will lose them and they will have to be entered again.

28.1        Using the “Report a Problem / Release this Task” Button

You will report problems and release tasks using the “Report a Problem / Release this Task” button on the rating interface.  This is what the “Report a Problem / Release this Task” button looks like:

Clicking the red button displays the following reporting and release options:

Here are some examples for the “lack expertise” and “technical problem” release categories.

Lack expertise: These queries require specialized knowledge and may be difficult for some raters.

Query	General Query Topic
[cortisol secretion shows a blank pattern of secretion]	Scientific query
[minecraft gun and vehicles 1.1 mod installer]	Gaming query
[python pandas index like matrix]	Technical query

Technical problem: These result blocks have question marks and other characters where text should be.

Note: Based on the number and/or type of tasks that you release, you may be asked to provide additional details about the reason for some of the releases.

29.1        Reporting Results with Duplicate Landing Pages

In addition to assigning Needs Met ratings to each result block, we would like you to tell us whether any results are duplicates (dupes).

29.2        Pre-Identified Duplicates

Some duplicate results are automatically detected and will be pre-identified for you. They will be highlighted by a grey or green background, and annotated by either “Same as...” or “Dupe of...” text right below the slider and above the result block.

This is what these pre-identified dupes look like:

Please note:

·         Results annotated by “Dupe of...” may be given different ratings/flags/comments.

·         Results annotated by “Same as…” may not be given different ratings/flags/comments. Their Needs Met and E-A-T ratings, flags, and comments will be automatically transferred to each other.

·         You cannot un-check dupes that have been automatically detected and pre-identified.

29.3        Rater-Identified Duplicates

We would like your help identifying duplicate results that have not been automatically detected.  Please mark two results as dupes if they have essentially the same content on the main landing page AND you would not want a search engine to return both results for the query.

Please note that in this project dupe identification is query-dependent.

·         Specific queries: For queries where the user is looking for a specific piece of content (such as queries looking for song lyrics, queries looking for a specific article, etc.), obtaining that piece of content from different sites could be helpful for users to verify the information, so they should not be rated as dupes.

·         Broad queries: If the query is broad, then returning the same piece of content is not what the user is looking for, and hence those results should be flagged as dupes. Results may be considered dupes even if they have different minor content on the page (such as different ads, images, or related links).

Please identify dupes both within the same side and across sides. Even for cross-side results, you should still ask yourself the question, “Would you want to see both results if they were returned by the same search engine?”

Note: If two result blocks have very different types of content or very different appearances, do not mark them as duplicates even if they have the same landing page URL. For example, these two blocks should not be marked as duplicates of each other.

Reporting Duplicate Results

When you notice that the results in two or more result blocks are duplicates, please click on the Select Dupe link at the top of one of the results. The current result (the result you clicked on) will be highlighted in red at the top of the block.

You can then check results that are duplicates of the current result, and the checked results will be highlighted in yellow.  You must select the “Dupe of…” button.

The duplicate results that you checked will be annotated by "Dupe of..." text at the top of the block. After selecting all dupes, click on the finish selecting dupes link to return to the normal rating mode. The link’s name will change back to Select Dupes, and you will be able to report other sets of dupes, if there are any. If you change your mind, you can always un-check a result.

Dupes

QUERY: [choosing and installing a motorcycle battery]

URL 1: http://www.caimag.com/wordpress/2010/03/06/motorcycle-battery-how-to-choose-install

URL 2: http://www.articlesbase.com/motorcycles-articles/choosing-and-installing-a-motorcycle-battery-47798.html Reason: Both of these results display the same article (which also appears on many other pages on the web). The only real difference between the landing pages are the Ads displayed around the article. The query is broad enough that users would not benefit by search engines returning more than one of these results.

Dupes

QUERY: [jason castro]

URL 1: http://www.myspace.com/jasoncastromusic

URL 2: http://www.myspace.com/jasoncastromusic?MyToken=503599bf-01cf-4427-bdf4-d63920c107f9

Reason: These two results have the same landing page, even though the URLs are different. Users would not benefit by search engines returning both results.

Not Dupes

QUERY: [material girl lyrics]

URL 1: http://www.lyricsfreak.com/m/madonna/material+girl_20086925.html

URL 2: http://www.lyrics007.com/Madonna%20Lyrics/Material%20Girl%20Lyrics.html

Reason: Even though both pages display the lyrics to the song “Material Girl,” users would probably want to have the option to visit both pages so that they could verify the accuracy of the lyrics. Users could benefit by search engines returning more than one page with the lyrics to the song.