UDiscoverIt Tutorial/Qualification Overview

This tutorial is intended to demonstrate the key concepts to be used in the UDiscoverIt HITs and to test your understanding of these concepts. You may repeat the test if needed to reach the required score.

Estimated Tutorial Time: 20-30 minutes. Estimated HIT time: 30-60 seconds per HIT.

The task to be performed in the HITs covered by this tutorial will be:

Look at an image similar to the ones that appear in this tutorial
Observe which vertical lines/blobs form patterns with each other, according to the rules below
Choose the clearest pattern as explained below. We will call this the Primary Pattern.
Identify some vertical lines/blobs that best define the Primary Pattern by clicking on them
Identify some vertical lines/blobs that best define Secondary Pattern(s) (other patterns besides the Primary Pattern) by clicking on them

The tutorial begins with the criteria for what features constitute a pattern, what can't be part of a pattern, and what features obscure the details of a pattern. This is followed by criteria for choosing which pattern to describe if you see more than one, and which lines/blobs to mark.

The tutorial will be broken into sections introducing groups of related concepts, with questions at the end of each section.

The examples (and related HITs) in this tutorial sometimes include numerical labels for features in the images (these are vertical lines/blobs, and horizontal row gridlines). If the labels are difficult to read, they are numbered sequentially, almost always beginning with 1. The questions are designed so that mistakes are unlikely due to mis-reading the numbers. In the tutorial, you can zoom when necessary by clicking the images.

In the tutorial, you will be asked to identify features like vertical lines/blobs or horizontal gridlines by numerical ID. However, in the real HITs you will identify vertical lines/blobs by clicking on them to toggle selection.

The colors of the vertical lines/blobs can be thought of in terms of brightness or intensity. The relative values are given by the following image. Low values are on the left (dark blue), and high values are on the right (red, white):

HIT Requirements and Helpful Skills

Javascript support (used to collect responses interactively)
A mouse (or equivalent) to click on specific image locations
Screen resolution of at least 1024x768 is recommended
This is a visual task, which requires discrimination of color-based patterns
This task involves the recognition and identification of abstract patterns

The tutorial is expected to take around 20-30 minutes to complete, and individual HITs are expected to take around 30-60 seconds each. The qualification test is graded automatically and you will be able to work on HITs immediately if you meet the minimum score indicated on the HITs. You may take the test multiple times if needed.

Academic Research Purpose

Responses for the HITs associated with this tutorial/qualification will be collected as data to be used for academic research purposes. Please indicate whether you accept this. If not, you can stop here (answering no will result in a failing score on the test).

I accept that my HIT responses for any tasks requiring this qualification will be used for academic research purposes.

I do not want my responses used for academic research and will not attempt to work on any HITs requiring this qualification.

Relationships Allowed in Patterns

The following are examples of the types of relationships that are likely to occur with patterns, paired with examples of relationships that can't be part of a pattern. Explanations are given for each. Typically, when you see many vertical lines/blobs that all have an allowed relationship, it is very likely to be a pattern. But when any pair show one of the incorrect relationships, they are almost certainly not a pattern.

Incorrect vs. Permitted Pattern Relationship Examples	Explanation
	Lines/blobs in the same pattern can have different widths, but not different vertical lengths.
	Lines/blobs in the same pattern must have their tops and bottoms aligned. Alignment is determined by the horizontal gridlines that are crossed, so the lines/blobs in the correct example are aligned even though the middle one might appear shorter.
	Lines/blobs in the same pattern can have different overall intensities and the intensities can change along their lengths, but they must change the same way.
	Lines/blobs in the same pattern can shift diagonally, but they will almost always shift together. Note: the diagonal shift isn't necessarily parallel- it is expected to be slightly steeper for vertical lines/blobs farther to the right in the image. Shifts that don't match should make you much less confident in a pattern, however two lines with different shifts is not sufficient to say they are definitely in different patterns.
	Lines/blobs in the same pattern can change widths along their lengths, but they must change together.

Question: Properties of Patterns

You see a group of five vertical lines/blobs with the properties described below. Which description most likely refers to a single clear pattern?

All five start and end at the same horizontal gridlines (rows). The first blob is brighter than the others, but all are brightest at the top of the image, and darker towards the bottom of the image.

All have exactly same intensity, but different lengths.

Four blobs have uniform intensity, fifth blob gets brighter towards the top of the image.

Four blobs all have constant width, fifth blob gets fatter towards the top of the image.

Relationships that Obscure Patterns

The following are examples of the types of relationships that make it difficult to identify a pattern, or make it difficult to identify specifically which lines/blobs are part of the pattern. However, they might still provide useful context for more clear lines/blobs.

Generally, you should be less certain of any pattern that relies on lines/blobs with these relationships. Also, when choosing lines/blobs to mark, it is best to avoid marking lines/blobs with these relationships, unless all of the other options are more obscured.

In some cases, it would be possible to infer a pattern from lines/blobs with these relationships, but it will typically require too much effort. It's better to spend the time on easier ones.

Explanation	Obscured Pattern Relationships Examples	Explanation
A single vertical line/blob generally will not provide enough information to identify a pattern on its own.		Vertical lines/blobs that only intersect a single horizontal gridline generally do not provide enough information to identify a pattern, the way they change is not visible. Note: there will usually be other images in the series providing more information about such patterns.
It is difficult to see the extent and changes in very faint lines, therefore it is best not to include these in pattern assignments, or to rely on them for pattern identification. A group of lines being faint or invisible does not make them a pattern.		Sometimes vertical lines/blobs will appear in almost exactly the same horizontal position. In some cases, they will have separate markers/labels as shown here (in which case you will be prevented from selecting them), and in other cases you must infer this from the visible changes and context. Do not mark these, because it is either impossible to choose the correct label, or difficult to make out their relationship to a pattern.
Lines/blobs from a pattern can be overlapping. In these cases, again it is best to look for more clearly separated examples because the variations are obscured.		Lines/blobs from different patterns can be overlapping, which again obscures the variation in an individual pattern.
Generally when vertical lines/blobs from the same pattern shift diagonally, they do so together. However, it is possible (though unlikely) for them to have different shift angles. This makes them unlikely to be a pattern, but is not sufficient to be certain they are in different patterns.		Though unlikely, steep shifts can introduce visual artifacts, where the blobs appear "stepped" or disconnected. There appear to be two pairs of blobs in this image, when there is "really" only one pair. The only way to tell the difference is context from other features, and it is not worth trying to decide this. In general, don't assume one way or the other (that they are a continuation of the same blobs, or new ones). When possible, only make selections that would be correct either way.

Questions: Obscured Patterns

Why is it best to ignore, or discount the value of, vertical lines/blobs that overlap each other (check all that apply)?

It requires too much effort to infer the underlying relationships

It makes it more likely to make an error in interpreting the shape of the lines/blobs

It is impossible for vertical lines/blobs to overlap each other

It is hard to precisely identify which line/blob belongs to the indicated pattern.

You see in an image a group of 4 lines/blobs that are all too dark to make out details. They are:

Definitely their own pattern separate from more visible lines/blobs.

Too uncertain to assign to any pattern, or to explicitly exclude from any pattern, and therefore should be left unmarked.

Definitely part of a pattern including clearer lines in the same region.

A pattern together if there are no other lines in-between.

Pattern Identification Examples

Simple Example: 1 pattern

This image has 3 vertical lines/blobs and 1 likely visible pattern. The two blobs on the upper-right are a pattern because:

The top and bottom are aligned and they are the same length
They shift diagonally in the same way
They change brightness in the same way along their length
We would be more certain if more blobs were involved

It doesn’t matter that:

They are not the same overall brightness
They are the same width

The bottom-left blob is separate because:

The top and bottom are not aligned with the others
The diagonal shift, change in brightness and length are all different
It is part of some pattern, but there is not enough information to specifically identify it

Simple Example: 2 patterns (blobs added to previous example)

This image has 5 vertical lines/blobs, and 2 patterns. The bottom-left blob forms a pattern with the two blobs on the bottom-right because:

The top and bottom are aligned and they are the same length
They shift diagonally in the same way
They change brightness in the same way along their length

It doesn’t matter that:

They are not the same overall brightness
They are the same width
There are lines from a different pattern between them
The diagonal shift isn’t perfectly parallel (it is expected to be slightly steeper towards the right side of the image)

Questions: Pattern Identification

For the next 3 questions, refer to this image (click to zoom):

Question Image: Choosing primary pattern

What is the most likely total number of patterns visible in this image, assuming each vertical line/blob is part of a pattern? (This is your best guess for the total number, not just clear patterns with un-obscured blobs.)

Find the clearest, most identifiable pattern in this image. We will call this the primary pattern. Vertical lines/blobs are labeled with numbers at the top and bottom of the image. Which numbered blobs are clearly and unambiguously part of the Primary Pattern?

1, 5

4, 5, 6

4, 6

2, 3, 4, 6

Why do these form a pattern (check all that apply)?

The tops and bottoms of each vertical line/blob are aligned

There are no other vertical lines/blobs between them

They have similar changes in brightness along their lengths

They have the same overall brightness as each other

Target Row Definition and Question

The Target Row is indicated by a series of red-and-white circles marking each vertical line/blob. The target row indicates the region of the image to look for patterns, in other words, we are looking for patterns that intersect this row. Note that observing patterns elsewhere in the image can still provide useful context. Refer to this image for the following question (click to zoom):

Which row is the target row in this image (row gridlines are numbered at the left side of the image, with #1 at the top and #7 at the bottom)?

Primary Pattern vs. Secondary Patterns (click to zoom)

If there are multiple patterns crossing the Target Row, you will choose the clearest pattern and call this the Primary Pattern; any remaining patterns are called Secondary Pattern(s). The main idea is to choose the pattern for which you can provide the most certain response. Use the following criteria to determine which is the best to select as the Primary Pattern:

Has the most distinguishable vertical lines/blobs.
Has the largest number vertical lines/blobs that you can identify with certainty.
Has longer vertical lines/blobs. They must cross a minimum of 2 horizontal gridlines
The pattern is very clear at the Target Row, i.e. the Target Row is not at the top or bottom edge of the pattern.

If two or more patterns have similar clarity according to these criteria, just pick the one that makes it easiest for you to provide an answer with certainty.

This image appears to have 4 (or more) patterns, with a single vertical line/blob. Only 2 (or more) of them intersect the target row. We call the clearest pattern the Primary Pattern, and any other patterns are the Secondary Patterns. There can be zero, one or two Secondary Patterns, but we treat them all the same way and consider them together.

In this example image, we have chosen a Primary and Secondary Pattern, and we have marked the vertical lines/blobs that best identify each, and reasons are given below.

Primary Pattern (orange marks):

Crosses at least 2 horizontal rows
Longer
More distinct vertical lines/blobs

Secondary Pattern(s) (purple marks):

Only crosses 1 row (can’t tell if it’s a single pattern)
Many potential lines obscured

Other patterns:

The 4 big lines at the left, and the 2 at the top-right are probably part of other clear patterns
They don’t intersect the target row so they are not part of the answer
But they can help us see the Primary/Secondary patterns

Questions: Overlapping Vertical Lines/Blobs

For the next 2 questions, refer to the "Obscure Patterns" section of the tutorial, and apply to this image (click to zoom):

Question Image: Overlapping Vertical Lines/Blobs

Which vertical lines/blobs should we exclude from responses because the labels overlap, meaning they are too close to tell apart?

1, 2, 3, 4, 8, 9

5, 9, 10

6, 7

All can be included

Which vertical lines/blobs might we exclude from responses because the blobs overlap even though the labels are separated, meaning it is hard to see if they are in a pattern?

1, 2, 3, 4, 8, 9

5, 9, 10

6, 7

All can be included

It is questionable whether blob #1 should be marked as part of the Secondary Pattern(s). Why might we exclude it?

It appears to be part of the Primary Pattern.

It overlaps with other nearby blobs, and might be partly obscured.

It might not be part of the same Secondary Pattern from the other ones that are marked.

It is too faint to see clearly.

It is questionable whether blob #8 should be marked as part of the Primary Pattern. Why might we exclude it?

It is clearly not part of the Primary Pattern.

It overlaps with other nearby blobs, and might be partly obscured.

It doesn't intersect the Target Row.

It is too faint to see clearly.

Example: Harder Images (click to zoom)

Some images will have lots of vertical lines/blobs that seem likely to be patterns, but with many overlapping or faint ones.

The Primary Pattern is the one for which we can choose the best set of vertical lines/blobs to define it with the most certainty.

In this case, it is best to have the Primary Pattern be the ones that extend to the bottom of the image, because all of the lines/blobs extending to the top of the image overlap with other lines/blobs.

The Secondary Pattern(s) would be those vertical lines/blobs that extend to the top of the image and therefore can’t be part of the Primary Image.

Note: we can't always tell how many patterns there are. In this image, there are at least 2 patterns, but it's possible there are more. We would leave many of the vertical lines/blobs unmarked in this image because they are likely to be part of the Primary or Secondary Patterns, but we can't be sure.

Questions: Harder Images

The next 2 questions refer to the same image as in the preceding example. Again, we will call the clearest pattern extending to the bottom of the image the Primary Pattern, and the one(s) extending to the top of the image the Secondary Pattern(s). Here is the image again, without the box indicating the Primary Pattern (click to zoom):

To choose vertical lines/blobs that define the Primary Pattern, start with the one line/blob that you are most certain is part of the pattern. Always include at least one blob in the Primary Pattern, which will set the level of certainty needed for selecting others. Then add any additional lines/blobs for which you are just about as certain (if any are nearly as certain).

Which is the best list of lines to define the Primary Pattern (bottom)? The "most certain" line/blob is listed first, followed by the others.

4, 1, 6, 9

15, 16, 17, 18

6, 9, 16

Don’t assign any to Primary Pattern (all are too obscure)

To choose vertical lines/blobs that define the Secondary Pattern(s), identify those lines/blobs that are clearly part of pattern besides the Primary Pattern, and that you can assign with the same level of certainty as the ones in the Primary Pattern.

It is possible that all of the vertical lines/blobs are significantly more obscured than the Primary Pattern, in which case it is best not to mark any as definitely belonging to Secondary Pattern(s).

Which is the best list of lines to define the Secondary Pattern(s) (top)?

2, 5, 7

1, 3, 4

7, 18

Don’t assign any to Secondary Pattern(s) (all are significantly more obscure than the Primary Pattern)

Final Tips / Rules of Thumb

The following are some rules that you might find helpful for inferring additional information about the patterns you see:

A pattern can include any number of vertical lines/blobs, but there can be 1, 2, or 3 patterns present at any horizontal gridline (they might not all be easily identifiable).
There is no limit to the number of patterns that can be present within an entire image, except according the the per-horizonal-row limit.
Buttons are provided on HITs to change the color scale to emphasize the brighter or fainter features in an image; these can help you make out patterns more clearly.