Many thanks to the Cornell Usability Working Group for carrying out the usability tests and recording the results.  Special thanks to Wendy Kozlowski and Kevin Kidwell who facilitated the tests and took notes. After providing some context around the tests and high level outcomes,  we are including the text of the Usability Working Group's final report  below.  The report is also available here (TO DO: Link). 

GOAL

Understand whether the labeling and display of autosuggest results for authors, subjects, locations and genres makes sense to end-users, and gather feedback on the inclusion of variant, pseudonym, and see also information in the autosuggest.

METHODOLOGY

Huda discussed the goal for this study as well as potential tasks with the Usability Working Group.  The group discussed the format and helped refined the tasks for the study.  Kevin Kidwell and Wendy Kozlowski took notes and facilitated the sessions and the Usability Working Group formulated the final report included below.  The tasks used in the study with accompanying screenshots showing possible interactions in the interface are available here.  Given the current situation with the pandemic, and an already over-taxed faculty and student population, we decided to recruit participants from user representatives who work with the Cornell Blacklight discovery team.  We used Zoom to enable participants to share their screens and provided them a URL they could use to access the website and attempt the tasks.  Their interactions with the screen were recorded while not recording their faces.  The recordings are linked below (TO DO: provide a link). 

PARTICIPANTS

5 user representatives from Cornell staff

SCHEDULE

June 22-25, 2020

KEY LEARNINGS

  • In general, participants were able to use the information displayed in the autosuggest to distinguish between types of entities and to identify headings related to variant labels or connected using "see also" properties.

    • Participants were able to find authors, subjects, locations, and genres when asked to find entities from each of these types in the tasks.

    • Participants were able to use the number in the autosuggest results to answer the question regarding the number of items in the catalog for a particular topic.

    • Participants were able to use the descriptive text retrieved from Wikidata to distinguish between authors with similar names but different occupations.  One participant did want to see if typing in the occupation along with the name would help in retrieving results, but we noted that we are not matching on occupation text.

  • For both authors and subjects, all participants were able to search for a variant label and find the preferred heading. Most participants understood what the term "aka" stood for in the results and thought this display of information was useful. When searching for an author, one participant indicated they weren't sure which heading was authorized since both the variant and preferred versions ended with date strings.

  • For the shared pseudonym example, participants were able to find the names linked with "see also" but it was not clear to all that the searched name was a pseudonym shared between the linked names. Suggestions for clarifying this connection included using the term "pseudonym".

  • 4 out of 5 participants appreciated the resulting knowledge panel on the results page as a way of confirming the search they had conducted.
  • One participant noted the discrepancy between the authors and subjects where the former shows descriptive information from Wikidata while the latter does not.
    • We suggest that future work should explore how to provide consistent information about the same person regardless of whether the URI returned is for an authorized name or a for a subject heading. 

June 2020 – LD4P2 Auto-Suggest Report

Usability Working Group

Summary:

  • This study had 5 participants
  • Participants were identified by stakeholders

Task 1:

Find library resources written by Carl Einstein born in 1885.

Task 1 Summary:

  • All users found the proper author.
  • The auto-suggest feature helped a couple users to find the correct spelling of this author’s name

Task 2:

Find library resources about Carl Einstein born in 1885.

Task 2 Summary:

  • Half of the users would typically change the field from Author to Subject without going into the search again.
  • All users were able to search again and chose the author under the Subject heading

Task 3:

How many items exist in the catalog for any topic related to heart disease?

Task 3 Summary:

  • All users saw the numbers in the auto-suggest. A couple had to be directed back to the auto-suggest, as they would type and search then see the count on the results page.

Task 4:

How many search results are of the TYPE historical fiction?

Task 4 Summary:

  • All users decided to give the count from the Genre heading.
  • Users were asked the difference between Genre and Subject. It was stated that Genre was for items that fit into that specific search topic and Subject was for items that were written about the search topic.

Task 5:

Can you find any library resources about a place?

Task 5 Summary:

  • All users were able to search for a place and saw the Locations heading.

Task 6:

You found a book authored by “Dzheyn Edems” and want to find if the library catalog has additional resources.   Can you find library resources authored by Dzheyn Edems? Was this author known by any other names?

Task 6 Summary:

  • All users were able to search for this author and saw the auto-suggest with ‘aka’ shown.
  • One user wasn’t sure which was the authorized author and which she should be using, since both had the dates listed after the name.
  • 4/5 of users understood what ‘aka’ stood for and thought it was useful.

Task 7:

The library catalog has several resources authored by Mark Twain.  How many library catalog resources did Mark Twain author under a different name? Can you search for resources for this different name?

Task 7 Summary:

  • All users typed some variation on Mark Twain and found Samuel Clemens.
  • All were able to see that 12 titles were written under Clemens.
  • 4/5 of users clicked on Clemens to search for only those titles.

Task 8:

Which authors in the catalog are related to Jaynes, Roderick?

Task 8 Summary:

  • All users were able to find the Coen brothers in the auto-suggest under this name.
  • 2/5 of users weren’t sure what ‘see also’ would mean in this situation and recommended using ‘pseudonym’ instead.
  • 2/5 of users said that ‘see also’ made sense and that it means they are related to the current search and might be useful for what is being searched.

Task 9:

Is there another name that the catalog uses for “heart attack”?

Task 9 Summary:

  • All users found this as soon as they typed ‘heart attack’ in the search box.
  • One of the users searched for the term and liked the definition of the term in the Wikidata box.

Task 10:

Search for resources written by the civil engineer Max Weber. How do they input this to search, using what terms? Do they see the description next to the name in the auto-suggest?

Task 10 Summary:

  • All users searched for a variation on Max Weber and were able to see him in the auto-suggest.
  • All users noticed the descriptive text and used that to be able to choose the correct author.
  • One user tried typing the occupation in the search box to see if that would populate the same auto-suggest and would have expected it to do so.

Task 11:

Select the author from the previous task.  On the resulting page, please provide us any feedback on what you expected to see and what you are seeing. What more can you describe about Max Weber?  Is this information helpful?

Task 11 Summary:

  • 4/5 of users liked the Wikidata information box and found it useful for being able to confirm that they had searched for the correct author.
  • One user said that the digital collections were “cool but confusing” and weren’t sure if they were included in the catalog or existed somewhere else.
  • One user thought in the auto-suggest that this author’s occupation was listed under the Author heading but not the Subject heading.

Task 12:

Would something else be more helpful for you?

Task 12 Summary:

  • One user expected the title of the item under Digital Collections to go to the item in the collection.
  • One user didn’t notice the author’s name in the header area of the Wikidata box at first.
  • One user was curious if students would see and use the auto-suggest, since many students type quickly, and press enter to search automatically.
  • If you type a search and hit enter, what option (if any) is selected from the auto-suggest by default?



Find library resources written by Carl Einstein born in 1885.

typed in einstein, and saw that when trying to type carl with a k it had the autosuggest with a C so typed it with that.

Typed in the name and sees the auto-suggest and sees author and subject. Selected author to get to those written by him.

Typed in the author - had been intending to filter to author and clicked on the first one because of this.

Typed in Carl Einst - Found the author in the auto-suggest and did the auto search. Looked to see if the person with 1885 was there and was.

Goes to search bar and tried guessing how to type Carl and was looking for it with Karl and found the auto-suggest and clicked on it from the auto-suggest and searched.



 Find library resources about Carl Einstein who was born in 1885.

Went to use the Field box to select Subject as used to - Did the search again and then saw it as a Subject term and assumed that would take you there.

Interesting that the autosuggest changed the search type as well as selecting the option. Didn't notice that the first time. Sees that she could pick authors or subjects.

Would change the Author search field to Subject. Then was able to type the search again and clicked on the author under Subject.

Went to search and did backspace to get auto-suggest, then selected him as subject and it did the search. Was able to see that one section was about authors and one was about subject.

First instinct was to go to single title and then click on the info icon. Then saw the Works About in the wikidata panel. Then did the search and selected from under Subject and found information about him.


 

How many items exist in the catalog for any topic related to heart disease?

Used the Start Over button to clear everything - typed heart disease - sees the five different suggested subjects - congenital heart disease has 24 items based on the number next to the title in the auto-suggest.

Typed it in and saw all of the information in the auto-suggest and then went to go to search and saw the total there. Then was directed toward the auto-suggest and mentioned about the number next to the items.

Typed in heart disease and then clicked on Coronary Heart Disease and searched. Saw the count options in the auto-suggest.

Typed Heart disease - and found the subjects - would click on the one with the biggest number - number means how many entries are in the catalog for that.

Is thinking about something to type in, then typed heart disease. "Oh neat" - was going to look for heart diseases in women - Saw that there were 5 topics listed and that there were 25 items for that.


How many search results are of the TYPE historical fiction?

Typed historical and saw Genres below, decided on genre with 1337

Typed historical fiction and went with 1337 from Genre. Genre is what the item is and subject is what they are about.

Typed in historical fiction and was wondering about the difference between genre and subject in this situation - if wanted to items in historical fiction would go with genre but if wanted something about someone who wrote historical fiction would go with subject

Typed historical fiction - sees Genres and Subjects - saw the numbers next to each to see how many for each - Genre interpreted as something to read, Subject more likely to be more like an academic book about the topic of historical fiction

Typed historical fiction and then noticed the two different headings for Genre and Subjects. Wasn't sure if there was a limit on what is being populated in the auto-suggest box. Would say the 1337 under Genre heading.

 

Can you find any library resources about a place?

Typed Bahamas - found the resources in the dropdown - saw that it said Locations and would be useful

Searched for Nashville - saw the Locations heading and would be able to click on that.

Searched for Boston, then saw the Location under the auto-suggest and clicked on it to get the results.

Searched Tompkins County and then selected one of the options. Found resources right away.

Chose Los Angeles - saw that there were a number of different options under a heading of Locations


You found a book authored by “Dzheyn Edems” and want to find if the library catalog has additional resources.   Can you find library resources authored by Dzheyn Edems?Was this author known by any other names?

Put in the name with a copy/paste - the aka is really handy - looks like it's the same person - would click on Jane Addams

Typed in Dzheyn and is prompted to with the author and subject - if only knew her by this name, would get directed to the other name. Very clear what she's seeing.

Typed it in and then saw the aka, was comfortable clicking on that first one - knows that was known as Jane Addams

Put that in and saw that it was an AKA for Jane Addams and saw 81 titles by that author

Took some time to figure it out. Auto-suggest doesn't pop up with copying and pasting the name in the box. - With the date added to both of them, wasn't sure which was the authorized and wasn't sure which she should be using. Can see why one is known as and the other is the authorized heading. Was also known as Jane Addams.



The library catalog has several resources authored by Mark Twain.  How many library catalog resources did Mark Twain author under a different name? Can you search for resources for this different name?

Typed Mark Twain - looks like there are 12 under Clemens - hovered over and clicked just on Clemens to get just those - was a bit thrown off by the number in the auto-suggest not matching the catalog results number - would be good to let Ref staff know about that when put into production

Typed twain, mark - found the Samuel Clemens in the auto-suggest - see also meant that he wrote under two different names - was able to see how many he wrote under clemens

Typed in mark twain's name and saw the see also with Samuel Clemens and was able to find those by clicking on that name.

Typed in Mark Twain and sees that there is a See Also for Samuel Clemens with 12 titles, noticed they are separate sections able to be clicked on.

Searched for Twain and then found Samuel Clemens and was able to find that 12 existed. Was able to click on that name and search for that information for Clemens.


Which authors in the catalog are related to Jaynes, Roderick?

Copy/pasted the name - wouldn't automatically know what See Also means - clicked on Jaynes, Roderick and was looking for something that said 'related to' - would have expected a bio or something else about this author - if got to the page and scrolled down to see Related people and organization searches it would make sense if seeing the same names - If she knew more about the coen brothers, it would be obvious with the background knowledge, this is a unique situation that might make sense to somebody who knows they're searching this - aka is not quite right in this situation - see also is probably the best to go with this situation - pseudonym might be a good one to use if possible

Found Joel and Ethan Coen in the catalog - see also still makes sense. If a patron would be wondering why he'd be using 3 names but as getting around the catalog it worked. Good authority record question.

Typed it in and saw the see also with both Coens - wasn't sure what see also would mean in this situation unless Jaynes was a pseudonym for both of them.

Copied and pasted the item in and saw both Coens - see also means to her that it is related and may be useful for what you're looking for

Searched for that and found the Coen brothers in the auto-suggest.



Is there another name that the catalog uses for “heart attack”?

Myocardial infarction - expected to have an aka or see also - would be really helpful when people aren't sure what words to use in a search to get better results

Myocardial infarction - found right away

Typed in heart attacks and then changed to Subject and searched. Tried as just heart attack and then thought it was cool that it had that LoC name. Liked the definition in the wikidata box for this.

Typed in heart attack and found myocardial infarction

Myocardial infarction. Found it in auto-suggest



Search for resources written by the civil engineer Max Weber.How do they input this to search, using what terms? Do they see the description next to the name in the auto-suggest?

Typed Max Weber - saw it in the list but was wondering if you could type civil engineer next to the name - used the occupation area in the auto-suggest

Typed it in and used the occupation - very handy to have in the dropdown to be able to differentiate.

Searched for weber, max - used the descriptive text to determine which one to click on.

Typed Max weber and was able to use the text next to the name to pick the correct one.

Searched for weber, max and clicked on the item using the auto-suggest - saw that the civil engineer was with the name and chose it based on that.



Select the author from the previous task.  On the resulting page, please provide us any feedback on what you expected to see and what you are seeing. What more can you describe about Max Weber?  Is this information helpful?

Likes the box with the little bit of information at the top - clear what is there and that you can click on the works by and about - digital collections are cool but confusing, not sure if they're included in the catalog or somewhere else

On the right hand section seeing what is used to be seen but the additional box on the left with the wikidata is extremely helpful. Occupation is most helpful to correlate to search.

Likes the photo and is a nice feature - would hope if looking up something specific by this guy would already know that kind of information about him but not necessarily - When using the auto-suggest, thought it was weird the occupation was listed under Author but not with the same name in Subject.

Says the occupation of this person and confirms that she clicked on the right place. Was a little confused to see the first results but then thought Max was brother/son of that person.

Asked if it was the right page because saw Carl Maria von Weber on one of the items but then saw Max under that.


 

Would something else be more helpful for you?

Previously mentioned the biography for the coen brothers task - in the digital collections, expected the link of the title to go to the item in the digital collection - thinks that a lot of the patrons want catalog to work like google and would search by max weber civil engineer and expect it to work, thinks it would be useful for users

Not off the top of the head. Handy that the dropdown menu had the author and subject, didn't see it the first time but then after doing it it's clear and found it useful - "thought it was great to not have to go over to the box and select the type of search"

Likes that it doesn't start auto-suggesting right away and lets type in more so there's less to wade through and get what they were looking for.

Thinks it is all there and is generally really bad about reading what's in title bars - above the wikidata box, wasn't noticing the title bar with his name at first. Thought it was pretty cool and sometimes works with students with things with these suggestion dropdowns and students tend to type really fast and hit enter without seeing the dropdown suggestions. If you type one out and hit enter, what option from the auto-suggest, if any, is selected by default?

Box in the upper left - helps to confirm that it's a page for the author she searched for and reminds of his occupation that was searched for previously. Other than that, doesn't really think that it does much else.  Works by and Works about: understood that it was books he had written and books written about him respectively.  Would any other information in the info panel be more helpful? - Might find the photo mildly distracting but wouldn't think of anything else to add.

  • No labels

All content on the LYRASIS Wiki is licensed under the CC BY (Attribution) license, unless otherwise noted.