STEM Teacher Leaders Network
User Feedback from In-Person Testing
Think-Out-Loud, Task Performance & General Review
03.29.2019
Our goal for this round of result of this testing was to assess the overall usefulness, appeal and engagement quality of the site. We asked testers to consider visual appeal and appropriateness, organizational clarity and the value of the content.
Testing was conducted in person, using the “think-out-loud” protocol. Testers were asked to explore and use the site, then were asked to join the site, find members, explore groups and submit comments. They were also asked to explore the other sections of the site and comment on value, appeal, facility of use and engagement. The form used by testers is attached to this report.
Five of the six test participants were STEM subject teachers ranging from grades 8-12. One tester was an information systems professional included as an expert evaluator. Sessions were 45 minutes long, in the testers’ schools or classrooms, on the computers they use daily. Guidance was not offered in using the site, and was provided only in the (infrequent) case of confusion.
We had three general questions to answer, as well as a number of more specific items to assess and note if problems arose.
Our three general questions were:
- Do the users feel that this would be a valuable resource?
The answer is “yes”.
- Do the users feel that they can find and use resources and information easily? The answer is “yes”.
- Do the users feel that they will interact and contribute to the resources and discussions on the site?
The answer is “very likely”.
From this, we can conclude that the site has value to its users; users find what they’re looking for; users will be likely to interact and contribute–once the site has evidence of wider participation and a larger wealth of resources.
The most mentioned requests were “more professional development resources” and “add student opportunities for teachers to pass along on the Opportunity Board.” These are both meaningful in that they convey the enthusiasm and involvement unanimously shown by users. Even with the limited content in this testing version of the site, users paused frequently to read, comment, scroll; every user watched the video they chose all the way through—this is unusual, especially in testing with busy educators.
Users selected were not current Teacher Leaders and had little or no familiarity with the meaning of associated groups and titles. In further releases of the site, some care should be taken to explain these organizations, with links to more information and how to get involved. All testers were interested in learning more.
Overall, the site was met with enthusiasm, interest and even relief in seeing a well-focused resource with group/topic specificity for focused exchange, and an ever-growing archive of assets and tools for professional development and in some cases classroom use.
User Feedback
Home page
- The colors and design immediately spoke “education” to testers.
- Some felt the image was too busy, while others wondered who this group was
- An equal number of testers were drawn to the squares below the photo and to the navigation above the image.
- The majority of users went to Resources first, feeling that it would have the most useful content.
- Several testers wished for more timely “News” on the homepage and weren’t sure what “Featured” meant.
Monthly Theme
- Surprisingly few users were drawn to this, expressing a feeling that it was “probably too general” for their use.
- The word “Theme” confused users
- Here, as everywhere on the site, users looked for names they recognized, and topics directly related to their subject.
- While users liked the idea once they had seen the content, they were more interested in material specific to them.
Resources
- Users spent most of their time here, easily using the filters and page views.
- Slightly more preferred the list view over the grid view.
- All expressed their desire to see this area grow and stay up to date
- 50% of users were unclear initially about the format of material before identifying the icons and colors as labels.
- Three users asked why video was both here and in the Video Showcase, and wondered if they were the same videos in both sections, or if each had a separate selection of videos.
- All expressed the greatest interest and excitement in the potential for helpful and useful resources to be found here.
Members Network
- This area, while appealing, took time for users to understand. There was some feeling that they didn’t belong if they weren’t in the list of affiliations.
- All testers joined easily. Two testers went back to their profiles to add information and make adjustments, again, without any hesitation or confusion.
- Three testers requested avatars rather than requiring profile photos, even saying they would leave before posting photos of themselves.
Video Showcase
- Tellingly, users needed continuous urging to speak out loud while exploring the Video Showcase. They immediately began looking for a video to play, and once engaged, watched all the way through.
- There was excitement for subject learning and teaching best practices, as well as classroom use.
- Three testers suggested that when there is more content it will be necessary to have filtered search as on other pages.
- All testers wished for more information on mouse-hover over video boxes.
Opportunity Board
- This was the second-most exciting area for all testers. Hungry for professional development, they were very positive about this feature.
- Four testers were outspoken in hoping there would be opportunities for students as well as teachers, so they could pass them along.
- Expected content here, as described by the combined testers was jobs, internships, funded programs, scholarships and internships for students.
- All said they would come here frequently to look for opportunities and would post opportunities here when they had them to share.
Blogs
- Testers were not unanimously positive about this feature. Two testers said they would not be very unlikely to read.
- Three testers said they would scan the topics but would need “something to really grab me” to read posts.
- One tester was enthusiastic about more in-depth posts if they were relevant to her subject and age group, but like other testers, she was emphatic that the content be directly useful to her.
- Because of the limited content on the blog in this release, it should be reviewed in more detail in the next round of testing.
Events
- All testers were excited to see events listed on the homepage.
- Testers expected Events to be on the site, summarily regarding the listings with comments like “good, this is here.”
- Testers liked the design of the listings, finding them clear and quickly scannable.
- It was mentioned that the location and day of the week would add to the usefulness of listing.
Groups
- Testers were excited at the idea of having very specific peer/topic groups for discussions and advice.
- Two testers, who joined and wanted to participate in groups, were unclear about how content would be published and if it would be moderated.
- Most testers noted that even with the limited sample content there would clearly be valuable exchanges here.
General feedback
- This is an audience group with unusually high detail orientation:
- The map on the Members page fascinated many testers, but they were not sure why when they clicked “Join” (which is also confusingly called “Register” elsewhere in the site) they landed on a page with a prominent map rather than a sign-up form at the top.
- They unanimously loved the “math-problem” Captcha widget.
- Most testers found inconsistencies in terminology (see “Join” above), and while in beta they are forgiving, such inconsistencies reduce legitimacy.
- There was a general sense among testers that this was a site for a group they weren’t part of.
- All agreed that they would use the final site, and most requested inclusion in the next round of testing
Summary
Users were generally excited to see a resource with a variety of kinds of material they could genuinely use, as well as interaction opportunities with the potential to be very specific to their peers and subjects.
There was confusion about who was publishing the site and for whom it was being produced. More than a fear of inappropriateness though, teachers seemed put-off by a “club” they weren’t members of offering this resource. While not a serious issue, it did arise with every tester.
Even with limited content in place for beta, testers were enthusiastic about being able to access and to contribute to an ever-growing body of helpful, specific material. The most complex mechanisms were not at all stumbled over; filtered searches, registration, posting comments—all were easily accomplished by all testers. While there were minor notes, this is evidence of their inclination to become involved and to stay engaged.
Recommendations
Home Page:
- Use a more broadly appealing, simpler hero image
- Add clearly timely “News” section
- Reconsider “Featured” as a section name
- Offer explanation of who is behind the site and who it’s for
Monthly Theme:
- Clarify what this is and what its purpose is
- Make timely content obvious
- Guest curators?
Resources:
- Allow for many more in the future
- Clarify what format materials are in (PDF, video, etc.)
- Explain whether same videos appear both here and in resources
- Offer more detailed information on hover (and consider more info in square for mobile)
Members Network:
- Create a separate page for signing up
- Explain who, what, why of membership in a prominent place on the page
- Do not require photos
Video Showcase:
- More information on each video
- Plan for many more videos
- If users are encouraged to submit videos, make a clear call to action
Opportunity Board:
- Include student opportunities
- If users are encouraged to submit opportunities, make a clear call to action
Blogs:
- Consider very focused topics
- Strategize ways to make relevant and useful
- Invite guest posts in various place throughout site
Events:
- Include place and day of week on listings
Groups:
- Clearly explain procedure for contributing
- Make “Recent Group Activity” column on right less prominent and more clearly organized
- More clearly separate “Webinar” from “Recent Group Activity
Testers
Private
Newton South High School, Newton, MA
Subject: Math
Grade: 10-12
Platform: MacBook Air, Sierra 10.12.6
Browser: Chrome 72.0.3626.119
Wifi: Strong
Private
Newton South High School, Newton, MA
Subject: Math and Computer Science
Grade: 9-12
Platform: MacBook Air, Sierra 10.12.6
Browser: Chrome 72.0.3626.119
Wifi: Strong
Private
Boston Collegiate Charter School, Boston, MA
Subject: Math
Grade: 5
Platform: Lenovo, Windows 7
Browser: Chrome 72.0.3626.119
Wifi: Medium
Private
Boston Collegiate Charter School, Boston, MA
Subject: Environmental Science
Grade: 12
Platform: ThinkPad, Windows 7
Browser: Chrome 72.0.3626.119
Wifi: Medium
Private
Boston Collegiate Charter School, Boston, MA
Subject: Physics
Grade: 9-10
Platform: MacBook Pro, 10.14.3
Browser: Chrome 72.0.3626.119
Wifi: Medium
Private
Nexxus Group, Beverly, MA
Subject: Information Science
Grade: Professional
Platform: Lenovo, Windows 7
Browser: Chrome 72.0.3626.119
Wifi: Strong