Usability test proposal for graduate studies project.

[btelles]

Hello all,

This post is a proposal for a formal usability study on Joomla! as part of a graduate studies Design of Experiments project.

The Joomla! organization would get the following benefits from the study (a study which would be conducted free of charge and for educational purposes only):

What Joomla! gets:

• A usability test conducted with tightly controlled research methods,
• A full statistical analysis (provided with aid from the professor to ensure validity of the information),
• The facilities and video equipment to run a usability study on between 10  and 20 participants (well, you don’t actually get these, but they will be used for testing your product)
• The participant pool with which to run the study (also not for sale)

In order for Joomla! to get the most out of this, though, we would need two things from your team:

What we ask of you:

• A detailed list of Joomla!’s 10 most commonly completed tasks for basic Joomla! usage as a content management system. We would then use these tasks to gather users' perceptions of the 5 or 6 most needed usability changes,
• Then, in order to conduct a fractional or full factorial experiment to determine the most useful usability modifications, we would need someone (or a group of people) to help us make these modifications within the next few weeks.

Naturally, Professor Ramsier (http://stat.fsu.edu/people/faculty.php?id=13) and I will be glad to sign a non-disclosure agreement with Joomla! if your organization thinks one would be necessary.

My qualifications include my status as a graduate student who has designed and implemented 3 usability studies and several heuristic analyses on 4 web-based products at Florida State University's College of Information http://ci.fsu.edu/.

If anyone is interested in helping out with this study, please let us know as soon as possible by replying to this post.

Best regards,

Bernie Telles

P.S. I sincerely hope this post doesn't sound too dry or unfriendly...I am a really nice guy, but wanted to show a certain level of professionalism on the proposal....'guess this post-script kind of diminishes that attempt hm?...well, either way, hope to hear from you soon!

[eyezberg]

Bernie,
this is imho a great offer, I hope someone sees the potential in tis and gets in touch with you; I was hoping something like this would be done someday, but as these test are really quite expensive, I was not really seing this happen -until now!
It is -again imho- not so easy to define a global, valid for all usages, "10 most common tasks" list, as Joomla can be used for a large variety of sites, anything from a magazine style, content focused news site, to something more community oriented where you'd be busier managing new users, reviewing forum posts and comments, to e-commerce using extensions like Virtuemart to sell products -update and maintain product specs & stocks, review orders etc..
It might be easier if you would for example go for: the 10 most common tasks when setting up a new site (in logical order), such as install, add requiered extensions, organise content into sections & categories..
Or: the 10 most common tasks on a day-to-day basis to manage content items (create/modify articles, upload and usage of media, settings on a per-item basis..) -as in the end, Joomla! is a content managing system..
or something else more focused on a specific area.
Personally, I'd like to say: thanks for the offer!

PS: forgot to ask: why did you choose Joomla! ?

[btelles]

Hi Joe,
Thank you for your encouraging words and for recommending that we narrow the scope of the proposal (you responded much faster than I thought possible ). I agree with your idea, and have modified the original post by adding the following:

10 most commonly completed tasks for basic Joomla! usage as a content management system.

Thanks again for the quick response!
Bernie

[louis.landry]

We have indeed been looking into usability studies and I have been talking personally with a few friends of mine with similar areas of expertise.

I'll see if I can put you in contact with the right people and see what comes of it.  Thanks for your interest and willingness to help.  There is often a lot of people telling us what is wrong in their minds -- which is good feedback -- but rarely do we get a chance to see a well thought out and analyzed data set on specific aspects of our UI.

Louis

[btelles]

Thank you Louis,

I'll look forward to getting in touch with whomever is willing to help.

Regards,
Bernie

[btelles]

Hello everyone,

Even if you just write down a quick list of 3 or 4 tasks that you or your users do most often, and the current (Joomla! version 1.5) fastest way to conduct these tasks, we would greatly appreciate your help!

Hope to hear from you soon!

Bernie

[Jenny]

My qualifications include my status as a graduate student who has designed and implemented 3 usability studies and several heuristic analyses on 4 web-based products at Florida State University's College of Information http://ci.fsu.edu/.

Bernie it is great to see someone so enthusiastic to help! I looked at the link to the College of Information site you gave but couldn't find any links to the studies you have already completed.  Are they posted there or is there another link where we can see the results? I have only seen a few usability studies scattered throughout the web (there are far to few in my opinion) and would love to see the results you have already compiled, and some of the details as to how these studies are done.

[eyezberg]

MMM,
may I recommend these:
http://www.amazon.com/Web-ReDesign-2-0- ... F8&s=books
http://www.amazon.com/About-Face-2-0-Es ... F8&s=books
http://www.amazon.com/Paper-Prototyping ... F8&s=books
http://www.amazon.com/Dont-Make-Me-Thin ... F8&s=books
http://www.amazon.com/Design-Everyday-T ... F8&s=books
http://www.amazon.com/Inmates-Are-Runni ... F8&s=books
http://www.amazon.com/Information-Archi ... F8&s=books

plus a few others

[eyezberg]

Hello everyone,

Even if you just write down a quick list of 3 or 4 tasks that you or your users do most often, and the current (Joomla! version 1.5) fastest way to conduct these tasks, we would greatly appreciate your help!

Hope to hear from you soon!

Bernie

1. create a new folder in Media Mnager & upload images and or other media such as swf files
2. create a new article, fill in parameters first such as "frontpage", "published" (not untill it's final -draft stage missing here), author, dates, title etc..
3. write main article text with media inserted in context.
generally I save to published / "special" state so I'm the only one who can see it from frontend when logged in as superadmin, then check if all is ok. if yes, change state to public, if no, edit as needed & recheck.
4. when article content is 100% ok, take care of meta (and tags -3rd party extension)

other common tasks (3rd party extensions):
delete or approve and or edit comments on articles
delete users who never confirmed their signup

that's for a simple site

[Jenny]

Eyez, I am not looking for books on the subject.  I just wanted to see some of the studies that come up with the information behind the books.  Perhaps I didn't explain it well.  There is lots of information on usability all over the web, but the studies that the information is based on seems to be rather lacking.  All those books have a lot of good information re design etc in them though.

[btelles]

Hi there,
Thanks Eyezberg! We will probably end up using all of your ideas plus another 6 tasks that are conducted on a regular basis.

By the way, last night I posted a response to your question (MMM) but it hasn't been posted yet. In the post, I attached an ISO-standard report that I created from one of the studies. Are PDF's not allowed, or was the post considered self-promotion (which I believe is considered against the rules of the forum)?
If you want me to report I'll be glad to do so.

[Jenny]

I don't think those sorts of attachments are allowed on the forums.

If you have a link to your site or a site with the information that would be great.. or you can email me.. I can PM you my email address if you like. 

[btelles]

Okay, in that case I'll repost last night's comment with a link to the study on the internet:

Hi there MMMedia (Jennifer...I think),

I would be glad to brag  . Here goes:

Study/Heuristic Evaluation 1:
The first study was the main project of an internship with the Florida Network of Youth and Family Services. The study was geared at improving a statewide database for that tracked youth prevention services for 67 youth agencies in Florida. This is the only study for which I completed a full ISO/IEC 25062 report. I've attached the report in PDF format for your reference. I'd love to take the time to explain all of the details, but the attachment pretty much says it all, and it's getting a bit late for me. (http://btwebdesigns.com/FNYFS%20Usability%20Study.pdf)

Study/Heuristic Evaluation 2:
The second study and heuristic analysis was on the College of Info Helpdesk website http://ci.fsu.edu/go/userservices. I was in charge of recreating the Helpdesk site throughout the past 2.5 semesters, at a pace of 1-5 hours per week (much slower than the my intended pace for the Joomla! project which needs to be done by December 15th). Although the site is highly usable, the design (egg-shaped semi-buttons instead of more professional looking buttons) were not my preferred choice. In this study each of 7 users (including a mix of faculty, students and staff) were asked to complete 5 of the most common tasks we envisioned our site being used for. Tasks were timed and users were asked to verbalize their thoughts as they completed the tasks.

Study/Heuristic Evaluation 3:
A short investigation into possible content management systems for the College of Information with one focus group. Five individuals were asked to look into login systems, news posting and content organization schemes for 5 CMS solutions. Each user investigated one CMS for about 15 minutes. Then they were asked to discuss their results openly in a group for another 45 minutes.

Heuristic Evaluation 4:
Analyzed the website for the Test Object Mode" (TOM) database. TOM will house usabilitly test findings with granular data objects such as country of origin, industry, use cases and objects that should help international users find relevant usability test data for cultures other than their own. I am currently creating a website-literature review which justifies design decisions for the Test Object Model database.

I'll be glad to elaborate on Dr. Ramsier's and my vision of the Joomla! study, but it's getting a bit late and this post is already unusably long...hehe. How about tomorrow?

To Jennifer:
On a side note, Jennifer, since you're a moderator, should I continue detailed discussions about the study on this forum, or should I start different threads for different portions of the study, or maybe a mailing list of some sort? Also, I haven't heard from people in the design/accessibility team yet on this. Do I need their approval before continuing the study?

To Louis:
Hi Louis, do you think the contacts you mentioned earlier would still be interested in helping out with the study?

Best regards to all!
Bernie

[Jenny]

Bernie,

Continuing on in this thread so it stays consistently in one place would be great.  Thanks for the information, sorry to put you under pressure so late.  I look forward to reading the details in the PDF linked to.

Things may seem a little slow on the response time from the U&A team at this time as they are all bug squishing and getting the roll out on 1.0.12.  I am sure the team will get back to you as soon as they can with more details. 

It looks like great work you have done.  I am eager to dive into the details.

[Jinx]

On it, send Bernie a PM, we won't let this oppurtunity slip  More news to follow.

[btelles]

Hello all,

First, thank you all for your interest in the study! Special thanks to Eyezberg for starting the list of tasks.

Here is a very high-altitude view of the proposed study:

• Define 10 critical tasks (with four or five scenarios) that are representative of a successful product.(mentioned later in this post)
• Test 4 users on these ten tasks and find the 5 tasks that are least usable
• Implement user-recommended improvements to Joomla! on each of the five tasks
• Retest the five least usable tasks and measure their usability on many users (number to be determined) to get a more representative sample of the effects of the changes.

We will operationalize (define) usability via two measurements for each task and one measurement for the system as a whole for every user. They are the following:

• Efficiency: Time required to complete each task
• Effectiveness: Whether Joomla! supports the users' conceptual schema for each task or whether Joomla!'s organizational method is different from the users'. A quantitative value will be assigned to this measurement based on the number of instances users use phrases analogous to "I would expect the system to do this instead of that."
• Satisfaction : The users' affective reaction to using the system as measured by the System Usability Scale (SUS)

We have developed the following scenarios based on context data from the Joomla! 1.5 Beta demo site. Before we go any further with the study, it is very important that the developers, users, marketing dept. (if there is one) and product managers verify that the following scenarios are representative of the critical tasks they expect users to accomplish with Joomla!. If this analysis is inaccurate then the usability study will not map on to your organizational goals and the information you gain will probably help you improve your product in ways that are divergent with your definition of success. For example, we realize that the eCommerce scenario may be outside the scope of the Joomla! organization and need your feedback on whether this scenario should be included.

We would like to begin screening for the most difficult tasks by the end of the week, so if we can work out the quirks with these scenarios by Thursday night, we should be on schedule to finish the entire study by the deadline date.




    Type of Site      Least tech-savvy AdministratorsEnd-User
On-line CommunityA church official would like to add an on-line component to his/her local church. She would like to send mass emails to church members and place an on-line reservation system for church facilities to be managed by several church officialsUsers should be able to submit news items for approval by church officials. They should also be allowed to submit reservation requests and view church facility availability on-line.
Media PortalAn independent band manager would like to reach out to his/her band's audience by building a website with sound clips, interviews, news article references and show dates and locations.End-users would like to quickly view the local band’s show dates and locations as well as listen to their newest music and get updates on most recent news.
BlogA graphic design student has been assigned to create her own Blog. Her assignment requires her to create a blog with a new design and to post few articles as quickly as possible. (We assume she is familiar with CSS)The software should allow her classmates and teacher to post comments to her blog intuitively. The software should also allow users to subscribe to the blog's RSS feed via a standard button.
eCommerceA flower shop owner would like to expand his business to the internet by allowing users to order flowers via his website. The software should allow him to set appointments for flower deliveries, accept payments on-line, and to categorize his flowers by color, and country of origin. The software should allow him to add various payment methods as needed.The software should allow shoppers to sort and search for flowers with any of the listed attributes and purchase a flower via a credit card or PayPal easily.


As always, with best regards,
Bernie

[Jenny]

I don't think the ecommerce scenario should be included.  I think that as an ecommerce, while attainable using Joomla! and various ecommerce solutions, I think what you are proposing:

A flower shop owner would like to expand his business to the internet by allowing users to order flowers via his website. The software should allow him to set appointments for flower deliveries, accept payments on-line, and to categorize his flowers by color, and country of origin. The software should allow him to add various payment methods as needed.

is so specific as to be an overly-complicated task whether it was being tested on Joomla! or any other software out there.  I would expect the task of setting up a website with that level of complexity, completed by a professional with excellent skills, considerable time to complete no matter what software or development platform used.