Nokia E71 3g Media Player, Peter-Paul Koch, writing on the QuirksBlog, shares the results of a usability test comparing 10 different phones. What's most interesting is that the user in question might be considered the 'average' user rather than a tech-influencer (as is typical in online media). Consequently many of the observations offer insight into what matters in the real world compared to rarefied atmosphere of the 'media bubble'. Read on for more.
The phones in the test include the Nokia N900, Nokia N97, iPhone 3G, Sony Ericsson W960i, HTC Touch Pro, Blackberry 9500 and HTC Magic capturing a broad spectrum of current touch screen phones.
Ultimately the N900 comes out as the winner, with the iPhone second and N97 third. Clearly the results will vary depend on what task is being tested (in this case it is accessing a mobile banking website) and the user being tested, but the relative results are less important that the observations themselves.
Writing on All About Symbian and testing a variety of high end phones it is easy to lose sight
Some the observations that run against the grain for current trends include:
* Stylus usage (and included stylus) seen as a good thing.
* N97 (widely perceived as a 'failed' phone rated highly).
* Hardware controls for zooming (and in general) working better than software controls.
* Problems of too many actions hanging of the finger touch input method (select, scroll, zoom).
* Windows Mobile was ranked above Android.
These should be considered in the context of the following:
"More in general, B [user being tested]. is a hardware man. He consistently put the phones with better hardware controls higher on his list, with the iPhone and the E71 as the only exceptions. He has also decided to buy a stylus no matter which phone he’ll eventually get."
B. is an old friend of mine who owns an old Nokia. And when I say old, I mean really old. It was released somewhere in 2000 or so (the Nokia, not the friendship). It’s not a smartphone, to put it mildly, and B. does not use the mobile Web.
Yet.
Pretty soon, however, B. is going to spend a few months in the outlying parts of Indonesia, and during that time he has to be able to access his business bank account. He was wondering if a modern mobile phone would fit this use case, and, if so, which one.
When he told me all that I whipped out my iPhone. “Something like this, you mean?” He was suitably impressed, and when I told him I regularly have six to twelve phones lying around on my desk he practically begged for an opportunity to come by and try them all in order decide what kind of phone he wants.
That was of course fine by me. User testing is never to be despised, and since B. is not technical and has no experience with touchscreens to speak of, he is the perfect test subject.
Last week we held our session, and this entry is the report.
Tested phones: Nokia N97, Samsung M1, HTC Touch Pro (Windows Mobile), SonyEricsson W960i, Nokia E71, BlackBerry 9500, HTC Pioneer (Android), LG M900, Nokia N900, iPhone.
Set-up
B. is not technically inclined, to put it mildly. His knowledge of the Web is best summarised by this short discussion that ensued when I gave him the HTC Windows Mobile phone:
Me
[gives Windows Mobile phone with home screen activated]
B.
Ah, Internet Explorer. [Clicks on it while ignoring the Opera logo right next to it]
Me
You know what that is?
B.
Sure, it’s Microsoft’s search thingy, right?
Me
Errr ... right ... [pours stiff drink]
On the other hand, B. has an excellent idea of what he needs. He needs access to his bank account and he has to be able to take common actions.
B. knows his workflow: go to Google, search for the entry page to his banking site, go there, log in, and do stuff. I warned him he’d better skip the first step in Indonesia. The network is bound to be slow, and skipping one page load is definitely a good idea. I also revealed the existence of bookmarks.
B.’s target is Mijn ING, which is definitely not the most advanced Dutch banking site. Most of the problems we encountered were in the later, password-protected pages, so you can’t view them in your browser unless you also bank with ING. (I do, so I might take a shot at researching a few problems.)
We tested ten phones, and in all cases I made sure the phone had a connection and was on the home screen. After that I handed it over.
The original plan was not to offer B. any further help, but pretty soon it turned out I had to teach him how to zoom. Not only was he completely unacquainted with any kind of zooming, but the eight phones that have zooming capability use five different mechanisms (double-tap, pinch, Android, BlackBerry, and Maemo), and leaving B. to discover them by himself would have taken far too much time.
I also explained other interface elements I thought might help him. I’m not sure if this is good methodology, but it cut down testing time considerably.
The phones
As to the actual phones we tested, that was mostly a matter of coincidence. I’d taken a lot of touchscreens with me from Düsseldorf because I’m doing some complicated touch research for Vodafone. In addition, all four phones I received as a gift from interested parties are touchscreens, too.
I’d have loved to include the Palm Pre in this test, but at the time of writing I have not yet figured out how to skip the sign-up screen on my locked British one.
I also left out the Samsung F700 because I exclusively use it to torture UX experts. That brought the total number of phones down to ten.
Nine out of these ten were touchscreens (with the Nokia E71 as the exception). In retrospect it would have been better if we’d had one or two more non-touchscreens to test.
I decided on the testing order in advance. On the one hand I wanted to bunch up the phones I thought good toward the end of the session, because I wanted to see if they’re really that much better than all the others in the opinion of an outsider without any experience.
On the other hand I didn’t want to start with the very worst phones because that would only depress B. So I started with the most medium phone I could think of, the N97.
As a practical matter, I had to alternate wifi-capable phones with SIM-only phones in order to have time to insert the SIM card and start up the next SIM-only phone while B. was working with a wifi-capable phone. The BlackBerry, especially, needs a lot of startup time.
Finally, I decided to use the default browser on every phone. I do not believe that many non-technical users will download another browser when their phone contains one that appears to be working. Besides, I’d have to explain the “other browsers” concept to B., and I decided that that would take too much time.
The testing
I handed B. the phones in the order they’re treated below and recorded his impressions.
Nokia N97
The phones in the test include the Nokia N900, Nokia N97, iPhone 3G, Sony Ericsson W960i, HTC Touch Pro, Blackberry 9500 and HTC Magic capturing a broad spectrum of current touch screen phones.
Ultimately the N900 comes out as the winner, with the iPhone second and N97 third. Clearly the results will vary depend on what task is being tested (in this case it is accessing a mobile banking website) and the user being tested, but the relative results are less important that the observations themselves.
Writing on All About Symbian and testing a variety of high end phones it is easy to lose sight
Some the observations that run against the grain for current trends include:
* Stylus usage (and included stylus) seen as a good thing.
* N97 (widely perceived as a 'failed' phone rated highly).
* Hardware controls for zooming (and in general) working better than software controls.
* Problems of too many actions hanging of the finger touch input method (select, scroll, zoom).
* Windows Mobile was ranked above Android.
These should be considered in the context of the following:
"More in general, B [user being tested]. is a hardware man. He consistently put the phones with better hardware controls higher on his list, with the iPhone and the E71 as the only exceptions. He has also decided to buy a stylus no matter which phone he’ll eventually get."
B. is an old friend of mine who owns an old Nokia. And when I say old, I mean really old. It was released somewhere in 2000 or so (the Nokia, not the friendship). It’s not a smartphone, to put it mildly, and B. does not use the mobile Web.
Yet.
Pretty soon, however, B. is going to spend a few months in the outlying parts of Indonesia, and during that time he has to be able to access his business bank account. He was wondering if a modern mobile phone would fit this use case, and, if so, which one.
When he told me all that I whipped out my iPhone. “Something like this, you mean?” He was suitably impressed, and when I told him I regularly have six to twelve phones lying around on my desk he practically begged for an opportunity to come by and try them all in order decide what kind of phone he wants.
That was of course fine by me. User testing is never to be despised, and since B. is not technical and has no experience with touchscreens to speak of, he is the perfect test subject.
Last week we held our session, and this entry is the report.
Tested phones: Nokia N97, Samsung M1, HTC Touch Pro (Windows Mobile), SonyEricsson W960i, Nokia E71, BlackBerry 9500, HTC Pioneer (Android), LG M900, Nokia N900, iPhone.
Set-up
B. is not technically inclined, to put it mildly. His knowledge of the Web is best summarised by this short discussion that ensued when I gave him the HTC Windows Mobile phone:
Me
[gives Windows Mobile phone with home screen activated]
B.
Ah, Internet Explorer. [Clicks on it while ignoring the Opera logo right next to it]
Me
You know what that is?
B.
Sure, it’s Microsoft’s search thingy, right?
Me
Errr ... right ... [pours stiff drink]
On the other hand, B. has an excellent idea of what he needs. He needs access to his bank account and he has to be able to take common actions.
B. knows his workflow: go to Google, search for the entry page to his banking site, go there, log in, and do stuff. I warned him he’d better skip the first step in Indonesia. The network is bound to be slow, and skipping one page load is definitely a good idea. I also revealed the existence of bookmarks.
B.’s target is Mijn ING, which is definitely not the most advanced Dutch banking site. Most of the problems we encountered were in the later, password-protected pages, so you can’t view them in your browser unless you also bank with ING. (I do, so I might take a shot at researching a few problems.)
We tested ten phones, and in all cases I made sure the phone had a connection and was on the home screen. After that I handed it over.
The original plan was not to offer B. any further help, but pretty soon it turned out I had to teach him how to zoom. Not only was he completely unacquainted with any kind of zooming, but the eight phones that have zooming capability use five different mechanisms (double-tap, pinch, Android, BlackBerry, and Maemo), and leaving B. to discover them by himself would have taken far too much time.
I also explained other interface elements I thought might help him. I’m not sure if this is good methodology, but it cut down testing time considerably.
The phones
As to the actual phones we tested, that was mostly a matter of coincidence. I’d taken a lot of touchscreens with me from Düsseldorf because I’m doing some complicated touch research for Vodafone. In addition, all four phones I received as a gift from interested parties are touchscreens, too.
I’d have loved to include the Palm Pre in this test, but at the time of writing I have not yet figured out how to skip the sign-up screen on my locked British one.
I also left out the Samsung F700 because I exclusively use it to torture UX experts. That brought the total number of phones down to ten.
Nine out of these ten were touchscreens (with the Nokia E71 as the exception). In retrospect it would have been better if we’d had one or two more non-touchscreens to test.
I decided on the testing order in advance. On the one hand I wanted to bunch up the phones I thought good toward the end of the session, because I wanted to see if they’re really that much better than all the others in the opinion of an outsider without any experience.
On the other hand I didn’t want to start with the very worst phones because that would only depress B. So I started with the most medium phone I could think of, the N97.
As a practical matter, I had to alternate wifi-capable phones with SIM-only phones in order to have time to insert the SIM card and start up the next SIM-only phone while B. was working with a wifi-capable phone. The BlackBerry, especially, needs a lot of startup time.
Finally, I decided to use the default browser on every phone. I do not believe that many non-technical users will download another browser when their phone contains one that appears to be working. Besides, I’d have to explain the “other browsers” concept to B., and I decided that that would take too much time.
The testing
I handed B. the phones in the order they’re treated below and recorded his impressions.
Nokia N97
User testing observations from QuirksBlog
Reviewed by Anna Bulgaria
on
1:10:00 AM
Rating:
No comments: