I’ve realised why it is that people on all sides of the development process hate testing. Testing should be done by those who actually USE the system. Now, users get rather annoyed if you give them things to test that aren’t really quite close to correct. They find problems within seconds and you have to go away and fix things, leaving them frustrated at having made time for testing and annoyed that you hadn’t got it right. So users hate testing early on.
The result is that testing is carried out by developers (of one type or another) until quite late in the process … who are largely testing what they built. They won’t realise that the doohickey shouldn’t allow selection of the flobbit, that the order of screen elements is both stupid and clumsy, or that the design makes the process far too slow. Only real users will realise this. So developers hate testing, because they essentially regard it as useless. They know it works how they told it to, so why should they run tests??
The theory, then, is that tests should be specified at the outset, by the users. If they write the tests, then the developers can just follow them and identify the same problem that the users will. Right? But how do you convince users to write tests that a) someone with limited domain knowledge can follow and b) can expose problems that only a real user would find? The answer is that you really really can’t. In my own experience, often the users are also not anywhere near technically/logically minded enough to write tests that cover the various scenarios, never mind expose deep dark flaws in the operation of the software!
So what ends up happening is that the developers develop & test, then eventually pass the “working” product to the users, who quickly find flaws. Then the developers have to go back into their development & testing cycle. This repeats until developers hate testing, users hate testing and developers & users also both hate each other.
There are some things that can mitigate this hatred. Dynamic languages can make collaboration between users & developers more of a reality. If the time between the user pointing out that something is wrong and the developer fixing it and offering it up for retesting is measured in minutes rather than days, this can ease the burden greatly. Good requirements gathering is invaluable in making sure the right thing is being built in the first place. User- or task-centred design can make a massive difference.
But at the end of the day, everyone still hates testing.
So I was thinking, regression tests are a brilliant idea when test can be automated. Is there any way of automating user testing? It occurs to me that you can easily record macros of what the user is doing — so presumably it should be possible to record a macro of a user testing and then be able to rerun that at a later date. Does anyone know of a way of doing this already out there?
Comments (5) Permalink
August 22nd, 2005 at 9:09 PM
Perl has HTTP::Recorder, which you set up as a proxy, then browse the site through it and it outputs scripts that replay the HTTP traffic. you can then modify those scripts into a regression test of some sort. someone’s probably written a python version by now, but i haven’t found it yet. see: http://www.perl.com/pub/a/2004/06/04/recorder.html
i am also a really big fan of Selenium, which is a browser based testing framework in javascript. it doesn’t record the interaction, but the FIT style unit tests in a simple html table might be simple enough for advanced users to write their own tests. also really useful for testing apps which rely on javascript for important functionality. at the very least, selenium makes it pretty easy to take a use-case and turn it into a test. see: http://selenium.thoughtworks.com/index.html
overall, the best approach i’ve found is just for developers to become users themselves. it takes a certain kind of talent to be able to put yourself in the position of the user and really approach the application how they might. obviously this is easier on certain kinds of applications than others (eg. the developers of most weblog engines seem to be bloggers themselves).
August 23rd, 2005 at 12:43 AM
So, call me crazy…but I love to run user testing. In my world, I often form a Joint
Application Development Team that is comprised of developers and the users we are developing
for. I always have a spot in the development process where the users test the system
while it is in beta. I’m quite prone at the time to give out prizes for finding bugs.
So the process turns into an official work assignment with rewards. Most importantly, while
the system is still in beta, real users are getting their fingers in the system, and giving
critical feedback before the project is marked “done”.
I just don’t think you can ever replace that experience with machines. Or sure, you may
find some bugs with automated testing…but I’m talking about deeper issues, like does
this system actually work for real humans!
August 23rd, 2005 at 4:30 PM
Anders — thanks for the links! I agree that the developers becoming users themselves is desirable, but I think that there are some aspects of user-experience that they will not be able to mimic/reproduce. For instance, I’ve been involved in development of transactional systems quite often where to REALLY understand, you need to be able to process 100 transactions per hour. There’s also nothing worse than a developer who thinks they understand the user but doesn’t! 😉
Goodwitch — I completely agree that users getting into a system at beta stage is a great idea and it can be really useful and enjoyable. The phase I was more talking about was before that — where if the developers don’t understand the users’ needs, then the alpha release will be completely wrong and really major misconceptions will get into the beta!
The problem then with including users in this kind of testing is that they get really annoyed with having to go back and test everything that previously worked, just to check that it hasn’t been broken with recent changes. This is the sort of testing where I think automation could be really useful — so the user can trust the computer to go back and do everything they did before in the areas that worked and they can focus on the new developments!
August 24th, 2005 at 11:42 PM
Excellent point. Yes, first impressions with users are quite powerful. We are in agreement.
August 25th, 2005 at 10:14 PM
We could implant remote control devices in users’ brains…?