Recently I have been appointed “Verification Leader, Core Foundation Team” whatever that means. Essentially it translates into taking care of functional testing. Ah, testing, well regarded, often ignored, always controversial.
“Testing” usually means “unit testing” to developers. Code is what we understand (I can’t speak Russian but I can speak C++ 🙂 ) “I’ve kept my API stable but I’ve optimized the internal implementation. What did I broke?” With proper unit testing, you can go for a coffee and when you are back answer to the question is on screen: “oh, after optimization I forgot about corner case X, I’ll fix it”.
So the code is right. As Knuth would say “Beware of bugs in the above code; I have only proved it correct, not tried it”.
Try. That’s the key. The user does not care whether the application is implemented in C++2011 using lambdas, concepts, variadic templates and throws exceptions across shared libraries, or in plain old Visual Basic 6. Does it work? Does it load the file when I click File -> Open? Will it show a computer when I click and drag the computer icon from the palette to the canvas? That’s what matters.
In KDE we have unit tests in kdelibs (in fact it’s a requirement to get code in) and there are a few UI tests for the KDE User Interface module, which contains essentially widgets for use with your KDE-based application.
What about applications? What about Amarok, Gwenview, Kexi, Okular, etc? How do we know they work fine? How do we know they actions do what they should do and the application renders as it should? Correct me if I’m wrong but I think currently we rely on the input from our users: when they report something’s misbehaving, we oblige.
Can we do better? Yes, we can. UI testing has been there for decades and even the venerable expect can help with that.
There are many tools for automated GUI testing. Some are open source, like Xnee or White, others are closed-source, like Froglogic Squish and SmartBear TestComplete, which also happen to have the best support for Qt-based applications.
Are those tools useful? Why, yes, they are. For instance, there is a well-known bug in KDE Windows which makes Digikam on Windows show empty context menus when the Qt style is set to CDE (the default; if you use the Oxygen style everything looks fine). This kind of bugs can be easily detected by simulating user actions, then OCR‘ing the text of the popup menu. Or even just telling the automatic UI test to take a screenshot of the popup, then comparing pictures. Same for PDFs that render oddly in Okular on some Linux distribution due to fonts, FreeType configuration, an old version of Poppler, etc
The tool I am most familiar with is TestComplete, which suffers a bit with alien widgets (you need to build Qt without accessibility support). I have been told Squish is much better but given that there is no freely downloadable version and it does not support WPF (a requirement for the project I’m working on now), I can’t try it.
Here is a wish for KDE users who want to join development but have little or no programming skills: go to SmartBear’s website, download TestComplete and record/write a few UI tests. Let us know (in the forums, in the mailing lists) how far you got. Maybe we can make UI testing -a must in the corporate software development- part of the KDE development and release process.
Update: apaku says froglogic provides free licenses for KDE, see the comments
I was going to try LDTP2 (Linux Desktop Testing Project) for my own project, and I’m surprised it’s not mentioned here. Has anyone tried it in KDE-land?
Indeed Squish is not freely downloadable, but we do provide a free-as-in-beer community edition for KDE projects that want to do gui testing. In addition getting a fully functional trial for your work-applications also does not cost you anything, except filling out a web form at http://www.froglogic.com/products/evaluate.php.
PS: For the community edition for KDE projects its best to contact us via email@example.com and please let us know the KDE project you want to use Squish on.
by GUI usability testing you mean testing to check if a button is till there (how does it dynamic context menus, etc…) or also usability testing, like that place is the best for that button?
Great! I have updated the post to reflect this.
IMHO if we want UI testing to achieve some real results, there should be an organized testing team, with clear guidelines on how to join, what to do, how to get a Squish license, etc.
Will we have a UI testing talk at Akademy?
Maybe this could be also discussed in the Platform 11 meeting (I won’t be there)
UI testing = simulate user actions. I have not mentioned “usability” in the post at all. Usability is a different field, completely orthogonal to UI testing.
Seems to be a nice free alternative to squish.
I’d like to second that comment by The User. I’ve used it to experiment with KDE UI testing, and it works great, with the added bonus of actually being open source. (I wrote a comment about this yesterday, but for some reason it seems to have been swallowed up).
TDriver developed for Meego looks good too:
Havn’t tried it myself yet.
I’m one of the TDriver developers and I can say that it works great with KDE apps. I tried to make a quick demo on how to test that digikam issue, but I cant reproduce it 🙂
If have a good use case in mind I can create an example for you.