Testware for Free
Testware for Free
Adventures of a Freeware Explorer
Want the 4-1-1 on freeware but don't know where to start? A virtual cornucopia of programs awaits your discovery. In this week's column, Danny Faught details some of the testware he has researched, and explains why it's important to have freeware in your testing tool bag of tricks.
There's gold in them thar hills! For the past several months, I've been wandering the wild frontiers of free test tools, panning for tools that are free to anyone adventurous enough to search them out. I've found a few nuggets, plus a lot of stuff I've put back into the stream. My adventure has barely begun, but I can share with you some of what I've discovered so far.
What is freeware?
That's what I call any tool, software library, or web-based service that is free to set up and use, and that doesn't have unreasonable license restrictions like limiting the number of users, the amount of data it can process, or the amount of time you can use it. This includes Open Source software, public domain software, and some free binary distributions.
When I write about freeware test tools for Open Testware Reviews, I make sure to add them to the tools list on testingfaqs.org. I've done broad surveys of three categories so far-test drivers, defect tracking tools, and unit test tools-so these are where you'll find the most up-to-date entries for freeware. There are nine freeware entries for test drivers/test management tools, each solving a roughly similar problem in a different way. There are thirteen defect tracking tools listed. I'm amazed at how many defect tracking tools that the market continues to support, both commercial and free. For unit test tools-drum roll please-I've found forty-three tools worth listing, but that's only a sampling of what's out there. The unit test tools cover a broad range of programming languages, but there are only a few to choose from for each language.
Where will my adventures take me next?
Scouts tell me there are dozens of free load test tools out there. In that category, I've explored OpenSTA, which many people rave about, though your mileage may vary. I also have my own entry in this category, the stress_driver.
Tools that I classify as "test implementation tools" also promise to give us a few valuable nuggets. These tools help out while tests are running, and the category includes free memory allocation debug libraries like dmalloc, and data comparators like MDBDiff. I'll be cataloging many more in the near future.
Another type of treasure I'm hot on the trail of is static analysis tools. This category includes tools that pore over your source code looking for bugs, such as Jlint,tools that count lines of code and other metrics like sclc, and html link checkers like Xenu's Link Sleuth.
There are some places where the choices are fewer, but there are still some gems. A good example of a test design tool is ALLPAIRS. There are a handful of free GUI test tools, like Perl's Win32::GuiTest library. ATAC is a representative of the test coverage tool category. In the dreaded "miscellaneous" category is Codestriker, a tool that supports collaborative code reviews.
Further on the horizon is exploring territories never before covered by testingfaqs.org. This may include canned test suites, scripting languages and environments, and if I can summon the courage, security testing tools. There's plenty to be discovered in those areas.
Free Tooling: When are they most useful?
The freeware test tool territory is still wild and woolly. There are few free tools that rival their commercial counterparts in terms of features, reliability, and support. But they still have a place in your chuck wagon. Here's why.
You may not be sure that you need a particular tool, for example, a GUI capture/replay tool. You may have to invest a lot of time and money to pilot a project using one of the big-name commercial GUI tools. But if you recognize that you're going to have to do some programming even with a capture/replay tool, you may choose to skip the capture feature and code your tests directly using one of the free GUI test libraries.
You may decide that's all you need, and you'll have saved a lot of money and hassle. Or you may find yourself wishing you had features that are only available in a commercial tool, and then you'll approach the tool purchase decision much wiser. Or maybe you need to do some rudimentary link checking on your web site. There are several freeware tools that are up to the challenge. Most commercial web test tools offer many more features than link checking, and you can step up to them if your needs expand.
Maybe you're testing on a platform that is not well supported by the commercial tool vendors. I recently worked on a platform that was not supported by a single commercial tool vendor. If you have an open source test tool, you're free to apply the resources necessary to port a tool to your platform, without being at the mercy of the whims of your tool vendor.
One more argument in favor of freeware tools is the educational factor. You can make yourself more attractive on the job market by learning a new type of tool. It may be difficult to get licenses for a commercial tool just to play with it, but with freeware you can be up and running in minutes. Freeware is also well-suited for the classroom, and in fact, the open source culture has its roots in the academic research environment. I'm setting up a class to teach kids how to program, and I'm using an open source language so that my students won't have to spend any more money to be able to practice at home between classes.
There's room for both freeware and commercial tools in your arsenal. Because the freeware tools arena doesn't tend to have a big advertising budget, it's harder to get the word out about them. It's all about people telling people which ones worked best for them.
Further reading:
- Boost Your Testing Super Powers by James Bach
- Software Quality Notes: Test Tools for Free by Danny R. Faught
- A Survey of Freeware Test Tools by Danny R. Faught (slides)
Comments
#1 Submitted by carol long on Tue, 05/20/2003 - 12:34pm.
Neill Mccarthy has a point. Some of the freeware licences (especially for libraries) insist that anything developed with them must be open source or freeware. If you are developing commercial products using freeware, shareware or open source tools with these sorts of restrictions, it could negate your rights over your intellectual property and give you problems in the market place. For this reason some R&D shops have locked down every machine only allowing known and approved software. However, many places are sensible enough to have a policy that they will evaluate useful freeware (etc) and get their lawyers to check out the licence that came with it. Always make sure you know what this process is. There are good development reasons to have control over the tools - especially in the longer term. You are making a product for a customer and their demands on you may require you to recreate a past event or fix a bug in their "special" version - you need to be able to do that with the same tools or you have to rework everything to make sure it works.Don't through away the commercial advantage of having lower cost tools without looking at the possibilities. Similarly, don't assume that because something is as readily available as the normal "glossy box" tools you've used in the past that the terms are the same. Always check your licence agreement makes sense for your purpose.
#2 Submitted by neill mccarthy on Tue, 05/20/2003 - 6:06am.
I must admit I have started to smuggle these free and shareware tools into my tool box for testing. My two biggest challenges are getting these tools past corporate rules and software policy and the corporate fear of the unknown. Many sites I have worked have desk top and server lockdowns on what can be loaded and unless it is a recognised boxed product on a list then it aint going to get installed as no mater how well i can prove it as a pilot it will not go any further till it has gone through due process.The other is the fear of the unknown, with departments taking the view, we have not heard of this supplier/tool and they are not a major player so we are not taking it and anyway it will add another learning curve and overhead as it is something else that will need to maintained. I still struggle with this even though I am often enabled to write my own perl scripts and VBA apps as test harnesses. Good article and good news letter btw.
#3 Submitted by Mitch Krause on Mon, 05/19/2003 - 3:25pm.
Danny,Just out of curiosity, what was the platform that your recently worked on that wasn't supported by any commercial tool vendor?
#4 Submitted by Leo Hussein on Wed, 05/21/2003 - 7:47am.
This is just what I have been looking for, i.e. someone who has already found the freeware tools and also has/is prepared to review them. I look forward to giving you my experiences of using them once I get started. One thing that concerns me about freebie tools are they do not come with ample documentation, I can understand if the tool is easy to setup and use then there is no need for documentation. can you mention this aspect when you're trawling through these tools and I will do them same. The main reason why I am interested in these tools is because the project I am working on as very tight budgets and cannot afford to spend the 000's of Dollars/Pounds that most vendors are asking for test tools, especially when you are not sure if they will do the job now and in the future.
#5 Submitted by Ryan Wilcox on Tue, 05/20/2003 - 6:13pm.
bret : i actually have worked on a few projects where the testing tools were eventually rolled into the product (you may actually know them :)). granted, this is not a practice that is right for everyone, however, given the situation we were in, it made perfect sense. after (x) amount of time, we as consultants were rolled off the project, and the customer was able to continue the enhancement and use of the tool.i should note in both cases these were opensource tools that were used, one of which was home grown by our project team which later became opensource.all things being equal, i feel rolling a testing tool into a product is a win-win situation for most.
#6 Submitted by Bret Pettichord on Tue, 05/20/2003 - 5:29pm.
I use a lot of open source tools in my work. Any tools, open source or not, can have heinous license terms. Even so, i don't see how this affects test tools. In almost every case, test tools are used for testing a product. They are separate from the product, and therefore can't affect your companies intellectual property rights to the product itself. Is anyone actually incorporating their test tools into their product?