Table of Contents
- Test Tool Comparison
- Quick Test Pro (QTP)
- Silk Test
- Selenium
- (1) Features Comparison
- (2) Usability Aspects Comparison
- (3) Comparison of Debugging Features
- (4) Performance Comparison
- (5) System Requirements Comparison
- (6) Technical Support Comparison
- (7) Customer Service & Pricing Comparison
- Conclusion
While manual testing and automated testing go hand in hand, one of the important benefits of automated testing is the assurance that the software is reliable even after updates.
In addition, redundancy detection tests ensure that the integrity of the software is maintained while operational costs are optimized.
The increasing usage of automation tools in recent times has brought in several products into the market, making it tough to choose the right product for your testing needs.
Here is an evaluation of three popular automation test tools.
Test Tool Comparison
Here are the top three automated testing tools that are widely used by testers across the globe. For each of the tools chosen, aspects like the history, striking features, performance, and target audience are considered. In addition, these aspects are considered.
- Features
- Ease of use
- Integration with other tools
- Quality and Performance
- Customer support and service
- Price
Quick Test Pro (QTP)
Quick Test Pro or QTP is one of the top performers in the automated testing tool arena. With an easy-to-use interface, reliable performance, easy integration with other testing tools and support for .NET environment, QTP has become the leader in the functional and regression testing segment. For a quick history, QTP is a product of Mercury Interactive launched in 2002. In 2007, Hewlett-Packard took over the product to lead the testing tools segment.
QTP is an object-based tool that allows you to automate test operations based on utility, automation or test objects. By being a keyword-driven tool, QTP allows you to use keywords in the creation of test scripts. The tool supports desktop-based and web-based applications and not command-line interface ones. For programming purposes, Microsoft VB scripting is used and data-driven approaches use Excel. Here are some of the striking features of the tool.
- Simple and easy-to-use interface
- Microsoft VB scripting language that offers extensive functions and resources
- Novice users can easily create test cases
- Test run iterations are made easy
- Supports .NET environment and XML
- Easy integration with other testing tools
- Object identification mechanism is improved
- Report generation for quick analysis
Silk Test
For enterprise applications, one of the powerful automated functional and regression testing tool is Silk Test. It was originally designed by Segue Software which was later acquired by Borland and then by Micro Focus International. It supports Object-oriented programming, though partially. You will find objects, classes and inheritance concepts that are seen in C++. It distinguishes each window and control in an application and associated specific attributed to each control for effective object-oriented implementation. The bitmap tool helps the software to capture windows and controls. Using silk test, you can easily identify mouse movements, images on the UI and keystrokes.
- Startup config settings are stored in partner.ini file. You can also reconfigure these settings from the options menu.
- Tests are terminated when an exception occurs
- Uses object-oriented implementation supporting classes and objects
- A true hierarchy that is object-oriented
- Effective object recognition and verification mechanism
- Built-in functions available for SQL queries and data-driven testing
- A built-in recovery system is available
- Uses 4test scripting language that is specific to a domain
Selenium
Selenium is a popular open-source testing tool for web applications. Not only that, but you can also automate administrative tasks as well. The increasing popularity for Selenium is due to the fact that major browser vendors made selenium a native part of their browsers. Whether you want to distribute scripts across multiple environments or create quick bug reproduction scripts, Selenium has the right answer for you.
Selenium is a suite of software testing tools. In 2004, Jason Huggins was testing an internal application when he thought it was a good idea to create a JavaScript library that allows a tester to rerun scripts instead of manually running them on each browser. This is when Selenium got its life. Later, it underwent several updates to turn into a powerful web testing tool.
Here are some of the striking features of Selenium:
- Highly flexible
- Good for directed tests
- Easy to setup and use
- An intuitive interface for unit tests
- Multiple languages supported
- Runs in multiple browsers and platforms
- Offers robust frameworks by integrating with other tools
- Allows you to connect multiple grids and run parallel tests at the same time
Here are some of the factors considered for the evaluation. The first one is the feature set offered by the tool:
(1) Features Comparison
| Features | HP QTP | Silk Test | Selenium |
| Installation | Required | Required | Required |
| Knowledge of scripts | To some extent. But you can read the QTP manual, run sample scripts to start using the tool | Required to certain extent | Coding knowledge is required. You need to know the technicalities of framework integration. |
| Cloud-based | No | No | No |
| Test Termination | Continues to check other conditions after an exception. Tests can be terminated using ExitTest | Terminates on exceptions | Continues to check other conditions after an exception. Tests can be paused at an exception by adding a plugin |
| Add-ins and Extensions | QTP creates test objects and Test object description to verify them on run-time. It uses Mandatory properties, Assistive properties and Ordinal Identifier to do so. You can conveniently use the Add-in manager to select the desired add-in. | Supports classes and objects when optional extensions are enabled | Selenium stores user extensions in separate files. Add-ins are available for Mozilla only |
| Visual recorders | Records different tasks like the creation of GUI maps for the complete application, data and object checks and capturing of user actions as well | Available to identify objects and verify them. You can create test frames, capture user actions, inspect and verify windows, objects etc | Does not have direct recording tool but you can use third party tools like Castro for Python and Monte Media Library for Javascripts |
| Object Hierarchy | Supports classes but does not support inheritance | Truly supported | Yes |
| Object recognition and verification | Yes, it uses GUI checkpoint recorders for validation of objects | Uses a lookup table mechanism to distinguish objects | Yes, uses DOM structure of HTML pages to recognize objects |
| Scripting Language | VB Script | Powerful OOP language 4Test | Selenese, a domain-specific scripting language that supports PHP, Perl, C#, Java, Ruby and Python |
| Database testing | Has visual recorders for database checkpoints. Can pass data values into test cases | Offers built-in functions to facilitate SQL queries. Can pass data values into test cases | You can use POI API to perform data driven testing. POI API can be configured through Eclipse |
| Windows application support | Yes | Yes | No |
(2) Usability Aspects Comparison
The second important aspect to consider is the usability feature. It is important to check how easy it is to install and run the tool
| Usability | HP QTP | Silk Test | Selenium |
| Installation | Easy to install / Self-explanatory | Easy but takes more time. You need to install IBM Installation manager as well | It is a time consuming process for first time users and requires knowledge of the tool as well. You need to install the RC server and choose the programming project |
| User-friendly interface | Yes | Yes | No |
| Automation Process | Pretty straightforward. | Not complex | Complex procedures. You need to run multiple components and programs working together to get the required solution |
| Help documentation | Adequate | Adequate | Communities and forums |
| Technical support | Dedicated | Dedicated | No |
(3) Comparison of Debugging Features
As automated testing tools allow you to identify errors in the code, debugging is an important aspect that needs proper attention
| Debugging | HP QTP | Silk Test | Selenium |
| Getting assistance during errors | Supports visual debugger with required capabilities | Supports visual debugger with required capabilities | Difficult to some extent. Take significant time to identify failures. You have to write a code to capture screenshots or DOM structures |
| Recording feature | Built-in | Built-in | Available with Mozilla plugin |
| Exception Handling | Built-in | Built-in | Have to write a code |
| Error messages | Failure report of each test case is documented separately | Failure report of all test cases is documented in a single file | Not very easy. It reports failures in a log console with failed tests in red color. You have to create a code to capture screenshots to identify the reason of failure. |
(4) Performance Comparison
Another important feature to look at is the performance of the tool when running multiple automated scripts.
| Performance | HP QTP | Silk Test | Selenium |
| Script creation time | Very less to create scripts but takes more time to run them | Considerably less | Requires more time and expertise to create scripts |
| Test Progress | HP QC is a comprehensive test management tool | Using SilkOrganizer, you can effectively track and manage tests | You can use Cucumber or other frameworks for test management tasks. |
| Consistency | Performs consistently | Performs consistently | Performs consistently |
(5) System Requirements Comparison
Another important aspect to look at is system requirements. Before purchasing an automated testing tool, you should know if it suits your office environment.
| System specifications | HP QTP | Silk Test | Selenium |
| Hardware | Pentium IV or higher ( 1 GHz and above)
1 GB RAM 1GB free space on the hard disk Windows XP SP2 and above |
2 GHz Intel or AMD processor
2 GB RAM 2 GB free storage space Windows XP SP3 and above |
Selenium requires more system resources
4 GB Memory 4 x Dual Core AMD Opteron |
| Programming Languages | VB Scripting language | 4Test, Silk4J, Silk4NET | Selenese, a domain-specific scripting language that supports PHP, Perl, C#, Java, Ruby and Python |
| Testing Interfaces | GUI, Web-based | GUI, web-based | Web-based, GUI |
(6) Technical Support Comparison
While choosing the right automation testing tool, it is important to check out how the tool supports different aspects
| Support features | HP QTP | Silk Test | Selenium |
| Windows Applications | Yes | Yes | No |
| Browsers | Google Chrome, IE | IE | Mozilla, Chrome, Internet Explorer, Opera, HtmlUnit |
| Environment | Windows | Windows | Solaris, Linux, Windows, OS X |
| Mobile Support | Supports phones and tablets but uses a different product called HP UFT mobile | SilkMobile supports all types of smartphones | Android, BlackBerry, iPad, iPhone |
| Supported frameworks | Integrates with HP Quality Center | .NET Framework | Supports multiple frameworks like Eclipse, Ant, Maven, Selenium, Jenkins, Hudson |
| Image recognition | Yes | Yes | Yes but requires expertise |
| File Upload | Supports uploading of all types of files | Yes | No option for file upload |
| Dialog Boxes | Supports all types of dialog boxes | Supports dialog boxes | To some extent |
(7) Customer Service & Pricing Comparison
| Support / Price | HP QTP | Silk Test | Selenium |
| Technical Support | Dedicated HP Support | Dedicated Micro Focus support | As it is open-source, there is no dedicated support. However, you can check out various communities and get answers. For any new issues, you need to explore workaround options or wait till someone posts a resolution in the forum. |
| Customer Service | Excellent | Good, but complaints are increasing | Not available |
| Price | Expensive. You have to purchase license and spend money on upgrades as well. | Expensive. | Being an open-source, it is free to download and use. |
Conclusion
Choosing the right automation testing tool is a tough ask. QTP, Silk Test and Selenium are all popular and powerful automated testing tools. However, choosing the right tool for your testing tasks depends on your infrastructure, testing requirements and expertise coding knowledge.
For large enterprises that can afford to spend dollars, QTP or Silk Test is a good choice. Being a complete suite, it offers a quick and easy way to automate testing tasks. On the other hand, Selenium suits small and medium businesses wherein the tool comes for free. However, testers need to have sound knowledge of coding languages.