White PapersThrowing more time and money at the problem is not necessarily the only—or best— answer. Consider re-evaluating your test strategy.
In this paper, we address how to maximize the return on investment by deliberately hitting high-impact areas. Find out how to augment your QA team with a high-power tool that you can implement at a fraction of the cost of adding additional resources or expanding schedules. The resulting impact on quality is impressive.
Software is often shipped without any guarantee it’s free of one or more show-stopping, intermittent bugs that have the potential to corrupt data, destroy profit margins and undermine credibility. Software litigation has become a multi-million-dollar industry, a sobering reminder of the enormous repercussions of these failures.
Re-evaluate QA test strategy to ensure successful software failures Application failures may not be the fault of QA managers but they are their responsibility. Conventional quality assurance can unknowingly let some of the most vulnerable areas in software go totally untested. Today, the average QA manager is in a no-win situation. He or she faces an ever-increasing test matrix while time-to-market and budget constraints sabotage QA s best objectives. Throwing more time and money at the problem is not necessarily the only or best answer. Consider re-evaluating your test strategy. In this paper, we address how to maximize the return on QA investment by deliberately hitting high-impact areas. Find out how to augment your QA team with a high-power tool that you can implement at a fraction of the cost of adding additional resources or expanding schedules. The resulting impact on quality is impressive. Universal QA challenges Software is often shipped without any guarantee it s free of one or more show-stopping, intermittent bugs that have the potential to corrupt data, destroy profit margins and undermine credibility. Software litigation has become a multi-million-dollar industry, a sobering reminder of the enormous repercussions of these failures. Most end users are forgiving when a menu entry is not enabled or a display is not as sharp or bright as it should be. Those problems can be easily communicated and resolved. But a report from the field that an application is suddenly disappearing or crashing can wreak havoc on both QA and Engineering an understandable impact considering this alarming statistic from Gartner: The average cost of downtime for a mission-critical application is 100,000 per hour.1 In the software industry, time allotted for QA is limited, and QA resources are often stretched meaning you must maximize the value of each. If you cannot afford derailed schedules and lost productivity, it may be time to make a radical change in your approach to QA. You can guarantee software stability before it ships and ensure the software will alert you to problems once it is deployed. Simulation of failure why it s vital to QA An explosion in the fuel cells of Apollo 13 could have killed the on-board astronauts quickly or, perhaps worse, left the crippled spacecraft in a perpetual orbit above the earth. Averting catastrophe, crew members were able to maneuver Apollo 13 back to earth, and the mission was deemed a successful failure. The term successful failure applies to software testing, too. Software that causes a blue-screen crash, hang condition or data corruption is not just the fault of development, but also of QA. Redirecting QA strategy means changing priorities: To guarantee a software failure is successful, QA must test for error handling as well as errors. Software runs in an environment loaded with variables. Endless permutations of operating systems, service packs, system configurations, privileges and third-party components make the test matrix endless yet in the real world, software will still encounter unexpected environmental conditions. As much as 30 percent of your software (regardless of language) can be dedicated to error-handling code that executes when it encounters an unexpected condition. Like an aerialist who performs without a safety net, the odds are stacked against you if you ship software with untested, unproven error-handling capabilities. Error handling must be present and it must be good meaning it must communicate the parameters to the problem, not just the problem itself. In addition, it must preserve data rather than corrupt it, and gracefully exit or continue when possible. To ensure this, it must be tested. Consider the following error message that frequently announces a software issue: Object reference not set to an instance of an object. The million-dollar question is, What object? Nothing is more frustrating (or expensive to resolve) than deployed software displaying this near-meaningless error message, which omits the most important piece of information. 1 Software Quality in a Global Environment: Delivering Business Value; Theresa Lanowitz, Gartner Application Development Summit Presentation, September 2004. Untitled Document2In terms of error handling, a solid QA test strategy must: 1. Accept the things that cannot be changed. New service packs, oddball configurations, errant installations and unexpected parameter values could be missed by your QA test matrices. Acknowledge that these items have the potential to wreak havoc on your stability and credibility. 2. Build the best defense. Perform error-handling tests to discover quality risks. The QA manager s goal is to implement a robust quality strategy. While he or she is responsible for testing product features and functionality, a robust test plan must balance this by testing the application s error handling. Institute a solid practice of testing error handling and confirm that appropriate safety nets exist. Remember, exception handling is your first line of defense. 3. Confirm the quality of exception handling. Verify error handling will give the developer the exact information he or she needs to quickly resolve the bug. It is not enough to know where it broke; you also must know why it broke. Help QA with a third-party tool Testing error handling can no doubt play an invaluable role in your software s success. However, it adds yet another item to an already bloated test matrix. Consider your current QA strategy to determine: >> Areas most vulnerable to high-profile, destabilizing bugs. How do you test them once they are identified? Will they receive needed attention? >> How to connect product testing with the level of expertise on your QA staff. Do your QA resources lean toward testing what is needed or what is within their technical comfort range? >> The best way to avoid the mistakes of the previous release. Did two or three bugs absorb significant Engineering, QA and Technical Support resources because of diagnosis difficulties? If you are not confident your attack plan aggressively targets these areas, be open to the merits of using a third-party tool to supplement QA. When evaluating a tool to augment quality assurance, consider these attributes that define a smart investment: 1. It s easy to use.Will the tool impose a huge learning curve to get to the core feature set? Must a QA resource have a high level of technical expertise and/or make a significant time investment to become proficient at using it? 2. It delivers bang for the buck. Will the tool s expense provide a return on investment by increasing test coverage? Can it raise the bar, bringing the test strategy to a higher standard? Will it improve the skill set and technical savvy of QA staff? Can it be integrated into the current test plan and incorporated into an automation strategy? 3. It s a good fit with the application development life cycle. Will the tool deliver continued value by providing vital functionality in QA test strategy? Will its features be applicable for future test plans? Can it adapt to product changes and still return cost savings due to increased test coverage? Improve your odds with Fault Simulator By neglecting to test for the unpredictable, you are gambling your application will respond appropriately. If your test matrix does not currently account for vulnerabilities, take this opportunity to play the game a lot smarter. Consider a tool such as Compuware s DevPartner Fault Simulator 2.0 QA Edition to fill the void. It will analyze your application and generate environmental test cases that can produce an application s greatest vulnerabilities.Fault Simulator clearly points out potential failure points such as an offline network to QA analysts, making it unnecessary for them to understand the source code. Untitled Document3With a single button click, Fault Simulator s Watch My Target feature will record potential network, registry, COM and Disk I/O failure points. QA engineers never have to get involved with source code; they need only know how to run the application being tested. Technical expertise is not required. One execution of your program with Fault Simulator will generate environmental test cases that would otherwise: a) require in-depth source knowledge to create and, b) be difficult, if not impossible, to simulate/test. For example, without touching or interfering with your actual network connection, Fault Simulator s Watch My Target feature can discover and then simulate potential network failure points such as: >> timed-out connection >> offline network >> remote server crash >> unavailable server. Fault Simulator will reveal whether your program communicates the problem or crashes the system. Suppose you are testing order processing in a web application. How do you evaluate what happens when the user selects Purchase and the credit card server is not available? What is communicated to the user? Are items in the cart deducted from the inventory database? Does the user experience a crash? Is the order lost? There s no way to know without testing, and how do you test for such a scenario? Is it realistic or even feasible to take a credit card server offline for testing purposes? Fault Simulator offers a powerful advantage: QA teams can easily adopt environmental testing as part of their regimens. Find out exactly how your error handling performs using repeatable test cases set up with little more than a button click. Without such a tool, the only way to test for real-world events such as network failures is to manually pull network cables. Address COM-related problems head-on The reality is, one unavailable COM component has the ability to bring down an otherwise solid application. Will the problem be communicated or will the application simply crash with an obscure error message? The inherent sophistication of COM introduces a whole new opportunity for environmental issues. Deployed software can easily be broadsided by COM-related problems. The installation and deletion of software, presence of third-party components sharing a COM object and upgrades to existing COM components can combine to create a dangerous mix for an application that hasn t been thoroughly tested. Yet, COM-related problems are generally overlooked in the QA test matrix due to the level of technical understanding required. Fault Simulator QA Edition is a very simple solution to a very painful problem. It gives you the power to discover and then simulate the types of COM problems that typically bring software down, including: >> CLSID not found >> DLL not found >> Interface not registered >> ProgID not found. When these types of problems occur in deployed software, a technical support representative can spend days chasing them. Now, testing for them can be part of your regular QA process. If your test strategy currently lacks COM-related testing, you can easily plug a significant hole with Fault Simulator s test-code coverage. New Watch My Target feature learns what environment resources the application uses, then simulates faults in the environment. Untitled DocumentCompuware Corporation Corporate HeadquartersOne Campus MartiusDetroit, MI 48226For regional and international office contacts, please visit our web site at www.compuware.comAll Compuware products and services listed within are trademarks or registered trademarks of Compuware Corporation. Java and all Java-based marks are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries.All other company or product names are trademarks of their respective owners. 2007 Compuware Corporation1/07Testing for trouble QA s best defense One of the most common landmines in development is to assume users have adequate permissions and disk access. Without Fault Simulator QA Edition, it is extremely difficult to ensure an application responds appropriately in the event a disk, drive or file cannot be accessed as anticipated. Without touching the drive or disk, Fault Simulator can mimic failures in areas such as corrupt or missing files, insufficient privileges or locked files. Is your program written so it accounts for the possibility of a corrupt file, a locked file or a missing DLL? Or does it assume all will be present and healthy? The best defense is to test for it. Software all too frequently stops dead in its tracks when it must rely on information in the Windows registry that is either unavailable or inaccessible. Has your source been designed to communicate a missing registry key, insufficient privileges or corrupt values? Using Fault Simulator, you can create a test strategy to simulate registry problems that occur out in the field. Problems such as corrupt registry value, missing key and insufficient privileges are notorious for making their presence known in deployed applications. To learn more about DevPartner Fault Simulator, visitwww.compuware.com/products/devpartner/fault-simulator.htm Compuware Corporation (NASDAQ: CPWR) maximizes the value IT brings to the business by helping CIOs more effectively manage the business of IT. Compuware solutions accelerate the development, improve the quality and enhance the per formance of critical business systems while enabling CIOs to align and govern the entire IT portfolio, increasing efficiency, cost control and employee productivity throughout the IT organization. Founded in 1973, Compuware serves the world s leading IT organizations, including95 percent of the Fortune 100 compa nies. Learn more about Compuware at www.compuware.com.Compuware products and professional services delivering IT valueAugment the skills of your QA staff A tool such as DevPartner Fault Simulator QA Edition not only delivers the power to simulate, it also provides a full report complete with call-stack information. DevPartner Fault Simulator can also simulate a wide range of other types of failures, including memory and managed exceptions from .NET assemblies. Take it one step further and use Fault Simulator to generate batch files for test automation. Ship software knowing your error handlers will communicate and preserve data, giving you the best of all possible outcomes when problems occur. Think outside the box, as QA resources are limited and the test matrix is growing. Consider third-party tools as an option for strengthening your defense against the unexpected. The cost of a dedicated tool may be one of your smartest investments. Tools can be powerful teachers bolstering the skill set of your QA resources. If a significant portion of your test strategy is not dedicated to negative testing that is, testing for failure you are letting the most valuable part of your source code go out the door untested.
