True Test Of The Web

Any companies doing business on the World Wide Web are one click away from disaster. With Web sites playing an increasingly important business role, it's essential that these systems deliver--whether their job is handling product sales, distributing information, or linking up with partners' systems. Of course, gaining that assurance has been anything but easy. The task of testing the performance of a Web application involves more than just running a functional test on the specific application, though that's important. It's also critical to ensure that all the supporting infrastructure works under varying pressures and conditions.

A growing number of vendors are addressing that difficult task with testing tools that range from freeware to complex bundles of products and services that can cost $250,000 and up.

The high stakes of E-business mean it's no longer a question of whether to test but how. Consider the experiences of those companies that have suffered high-profile site failures. In October 1997, with a market rally in process, E-Trade Securities Inc., the Palo Alto, Calif., pioneer in online stock trading, suffered a lengthy and embarrassing service outage when trading volumes got too high for its system to handle. The company survived the crisis and made several important technical and procedural changes--but not before seriously damaging the confidence of its customers.

Similarly, a glitch during unscheduled maintenance at Amazon.com forced the site offline for several hours last year, prompting rumors that the online bookseller had gone out of business. Although Amazon.com wouldn't reveal the cost of the site shutdown, outside analysts point out that daily business on the Amazon.com site can be as high as $400,000. And in order to repair its customer relations, the company felt compelled to issue discounts to those who claimed to have been unable to access the site during the shutdown.

Troubled Debut
Worst of all might have been what happened to Egghead Software Inc. The very weekend that the retailer closed its last store as part of a much publicized move to Web-based electronic product distribution, its Egghead.com site went down for two days. Again, the company won't say what the glitch cost but outsiders note that Egghead probably has about $1 million a week in online sales, making a botched debut a matter of at least a few hundred thousand dollars in lost revenue.

These three examples of Web-site failures are indicative of what companies can expect as they put their most critical operations on the Web. Customers have little tolerance for failure. So the questions for these Web-based businesses are, who should test and how often?

New Venture, Or Old Rules?
Sandy Taylor, director of analyst services at the Software Productivity Group in Natick, Mass., divides these risky new Web ventures into two categories: companies that are developing a brand-new E-commerce Web site with built-from-the-ground-up functionality, and companies that are simply building Web functionality on top of their existing business rules and infrastructures. Taylor says companies in the first group are the ones taking the biggest risks and therefore are most in need of testing. Companies in the second category also must test their applications, but they at least have some existing metrics and experience to build upon.

Industry experts agree that Web testing is a good investment. Testing in general has achieved a lot more visibility lately, aided in part by the focus on the year 2000 remediation efforts. And, given the visibility of Web sites, it's fairly easy to argue for testing, says Liz Barnett, a VP at Giga Information Group. Barnett is especially bullish on the need to test sites involved in E-commerce. Although large intranets and extranets can face similar problems, she says, it's even more difficult for site developers to anticipate load once an E-commerce site is opened to the general public. Doing business on the Web is like opening a traditional store but not knowing whether it will be located in the middle of an Iowa cornfield or on New York's Fifth Avenue.

Furthermore, in the world of the Web, functional changes happen much faster and more frequently than in traditional mainframe environments, which tend to implement functional changes annually Client-server environments tend to implement functional changes on a monthly basis. But Web sites sometime need to make daily changes, which can impact back-end infrastructure and system functionality.

Fortunately, the problems faced in the testing of Web applications are not entirely new. Indeed, the history of most site-testing tools goes back to the traditions of client-server computing. In fact, from a conceptual standpoint, Web testing is much like client-server testing--but with more potential tiers and greater load variability. With a Web site, there is no way to predict with certainty how many users will hit a site at any given time. The Web environment has also introduced extreme heterogeneity to the support equation. Users may be accessing resources deep in a company's infrastructure, and the IT department can't control what browser they will be using. The potential interactions between technologies are almost limitless.

That's why some analysts prefer the term Web application testing to the more traditional functional and load testing. What matters is the application seen by the users, not whether the Web site is having trouble or the server is broken.

Three Functions Of App Testing
Within application testing there are three functional areas. The first is site testing: a structural test that looks to see whether there are pages that are inaccessible or whether popular features are unnecessarily hard to access. The second and third areas deal with whether the application functions as designed, and whether that functionality survives at various load levels and under a variety of scenarios.

Vendors offer each of these capabilities in a variety of flavors and levels of complexity. Some also offer integrated suites that provide life-cycle testing.

The Web-testing market is young and fragmented. Prominent vendors include Mercury Interactive and Segue Software, whose roots are in client-server testing. RSW Software is a new subsidiary of hardware test giant Teradyne and focuses exclusively on Web testing.

Mercury Interactive is generally regarded as the 800-pound gorilla of the business, with a mature and well-respected product line that includes LoadRunner and Astra. Segue is an aggressive company offering tools such as SilkPerformer and a sophisticated scenario testing product called LiveQuality (see story, p. 6A) that permits the generation of complex interaction with large numbers of virtual users. Other players include Compuware, Mindcraft, Rational Software, and RadView Software.

Web Watching
All of these products are currently being put through their paces in the real world. E-Trade for example, is using Mercury Interactive's LoadRunner. After its aforementioned meltdown, E-Trade vowed to never go through that again, says Gary Kattge, director of quality assurance. E-Trade looks back at that experience as positive because the company learned a great deal from it. "Since that time, we've had much higher peaks, which we've handled without any problems," he says.

The meltdown forced E-Trade to adopt a proprietary stateless architecture, which enables the E-Trade Web site to provide secure, reliable, and almost instantaneous service simultaneously to hundreds of thousands of users.

E-Trade, which rolled out its new and improved Web site this past summer to all 500,000 active accounts, experiences about 2.2 million page views per day, and expects further traffic increases as more noncustomers visit the site. During the quarter ended June 30, E-Trade processed an average of $1.5 billion in customer transactions per week.

In preparing for its public launch, E-Trade developed and installed a technology infrastructure able to handle massive traffic volumes. E-Trade's Stateless Architecture system allows more than 1 million visitors to use the Web site simultaneously and permits as many as 150,000 customers to place orders at the same time. E-Trade says the system can be scaled up to handle even heavier loads.

Getting the system to that level of reliability was Kattge's job, and testing was a big part of the project. Several testing-tool vendors were in the running initially, Kattge says, but based on functionality and extensibility, E-Trade reduced the field quickly to one: Mercury Interactive.

Mercury Interactive's LoadRunner product lets E-Trade put hooks in the test code. This allows E-Trade personnel look in-depth at application response time--right down to the SQL database-query level. The testing tool has given E-Trade the ability to calibrate and verify new system designs and to detect areas that might need additional tuning.

But LoadRunner's most critical role was in helping to measure response times between various tiers, where the potential for serious errors exists. LoadRunner is used on an hourly basis as part of an alert system to quickly escalate any performance issues likely to require intervention.

Commonplace Tests
Even in the slightly slower-paced world of mutual funds, testing is becoming commonplace. Mutual funds giant T. Rowe Price in Baltimore is a strong proponent of Web testing. T. Rowe Price recently implemented its first major Web site redesign. Time to market and availability were critical to IT manager Tom Dietz, who has been looking at Web testing issues at T. Rowe Price for more than two years. "We recognized Web-site testing as a very important area," he says.

After evaluating testing tools, the company settled on Segue's SilkPerformer, due in part to its regression testing, Web functional testing, and load-testing capabilities, Dietz says.

First Call Corp. in Boston is another organization with a Web site that gets put to the test. First Call keeps broker notes, research documents, earnings, and buy-sell recommendations flowing to its Wall Street professionals via its Web site. The company's roots in online services predate the Web, but when it came time to embrace Internet delivery, they selected Segue's SilkPerformer. "We were interested in testing costs, the ability to simulate user loads, and memory footprints per virtual user," says First Call IT manager James O'Keefe.

FirstCall also wanted a good scripting language. While it was important to demonstrate site viability through load and stress testing, it was equally critical that the scripting tool be able to show that real information was being delivered. SilkPerformer let the First Call team monitor specific deliverables like HTML code, or a copyright symbol that could verify that the site was delivering as intended.

Of course, not every mission-critical site is seen by the public. Paul Roach, a senior quality assurance specialist in the Richardson, Texas, office of Northern Telecom Ltd., got his exposure to Web-site testing during the requirements phase of Nortel's global intranet development project.

Nortel's intranet consists of many applications deployed across a global intranet--an environment that demands frequent testing of the Web servers and applications. Scalability, robustness, and a rich feature set were on Nortel's shopping list.

In one instance, Roach helped deploy an employee-satisfaction survey that would be accessible by 70,000 employees worldwide. Other sites required testing of functions as diverse as financial accounting, sales, and marketing. Dropping the ball with employees, even on a satisfaction survey, can be just as serious as offending paying customers, he says.

Cost Conscious
Choosing a product and determining what to spend are perhaps the greatest challenges to the newcomer to Web testing. CBS.com has opted for a freeware program. Compared with the tasks of supporting real-time stock trading or other "heavy duty" E-commerce activities, Michael Skorski's job might seem easy. Director of technology for CBS Net Media, Skorski makes sure www.cbs.com is up and running when the network's viewers come looking for program information and news. And the old adage about the short shelf life of news applies to Web news sites--specially when there's scandal in the air. "We did four times our normal traffic and an order of magnitude more hits and page requests when Kenneth Starr's report hit," says Skorski.

Skorski and CBS have been testing their site extensively, which may account for the fact that it survived the onslaught of Starr gazers with only minor problems.

That's attributed in part to CBS's use of Webstone, a freeware product now maintained and distributed by Mindcraft. "There are limitations on what it can do," Skorski says, "but the price is right and the product is now part of our testing regime."

Commercial testing tools have more bells and whistles, Skorski says, but do not necessarily produce more useful information. His biggest challenge with testing is to simulate the complex infrastructure of www.cbs.com.

Karl Van Neste, a software system architect at the College Board in Reston, Va., warns buyers to beware. "Test managers need to continually review the products and new capabilities available on the market." Van Neste has been using a product recently introduced by a small company called RadView Software. Van Neste says he first looked at Mercury Interactive's LoadRunner but that RadView's product had what he felt was comparable functionality for a small fraction of the price.

Indeed, new entrants keep cropping up almost daily, making it likely that increased capability and decreased price will be the order of the day. IT managers will also see a variety of service offerings for those shops that don't want to experience the cost and learning curve of Web testing tools.