ProfessionalSoftware Testing with Visual Studio 2005 Team System ppt

Arnold ffirs.tex V4 - 08/02/2007 10:24am Page v
Professional
Software Testing with
Visual Studio
®
2005 Team System
ToolsforSoftwareDevelopers
andTestEngineers
Tom Arnold
Dominic Hopton
Andy Leonard
Mike Frost
Wiley Publishing, Inc.
Arnold frontcover1.tex V4 - 08/02/2007 10:23am Page ii
Arnold frontcover1.tex V4 - 08/02/2007 10:23am Page i
Professional Software Testing with
Visual Studio
®
2005 Team System
Introduction . . . . . . . . . . . . . . .xxiii
Chapter 1: Introduction to VSTEST and VSTESD . . . . . . . . . 1
Chapter 2: A Quick Tour of VSTEST and VSTESD . . . . . . . 15
Chapter 3: Unit Testing with VSTEST and VSTESD . . . . . . . . . . 41
Chapter 4: Testing the Database . . . . . . . . . . . . . . 73
Chapter 5: Web Testing . . . . . . . . . . . . . . . . . . . . 125
Chapter 6: Using Manual, Ordered, and Generic Test Types . . . . . . . 165
Chapter 7: Load Testing . . . . . . . . . . . . . . 191
Chapter 8: Using Code Analysis and Dynamic Analysis . . . . 243
Chapter 9: VSTEST and VSTESD within the Software Development
LifeCycle 273

Appendix A: Installing Team Explorer . . . . . . . . . 305
Appendix B: Creating and Running a Web Test: A High-Level
Walk-Through 313
Appendix C: Creating and Running a Unit Test: A High-Level
Walk-Through 319
Appendix D: Creating and Running a Load Test: A High-Level
Walk-Through 327
Appendix E: Creating and Running a Manual Test: A High-Level
Walk-Through 339
Appendix F: Other Sources of Information . . . . . . . 347
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351
Arnold frontcover1.tex V4 - 08/02/2007 10:23am Page ii
Arnold ffirs.tex V4 - 08/02/2007 10:24am Page iii
Professional
Software Testing with
Visual Studio
®
2005 Team System
Arnold ffirs.tex V4 - 08/02/2007 10:24am Page iv
Arnold ffirs.tex V4 - 08/02/2007 10:24am Page v
Professional
Software Testing with
Visual Studio
®
2005 Team System
ToolsforSoftwareDevelopers
andTestEngineers
Tom Arnold
Dominic Hopton
Andy Leonard

Mike Frost
Wiley Publishing, Inc.
Arnold ffirs.tex V4 - 08/02/2007 10:24am Page vi
Professional Software Testing with Visual Studio
®
2005 Team
System: Tools for Software Developers and Test Engineers
Published by
Wiley Publishing, Inc.
10475 Crosspoint Boulevard
Indianapolis, IN 46256
www.wiley.com
Copyright © 2007 by Wiley Publishing, Inc., Indianapolis, Indiana
Published simultaneously in Canada
ISBN: 978-0-470-14978-2
Manufactured in the United States of America
10987654321
Library of Congress Cataloging-in-Publication Data
Professional software testing with Visual Studio 2005 team system :
tools for software developers and test engineers / Tom Arnold [et
al.].
p. cm.
Includes index.
ISBN 978-0-470-14978-2 (paper/website)
1. Computer software–Testing. 2. Microsoft Visual studio. I.
Arnold, Tom, 1967-
QA76.76.T48P76 2007
005.1’4–dc22
2007026308
No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any

means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections
107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or
authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood
Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600. Requests to the Publisher for permission should be
addressed to the Legal Department, Wiley Publishing, Inc., 10475 Crosspoint Blvd., Indianapolis, IN 46256, (317)
572-3447, fax (317) 572-4355, or online at
/>.
Limit of Liability/Disclaimer of Warranty: The publisher and the author make no representations or warranties
with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties,
including without limitation warranties of fitness for a particular purpose. No warranty may be created or extended
by sales or promotional materials. The advice and strategies contained herein may not be suitable for every
situation. This work is sold with the understanding that the publisher is not engaged in rendering legal, accounting,
or other professional services. If professional assistance is required, the services of a competent professional person
should be sought. Neither the publisher nor the author shall be liable for damages arising herefrom. The fact that an
organization or Website is referred to in this work as a citation and/or a potential source of further information
does not mean that the author or the publisher endorses the information the organization or Website may provide
or recommendations it may make. Further, readers should be aware that Internet Websites listed in this work may
have changed or disappeared between when this work was written and when it is read.
For general information on our other products and services please contact our Customer Care Department within the
United States at (800) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002.
Trademarks: Wiley, the Wiley logo, Wrox, the Wrox logo, Wrox Programmer to Programmer, and related trade
dress are trademarks or registered trademarks of John Wiley & Sons, Inc. and/or its affiliates, in the United States
and other countries, and may not be used without written permission. Visual Studio is a registered trademark of
Microsoft Corporation in the United States and/or other countries. All other trademarks are the property of their
respective owners. Wiley Publishing, Inc., is not associated with any product or vendor mentioned in this book.
Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be
available in electronic books.
Arnold ffirs.tex V4 - 08/02/2007 10:24am Page vii
For Siobhan. I love you.
—TomArnold

To all the underappreciated testers in the world.
— Dominic Hopton
To Christy, Stevie Ray, and Emma Grace.
— Andy Leonard
To my family for their love and support all these years,
especially Dad for encouraging my interest in computers.
And to Sarah, who makes my life complete.
— Mike Frost
Arnold ffirs.tex V4 - 08/02/2007 10:24am Page viii
Arnold fauth.tex V4 - 08/02/2007 10:28am Page ix
About the Authors
Tom Arnold was a program manager for Microsoft Visual Studio 2005 Team System, specifically for the
testing tools. He has also managed other commercial testing tools projects by Microsoft and Rational
Software and has spoken at such industry conferences as STAR, Microsoft Tech Ed, Internet World, and
many others. In addition to two other books on software testing, he has produced three video series on
the topic and co-founded a 250-employee software testing company (later sold to Lionbridge/Veritest).
Tom is currently a program manager at Microsoft on the
OfficeLive.com
team. Tom can be reached at

.
Dominic Hopton is a software development engineer on the Visual Studio Enterprise Test team at
Microsoft. One of the original team members, Dominic helped fashion the new development and testing
tools in Visual Studio 2005. He is currently working on the next version of testing tools for Visual Studio.
Andy Leonard is a SQL Server database developer, MVP, and engineer. He is a co-author of
Professional SQL Server 2005 Integration Services. He founded and manages
VSTeamSystemCentral.com
.
Andy’s experience includes web application architecture and development, VB.NET, ASP, and ASP.NET;
SQL Server Integration Services (SSIS); data warehouse development using SQL Server 2000 and 2005;

and test-driven database development.
Mike Frost is a senior ‘‘software design engineer in test’’ (i.e., a software tester who mostly writes code)
at Microsoft. Over the last 10 years he has worked on several releases of Visual Studio with a focus
on the infrastructure and architecture used for testing software. Currently he is working on the new
OfficeLive.com
service product.
Arnold fauth.tex V4 - 08/02/2007 10:28am Page x
Arnold fcredit.tex V4 - 08/02/2007 10:29am Page xi
Credits
Executive Editor
Robert Elliott
Development Editor
Kelly Talbot
Technical Editors
Paul Schafer
Chris Massie
Bill Barnett
Michael Fanning
Production Editor
Debra Banninger
Copy Editor
Cate Caffrey
Editorial Manager
Mary Beth Wakefield
Production Manager
Tim Tate
Vice President and Executive Group Publisher
Richard Swadley
Vice President and Executive Publisher
Joseph B. Wikert

Proofreader
Nancy Carrasco
Indexer
Melanie Belkin
Project Coordinator, Cover
Adrienne Martinez
Anniversary Logo Design
Richard Pacifico
Arnold fcredit.tex V4 - 08/02/2007 10:29am Page xii
Arnold fack.tex V4 - 08/02/2007 10:30am Page xiii
Acknowledgments
Tom Arnold: Thank you to Wiley Press and Wrox for inviting me to participate: Bob Elliott, Kelly Tal-
bot, Debra Banninger, and everyone else. Siobhan Quinn, thank you for joining me at countless coffee
shops while I typed and you laughed at my cheesy jokes. To the Visual Studio team — specifically Sam
Guckenheimer, Jim Sather, Chris Lucas, Gabriel Marius, and my fellow Program Managers — thank
you for the chance to be a part of something that reaches literally millions of developers worldwide. Rob
Arnold, my brother and mentor, thank you for your constant encouragement. My co-authors — Dominic,
Andy, and Mike — thank you guys for signing up! Thank you to our technical editors — Paul, Chris, Bill,
and Michael — for keeping us honest. Mikky Anderson, Bruce Johnson, and Sridhar Chandrashekar,
thank you for supporting my efforts. Thank you, Mom and Dad, for everything.
Dominic Hopton: Thanks have to go to the people who enabled me to understand and learn my product
and be part of it. Without their guidance and encouragement, I wouldn’t have understood enough to
be able to write this book. I won’t remember all your names, but I must thank Tom Marsh, Joe Rohde,
James Su, and David Williamson for nurturing me. I also have to thank Tom Arnold for encouraging me
to do the book with him — and his constant nagging! Finally, I have to thank Joanne Cunningham for
the special and unique encouragement she provided that allowed me to put pen to paper. Thank you.
Andy Leonard: Thanks to Rob Caron, Gert Drapers, Matt Nunn, Sachin Rekhi, and Cameron Skinner
for their blog posts about Visual Studio Team Edition for Database Professionals. I learned a lot about
creating custom test conditions from blog posts and samples by Sachin Rekhi and Gert Drapers. For
help with C# coding, thanks to my friends: John Brown, Craig Dahlinger, Jason Gerard, Frank La Vigne,

and Chris Massie. Many thanks to the author team: Tom, Mike, and Dominic — you guys are awesome!
Thanks to Kelly Talbot, editor and ‘‘coach,’’ and our entire Wrox Team at Wiley: Bob Elliott, Colleen
Hauser,andHelenRusso;tonameafew.WheneverIsignontowrite,sodoesChristy.Thanksforyour
encouragement and inspiration, Beautiful.
Mike Frost: Thanks to Bill Barnett for his technical knowledge and reviewing; to Cullen Waters, Mei-Fang
Huang, and the rest of the Office Live stress testing team for their help and examples; and to Sean Lumley
and Ed Glass for their informative blogs and helpful e-mails.
Arnold fack.tex V4 - 08/02/2007 10:30am Page xiv
Arnold halftitle.tex V4 - 08/02/2007 10:30am Page xv
Professional
Software Testing with
Visual Studio
®
2005 Team System
Arnold halftitle.tex V4 - 08/02/2007 10:30am Page xvi
Arnold ftoc.tex V3 - 08/02/2007 10:31am Page xvii
Contents
Introduction xxiii
Chapter 1: Introduction to VSTEST and VSTESD 1
The Software Development Process and Software Development Life Cycle (SDLC) 1
Planning, Analysis, Design, and Development 1
Existing Tools in the Marketplace 3
Where VSTEST and VSTESD Fit in the SDLC 6
What VSTEST and VSTESD Do 7
VSTEST Features 7
VSTESD Features 10
How Do They Help the SDLC? 11
Why Choose VSTEST Over Other Tool Sets? 12
Summary 13
Chapter 2: A Quick Tour of VSTEST and VSTESD 15

Test Types 15
The Manual Test Type 16
The Unit Test Type 17
The Web Test Type 17
The Load Test Type 18
The Ordered Test Type 18
The Generic Test Type 18
Future/Third-Party Test Types 19
The Test User Interface 19
The UI for Creation and Authoring of Tests 19
The UI for Management of Tests 27
The UI for Test Execution and Results 30
Analysis Tools 37
Code Analysis 37
Performance Measurements 38
Summary 39
Arnold ftoc.tex V3 - 08/02/2007 10:31am Page xviii
Contents
Chapter 3: Unit Testing with VSTEST and VSTESD 41
Overview of Unit Testing 41
Why Unit Test? 41
Approaches to Unit Testing 42
What Unique Unit Testing Features Do VSTEST and VSTESD Have? 43
Creating Your First Unit Test 45
Creating a Sample Library 45
Generating Unit Tests 46
Running Unit Tests 48
A Tour of Unit Testing in VSTEST 48
Anatomy of a Test Class 49
Passing and Failing a Test 49

Code Generation 49
Customizing Your Test Methods 54
Unit Testing Assertions 56
Debugging Unit Tests 57
Advanced Unit Testing 58
MSTest Command-Line Tools 60
Unit Test Life Time 65
ASP.NET 69
Data-Driven Testing 69
Test Context 72
Summary 72
Chapter 4: Testing the Database 73
Database Testing Is Different 73
Developing a Database Project 74
Exploring the Project 75
Adding Schemas, Tables, and Constraints 78
Adding Views and Stored Procedures 83
Inside the Development Database 87
Rename Refactoring 88
Database Project Deployment 90
Generating Test Data 92
Unit Testing 94
Unit Test Stored Procedures 97
Developing Custom Unit Tests 114
Adding the Custom Test Condition Project 114
Building the Custom Test Condition 115
Building the ‘Hook’ 120
xviii
Arnold ftoc.tex V3 - 08/02/2007 10:31am Page xix
Contents

Developing a Unit Test with the Custom Test Condition 123
Summary 124
Chapter 5: Web Testing 125
Basic Web Tests 125
Recording a Test 126
Using the Web Test Editor 127
Components of a Web Test 129
Making Web Tests Dynamic 135
Context Parameters 136
Data-Driven Web Tests 139
Capturing Data During a Web Test 142
Extraction Rules 142
Default (Auto-Captured) Fields 144
Verifying the Response with Validation Rules 144
Built-in Validation Rules 144
Adding a Validation Rule to a Request 145
Validation Level 146
Custom Rules 147
The Response Page 150
Testing a Web Service 150
HTTP Request-Based Testing Does Not Test the UI 151
You Cannot Test Your Web UI and Client-Side Script Using a Web Test 151
Dynamic Test Data That Is Affected by Scripts 152
Running a Web Test 152
Important Settings in the .testrunconfig File 152
Running the Test 153
Examining the Results 154
Reviewing Each Request and Response in the Web Test Viewer 154
Request Panel 154
Details Panel 155

Investigating Failures 155
Maintaining the Tests 157
Coded Web Tests 158
Coded versus Basic 158
Converting from a Web Test 158
Aspects of a Coded Web Test 159
Plug-ins 162
Web Test Plug-ins 162
Web Test Request Plug-ins 162
Plug-ins from Coded Web Tests 162
xix
Arnold ftoc.tex V3 - 08/02/2007 10:31am Page xx
Contents
Web Tests and AJAX 163
Summary 164
Chapter 6: Using Manual, Ordered, and Generic Test Types 165
Manual Test Type 165
Defining a Manual Test Scenario 166
Creating a Manual Test 167
Properties of a Manual Test 170
Executing a Manual Test 173
Ordered Test Type 177
Creating an Ordered Test 177
Properties of an Ordered Test 178
Executing an Ordered Test 181
Generic Test Type 182
Creating a Generic Test 182
Properties of a Generic Test 186
Executing a Generic Test 190
Summary 190

Chapter 7: Load Testing 191
Basic Terminology 191
Working with Load Tests in VSTEST 193
Creating a Load Test 193
Components of a Load Test 195
Scenarios 195
Counter Sets 201
Run Settings 205
SQL Tracing 209
SQL Tracing Specific Run Settings 209
Viewing SQL Trace Data 210
Using a Test Rig 211
Determining Hardware Requirements 211
Setting Up the Test Rig 214
Managing the Test Rig 215
Results Store 219
Creating a Results Store in an Existing SQL Server 219
Results Store Database Schema 220
Best Practices 221
Working from Clear Goals 221
Understanding Your Test Environment 222
xx
Arnold ftoc.tex V3 - 08/02/2007 10:31am Page xxi
Contents
Knowing When to Load Test 223
Walk Before You Run 223
Running a Load Test 225
Important Settings in the .testrunconfig File 225
Starting the Load Test 226
Examining in Progress Test Data 226

Using the Load Test Analyzer 227
Investigating Results 232
Understanding Your Application 233
Using Rules 233
Checking the Test Computers 234
Analyzing a Test Run 235
If There Are No Errors 240
Reporting Results 240
Summary 241
Chapter 8: Using Code Analysis and Dynamic Analysis 243
Code Analysis 243
Static Analysis Tools 243
Coding Standards 244
Enabling Code Analysis 245
Understanding Errors 246
Suppressing Errors 249
Dynamic Analysis 251
Other Tools to Use 251
Profiling 257
Code Coverage 265
Off-Roading on the Command Line 269
Summary 272
Chapter 9: VSTEST and VSTESD within the Software Development
Life Cycle 273
VSTFS at a High Level 273
The Software Development Life Cycle 275
Collaborating with Your Team 276
Installing Team Explorer 276
Connecting to a Project 278
High-Level View of the Project 279

How the VSTEST and VSTESD Tools Fit 283
Team Project Source Control Policies 284
xxi
Arnold ftoc.tex V3 - 08/02/2007 10:31am Page xxii
Contents
Associating Tests with Work Items 294
Team Build Process 297
Associating Work Items with Test Results 300
Publishing Test Results 302
Summary 303
Appendix A: Installing Team Explorer 305
Installing Team Explorer 305
Appendix B: Creating and Running a Web Test: A High-Level
Walk-Through 313
Appendix C: Creating and Running a Unit Test: A High-Level
Walk-Through 319
Appendix D: Creating and Running a Load Test: A High-Level
Walk-Through 327
Appendix E: Creating and Running a Manual Test: A High-Level
Walk-Through 339
Appendix F: Other Sources of Information 347
Author Sites 347
Blogs 347
Forums 348
Conferences 348
Other 348
Index 351
xxii