Preface p. xv
Acknowledgments p. xvii
About the Author p. xix
1 Introduction to Software Testing Management p. 1
1.1 Product Defect: A Case Study p. 2
1.2 Case Analysis p. 3
1.3 Return on Investment p. 4
1.4 Causes of Defects in Software p. 5
1.5 Factors Creating Problems in Software Development Process p. 7
1.6 Solutions to Software Development Problems p. 8
1.7 Definition of Software Quality p. 9
1.8 Definition of Good Design p. 9
1.9 Definition of Good Code p. 10
1.10 Definition of Testing p. 11
1.10.1 Case Study p. 11
1.10.1.1 Inputs p. 11
1.10.1.2 Outputs p. 12
1.10.1.3 Analysis p. 12
1.10.1.4 Conclusion p. 12
1.11 Software Testing Evolution p. 13
1.12 Software Engineering p. 14
1.13 Software Testing Methodologies p. 15
1.14 Tools p. 15
1.14.1 Test Case Execution Automation p. 15
1.14.2 Test Coverage p. 16
1.14.3 Defect Tracking p. 16
1.14.4 Test Management p. 16
1.15 Project Offshoring p. 16
1.16 Testing Being Commoditized? p. 17
1.17 Conclusion p. 17
2 Kinds of Software Testing Projects p. 19
2.1 Software Testing Types p. 19
2.2 What Needs to Be Tested? p. 23
2.3 Enterprise System Testing p. 24
2.4 Enterprise System Types p. 25
2.4.1 Banking, Finance, Insurance, and Securities (BFSI) Systems p. 25
2.4.2 Enterprise Resource Planning (ERP) Systems p. 25
2.5 Desktop System Testing p. 26
2.6 Device Driver System Testing p. 26
2.7 Stage of Software Life Cycle p. 27
2.8 Outsourced Software Testing p. 27
2.8.1 Software Vendor Perspective p. 27
2.8.2 Software Service Provider Perspective p. 28
2.9 Performance Testing p. 28
2.10 Security Testing p. 29
3 Software Testing Project Strategies p. 31
3.1 Strategy versus Planning p. 32
3.2 Complexity Management p. 32
3.2.1 Case Study on Complexity Management p. 33
3.2.1.1 Module Description p. 34
3.2.1.2 Strategy p. 34
3.2.1.3 Complexity p. 34
3.2.1.4 Problems p. 35
3.2.1.5 Solution p. 35
3.2.1.6 Pseudo Logic p. 35
3.3 Technology Management p. 39
3.4 People Managemenr p. 40
3.5 Skills Required p. 40
3.6 Risk Factors p. 41
3.6.1 Technological Risks p. 41
3.6.2 Scheduling Risk p. 42
3.6.3 Human Risks p. 42
3.7 Strategy for Automation p. 42
3.8 Strategy for Manual Testing p. 43
3.9 Automation Tool Selection p. 43
3.10 Strategy for Creating Automation Framework p. 43
3.11 Software Product Life Cycle and Automation p. 44
3.12 Test Case Prioritization p. 46
3.13 Ad Hoc Testing p. 46
3.14 Software Test Project Strategy Checklist p. 46
3.15 Challenges p. 47
3.16 Conclusion p. 47
4 Project Effort Estimation p. 49
4.1 Estimation by Experience p. 50
4.2 Estimation Using Test Point Analysis p. 51
4.2.1 Basic Components of TPA p. 51
4.2.1.1 ProjectSize p. 51
4.2.1.2 Test Strategy p. 52
4.2.1.3 Productivity p. 52
4.2.2 TPA Calculation Details p. 53
4.2.2.1 Dynamic Test Point Characteristics p. 53
4.2.2.2 Dynamic Quality Characteristics p. 54
4.2.2.3 Test Point Calculation p. 55
4.2.2.4 Productivity Factor p. 55
4.2.2.5 Primary Test Hours p. 56
4.2.2.6 Total Number of Test Hours p. 56
4.2.2.7 Phase Breakdown p. 57
4.2.3 Application of TPA p. 57
4.2.4 TPA at Bidding Stage p. 57
4.3 TPA-Based Effort Estimation Implementation p. 58
4.3.1 Identify Test Life Cycle Stages p. 58
4.3.2 Identify Activities for Each Phase p. 59
4.3.3 Size Estimation for Each Phase p. 59
4.3.3.1 Test Requirement Phase p. 59
4.3.3.2 Test Case Design Phase p. 60
4.3.3.3 Test Script Development Phase p. 60
4.3.3.4 Test Case Execution Phase p. 61
4.3.3.5 Regression Phase p. 61
4.3.4 Effort Estimation for Each Phase p. 61
4.3.5 Synopsis p. 62
4.4 Importance of Effort Estimation p. 63
4.5 Practical Advice p. 63
4.6 Schedule versus Effort p. 64
4.7 Task Elasticity p. 64
4.8 Effort Estimation Checklist p. 66
4.8.1 Checklist for Software Test Schedule Estimation p. 66
4.8.2 Checklist for Software Test Effort Estimation p. 66
4.9 Challenges p. 67
5 Software Testing Project Plan p. 69
5.1 Test Area Prioritization p. 70
5.2 Skill Matching p. 70
5.3 Resource Allocation p. 70
5.4 Tools Selection p. 70
5.5 Methodology Selection p. 71
5.6 Sample Project p. 71
5.6.1 Technology p. 72
5.6.2 Infrastructure p. 72
5.6.3 Testing Resources p. 72
5.6.4 Testing Process p. 73
5.6.5 Test Automation p. 74
5.6.5.1 Steps for Test Case Automation p. 74
5.7 Automation Framework p. 75
5.8 Test Data Management p. 75
5.8.1 Data Maintenance in Production Environment p. 75
5.9 Testing without a Formal Test Plan p. 76
5.9.1 The Drawbacks p. 77
5.10 Software Test Plan Checklist p. 78
5.10.1 Test Plan Checklist-Analysis and Review p. 78
5.10.2 Test Plan Checklist-Testing Activities p. 78
5.10.3 Test Plan Checklist-Test Environment p. 78
5.10.4 Test Plan Checklist-Organizarion p. 79
5.10.5 Test Plan Checklist-Test Schedule p. 79
5.10.6 Test Plan Checklist-Test Tools p. 79.
5.10.7 Test Plan Checklist-Configuration Management p. 80
5.10.8 Test Plan Checklist-Test Metrics p. 80
5.10.9 Test Plan Checklist-Project Tracking for Unit/Integration Testing p. 80
5.10.10 Test Plan Checklist-Project Tracking for Acceptance Testing p. 81
5.10.11 Test Plan Checklist-Project Tracking for System Testing p. 81
6 Software Testing Project Risk Management p. 83
6.1 Risk Measurement Method p. 84
6.1.1 Create a Scale and Assign a Score to Each Risk p. 85
6.1.2 Count Number of Times the Risk Occurs in the Project p. 85
6.1.3 Risk Analysis Case Study p. 85
6.2 Risks Related to the Application Being Tested p. 86
6.3 Kinds of Risks p. 87
6.3.1 Communication Risks p. 87
6.3.2 Effectiveness p. 87
6.3.3 Cultural Risks p. 88
6.3.4 Process Risks p. 88
6.3.5 Size p. 88
6.4 Challenges p. 88
6.5 Checklist for Risk Management p. 89
7 Software Testing Project Execution p. 91
7.1 Earned Value Management p. 92
7.1.1 Need for EVM p. 93
7.1.2 EVM Implementation for Software Projects p. 94
7.1.3 Audit Trail p. 95
7.2 Defect Tracking and Life Cycle p. 95
7.3 Monitoring of Production Systems p. 96
7.4 Test Case Execution p. 97
7.5 Checklist for Test Execution p. 97
8 Software Testing Project Reporting p. 99
8.1 Importance of Reporting p. 100
8.1.1 What Should Go in a Report p. 100
8.1.2 Case Study p. 100
8.2 Test Report p. 101
8.2.1 Test Report Components p. 102
8.2.1.1 Acceptance Criteria p. 103
8.2.1.2 Accessibility Testing p. 103
8.2.1.3 Status Report p. 103
8.2.1.4 Blocked Test Cases p. 103
8.2.1.5 Boundary Value Coverage p. 103
8.2.1.6 TestCharrer p. 103
8.2.1.7 Test Bed p. 104
8.2.1.8 Test Basis p. 104
8.2.1.9 Test Approach p. 104
8.3 Test Metrics p. 104
8.3.1 Metrics and Reports p. 105
Bibliography p. 106
9 Automated Software Testing Benefits p. 107
9.1 Considerations for Automation p. 109
9.1.1 Analysis of Test Case Activities p. 109
9.1.2 Financial and Execution Time Impacts p. 110
9.1.3 Workload Factor p. 110
9.2 Test Automation History p. 111
9.3 Case Studies p. 112
9.3.1 Business Case 1 p. 112
9.3.1.1 Wrong Customer Expectations p. 112
9.3.1.2 Problem Statement p. 112
9.3.1.3 IssuesFaced p. 112
9.3.1.4 Solution p. 113
9.3.1.5 Lessons Learned p. 113
9.3.2 Business Case 2 p. 113
9.3.2.1 Automation Strategy p. 113
9.3.2.2 Issues Faced p. 114
9.3.2.3 Solution p. 114
9.3.2.4 Lessons Learned p. 114
9.3.3 Business Case 3 p. 115
9.3.3.1 Tool Selection p. 115
9.3.3.2 Solution p. 115
9.3.3.3 Lessons Learned p. 115
9.3.4 Business Case 4 p. 115
9.3.4.1 Problems p. 115
9.3.4.2 Solution p. 116
9.3.4.3 Lesson Learned p. 116
9.3.5 Business Case 5 p. 116
9.3.5.1 Test Estimation p. 116
9.3.5.2 IssuesFaced p. 116
9.3.5.3 Solution p. 116
9.3.5.4 Lessons Learned p. 117
9.3.6 Business Case 6 p. 117
9.3.6.1 Technical Issue p. 117
9.3.6.2 Solution p. 117
9.3.6.3 Lessons Learned p. 117
9.3.7 Business Case 7 p. 118
9.3.7.1 New Technology p. 118
9.3.7.2 IssuesFaced p. 118
9.3.7.3 Solution p. 118
9.4 Keyword-Driven Automation Framework p. 118
9.4.1 Steps for Creating Keyword Framework p. 119
9.5 Data-Driven Automated Testing p. 120
10 Customer Expectation Management p. 121
10.1 Difficult Proposition p. 121
10.2 Service Level Agreement p. 122
10.3 Product Development p. 123
10.4 History p. 123
10.5 Challenges p. 125
10.6 Requirement Analysis p. 125
10.7 Project Process Information p. 126
10.8 Case Study; Electronics Retailer p. 127
10.9 Customer Expectation Management Strategies p. 127
10.9.1 Customer Involvement p. 127
10.9.2 KickoffMeeting p. 128
10.9.3 Get Approval for Delivery Methodology p. 128
10.9.4 Communicate Risks Early p. 129
10.9.5 Commit Less and Deliver More p. 129
10.9.6 Be Cool and Share Lighter Moments p. 129
10.9.7 Stick to SLAs p. 129
Bibliography p. 130
11 Software Testing Practice and Offshoring p. 131
11.1 Challenges p. 133
11.1.1 Customer Concerns p. 133
11.1.1.1 Commercial Concerns p. 133
11.1.1.2 Technical and Process Concerns p. 134
11.1.1.3 Legal Concerns p. 135
11.2 Benefits of Offshoring p. 136
11.2.1 Tradirional Approach to Testing p. 137
11.2.2 Cost of Late Detection of Defects p. 137
11.3 Proposed Organization Structure p. 138
11.3.1 Process Structure p. 139
11.3.2 Projecr Components p. 139
11.3.3 Infrasrructure p. 139
11.3.4 Tools p. 139
11.3.5 Operating Model p. 140
11.3.6 Organization p. 140
11.4 Software Testing Consolidation p. 140
11.5 Advantages of Offshoring Coupled with Centralized Quality Group p. 141
11.5.1 Offshore Team Srructure p. 141
11.6 Case Study p. 141
11.6.1 The Current State Scenario p. 142
11.6.2 The Future State Scenario p. 142
11.6.3 A 3-Year Labor Cost ROI Scenario p. 143
11.6.4 Challenge p. 143
11.6.5 Solution p. 143
11.6.6 New Model Implementation Details p. 144
11.6.7 Pilot Project Details p. 144
11.6.8 Process Model p. 145
11.6.9 Benefits of New Model p. 146
11.6.9.1 Productivity p. 146
11.6.9.2 Quality p. 146
11.6.9.3 Cost p. 147
11.6.10 Automation Benefits p. 147
11.6.10.1 Key Highlights p. 149
11.6.10.2 Sanity Testing p. 149
12 Software Testing as a Commodity p. 151
12.1 Software Offshoring p. 153
12.2 The New Concept p. 153
12.2.1 Old Organization Structure p. 153
12.2.2 New Organization Structure p. 154
12.3 Linking Test Project with Development Project p. 157
12.4 Mass Manufacturing p. 158
12.4.1 Setup p. 159
12.4.2 Idle Time p. 160
12.4.3 Sequencing p. 160
12.4.4 Disassembling p. 160
12.4.5 Aggregation p. 162
12.4.6 Pegging p. 162
12.5 Project versus Manufacturing p. 164
12.5.1 Projecr p. 164
12.5.2 Manufacturing p. 164
12.5.3 Mass Servicing of Software Test Projects p. 165
12.5.4 Rework Aspects p. 165
12.6 Conclusion p. 165
13 Quality, Standards, and Software Testing p. 167
13.1 Quality p. 167
13.2 Standards p. 168
13.2.1 Benchmarking p. 168
13.2.2 Six Sigma p. 369
13.3 Software Development Process Quality p. 169
13.3.1 Standards for Software Development Processes p. 172
13.3.1.1 Requirements p. 172
13.3.1.2 Software Project Planning p. 173
13.3.1.3 Software Project Tracking and Oversight p. 173
13.3.1.4 Software Subcontract Management p. 173
13.3.1.5 Software Quality Assurance p. 174
13.4 Software Testing and Quality p. 174
13.5 Quality Standards Evolution p. 175
Bibliography p. 175
Appendix I IEEE Standard for Software Test Documentation (ANSI/IEEE Standard 829-1983) p. 177
Appendix II Software Testing Glossary p. 181
Index p. 213