Model-Based Testing Essentials - Guide to the ISTQB Certified Model-Based Tester E-Book

Anne Kramer

• Herausgeber: John Wiley & Sons
• Kategorie: Wissenschaft und neue Technologien
• Sprache: Englisch
• Veröffentlichungsjahr: 2016

Seitenzahl: 425

Beschreibung

Provides a practical and comprehensive introduction to the key aspects of model-based testing as taught in the ISTQB® Model-Based Tester--Foundation Level Certification Syllabus This book covers the essentials of Model-Based Testing (MBT) needed to pass the ISTQB® Foundation Level Model-Based Tester Certification. The text begins with an introduction to MBT, covering both the benefits and the limitations of MBT. The authors review the various approaches to model-based testing, explaining the fundamental processes in MBT, the different modeling languages used, common good modeling practices, and the typical mistakes and pitfalls. The book explains the specifics of MBT test implementation, the dependencies on modeling and test generation activities, and the steps required to automate the generated test cases. The text discusses the introduction of MBT in a company, presenting metrics to measure success and good practices to apply. * Provides case studies illustrating different approaches to Model-Based Testing * Includes in-text exercises to encourage readers to practice modeling and test generation activities * Contains appendices with solutions to the in-text exercises, a short quiz to test readers, along with additional information Model-Based Testing Essentials - Guide to the ISTQB® Certified Model-Based Tester - Foundation Level is written primarily for participants of the ISTQB® Certification: software engineers, test engineers, software developers, and anybody else involved in software quality assurance. This book can also be used for anyone who wants a deeper understanding of software testing and of the use of models for test generation.

Leseprobe

TABLE OF CONTENTS

TITLE PAGE

COPYRIGHT

DEDICATION

FOREWORD BY GUALTIERO BAZZANA

FOREWORD BY ROBERT V. BINDER

PREFACE

THE ISTQB CERTIFIED TESTER FOUNDATION LEVEL – MODEL-BASED TESTER

HOW THIS BOOK IS ORGANIZED?

ACKNOWLEDGMENTS

1 INTRODUCTION TO MODEL-BASED TESTING

1.1 WHY DO WE NEED NEW APPROACHES TO TESTING?

1.2 WHAT IS MODEL-BASED TESTING?

1.3 BENEFITS OF MBT

1.4 PITFALLS OF MBT

1.5 WHAT CAN YOU REALISTICALLY EXPECT?

2 WHAT YOU SHOULD KNOW ABOUT MBT BEFORE STARTING

2.1 ISTQB MBT GLOSSARY TERMS USED IN THIS BOOK

2.2 OTHER TERMS TO KNOW

2.3 THE MODELING LANGUAGES USED IN THIS BOOK

3 PROCESS ASPECTS OF MBT

3.1 MBT AND THE FUNDAMENTAL TEST PROCESS

3.2 THE TYPICAL MBT PROCESS

3.3 MBT AND SOFTWARE DEVELOPMENT LIFECYCLES

3.4 HOW MBT SUPPORTS REQUIREMENT ENGINEERING

4 ASPECTS TO CONSIDER BEFORE YOU START WRITING AN MBT MODEL

4.1 PRELIMINARY CONSIDERATIONS ON MBT MODELING

4.2 SUBJECT AND FOCUS OF YOUR MBT MODEL

4.3 THE INFLUENCE OF TEST OBJECTIVES ON MBT MODELS

5 MODELING LANGUAGES – THE AGONY OF CHOICE

5.1 MAIN CATEGORIES OF MODELING LANGUAGES

5.2 UML AND BPMN

5.3 OTHER GRAPHICAL MODELING LANGUAGES USED FOR MBT

5.4 TEXTUAL MODELING LANGUAGES USED FOR MBT

5.5 HOW TO SELECT THE APPROPRIATE MODELING LANGUAGE

6 GOOD MBT MODELING PRACTICES

6.1 QUALITY CHARACTERISTICS FOR MBT MODELS

6.2 TYPICAL MISTAKES AND PITFALLS IN MBT MODEL DESIGN

6.3 LINKING REQUIREMENTS AND PROCESS-RELATED INFORMATION TO THE MBT MODEL

6.4 THE SIGNIFICANCE OF MODELING GUIDELINES FOR MBT

6.5 THE QUESTION OF REUSING MODELS FROM OTHER DEVELOPMENT ACTIVITIES

6.6 TOOL SUPPORT FOR MBT MODELING ACTIVITIES

6.7 ITERATIVE MBT MODEL DEVELOPMENT

6.8 OTHER RECOMMENDATIONS

7 HOW MBT RELATES TO TEST DESIGN TECHNIQUES?

7.1 EQUIVALENCE PARTITIONING AND BOUNDARY VALUE ANALYSIS

7.2 DECISION TABLES

7.3 STATE TRANSITION TESTING

7.4 USE CASE TESTING

8 DERIVING TESTS FROM AN MBT MODEL

8.1 TAXONOMY OF SELECTION CRITERIA

8.2 TEST CASE SELECTION IN PRACTICE

8.3 EXAMPLES OF COVERAGE CRITERIA

8.4 PROS AND CONS OF SPECIFIC TEST SELECTION CRITERIA

8.5 SOME RECOMMENDATIONS REGARDING TEST CASE SELECTION

8.6 DEGREE OF AUTOMATION IN TEST GENERATION

9 EXECUTING MODEL-BASED TESTS

9.1 UNDERSTANDING THE CONCEPTS

9.2 ADAPTING TEST CASES FOR AUTOMATED EXECUTION

9.3 ADAPTING MBT ARTIFACTS DUE TO CHANGES

10 INTRODUCING MBT IN YOUR COMPANY

10.1 FIVE STEPS TO MBT ADOPTION

10.2 RETURN-ON-INVEST CONSIDERATIONS

10.3 PRIORITIZE YOUR ORGANIZATIONAL OBJECTIVES

10.4 HOW TO MEASURE PROGRESS AND SUCCESS?

10.5 DEPLOYING MBT

10.6 INITIAL AND RUNNING COSTS OF MBT

10.7 INTEGRATING THE TOOLS

11 CASE STUDIES

11.1 ENTERPRISE IT MODEL-BASED TESTING – ORANGEHRM CASE STUDY

11.2 MBT FOR PROCESS-SUPPORTING SW – TOOL VALIDATION CASE STUDY

11.3 MBT FOR SECURITY COMPONENTS – PKCS#11 CASE STUDY

12 CONCLUSIONS

APPENDIX A SOLUTIONS OF EXERCISES

APPENDIX B TEST YOURSELF

APPENDIX C TAXONOMY OF MBT APPROACHES

ABBREVIATIONS

TERMS AND DEFINITIONS

ISTQB Terms

Other Terms

REFERENCES

INDEX

End User License Agreement

Pages

xi

xii

xiii

xiv

xv

xvi

xvii

xviii

xix

xx

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

268

269

270

271

272

273

274

275

276

Guide

Cover

Table of Contents

Preface

Begin Reading

List of Illustrations

PREFACE

Figure 1 Expectations of MBT practitioners

Figure 2 ISTQB MBT syllabus – content overview.

1 INTRODUCTION TO MODEL-BASED TESTING

Figure 1.1 Test of a normal working day with some variations (activity diagram; Note: Throughout this book, we use the simplified modeling languages defined in the ISTQB MBT syllabus [13] (see Section 3.3). However, you should be aware that this is only one out of many possible notations. Depending on the selected modeling language, notation paradigm, and tool support, the same MBT model may look different).

Figure 1.2 A well-known application (activity diagram).

Figure 1.3 Test levels reported by MBT practitioners

Figure 1.4 Application domains reported by MBT practitioners

Figure 1.5 Testing types reported by MBT practitioners

Figure 1.6 MBT model describing the measurement modes of a multimeter (activity diagram; Note: In this figure, we highlight the action where the verification takes place (rounded gray rectangle)).

Figure 1.7 Example of an MBT model with bug (state diagram of a CD player; Note: In this figure, we use a different modeling language. Rounded rectangles represent states instead of actions (see Section 3.3)).

Figure 1.8 MBT model for testing the data transfer in a chat room (state diagram).

Figure 1.9 Limiting test case explosion by combining the model with external test data (state diagram).

Figure 1.10 The Gartner “Hype Cycle” for technological innovations [12].

2 WHAT YOU SHOULD KNOW ABOUT MBT BEFORE STARTING

Figure 2.1 South African road signs.

Figure 2.2 Example of a textual model, using the formal modeling language B.

Figure 2.3 A decision table as MBT model element.

Figure 2.4 Example of a diagrammatic model describing the use cases of electric shutters (use case diagram).

Figure 2.5 Internal states of an electric shutter (state diagram).

Figure 2.6 Model elements of a simple graphical modeling language for workflows (activity diagram).

Figure 2.7 MBT model describing the game “musical chairs” (activity diagram).

Figure 2.8 Simple graphical modeling language for state diagrams.

Figure 2.9 E-mail states (state diagram).

3 PROCESS ASPECTS OF MBT

Figure 3.1 ISTQB fundamental test process.

Figure 3.2 Workflow with typical MBT-specific activities.

Figure 3.3 Artifacts and roles in a typical MBT process.

Figure 3.4 MBT process with automated test execution.

Figure 3.5 Logical expressions excluding inconsistent paths (activity diagram).

Figure 3.6 Manual and automated test scripts for calculator.

Figure 3.7 Subdiagram “Select operation (+−*/=)” with manual instructions and code snippet (activity diagram).

Figure 3.8 Generated artifacts reported by MBT practitioners (from 2014 MBT User Survey).

Figure 3.9 End-to-end testing of a system of systems.

Figure 3.10 Model-based testing in the Scrum process.

6

Figure 3.11 MBT artifacts in an agile development lifecycle.

Figure 3.12 Six of what?

Figure 3.13 Relationship between requirements and test repositories.

4 ASPECTS TO CONSIDER BEFORE YOU START WRITING AN MBT MODEL

Figure 4.1 MBT model for a fuel dispenser's credit card authentication process (activity diagram).

Figure 4.2 Testing a fuel dispenser with credit card payment (activity diagram).

Figure 4.3 Handling of technology-dependent aspects in the adaptation layer.

Figure 4.4 Example of an MBT model with optimization potential (activity diagram).

Figure 4.5 Improved model (activity diagram – part 1).

Figure 4.6 Improved model (activity diagram – part 2).

Figure 4.7 A system model of a cruise control system written in a state transition modeling language.

Figure 4.8 Interface of the cruise control system.

Figure 4.9 Different usage profiles for occasional and addicted players (activity diagram).

Figure 4.10 State diagram for car wipers (behavioral MBT model with subject “system”).

Figure 4.11 Activity diagram describing the wipers test (behavioral with mixed subjects).

Figure 4.12 MBT model top-level diagram of famous tile-matching game.

5 MODELING LANGUAGES – THE AGONY OF CHOICE

Figure 5.1 Package diagram of a party (example of a model focusing on structural aspects).

Figure 5.2 Music band state diagram (example of a behavioral model).

Figure 5.3 The music band states translated to an activity diagram.

Figure 5.4 Classification tree of a washing machine.

Figure 5.5 Examples of respecting and disrespecting the split/merge rule of UML.

Figure 5.6 Counterexample with correct UML being easier to read.

Figure 5.7 The 14 different diagram types of UML 2.2 4.

Figure 5.8 Example of a structural model describing interfaces (UML component diagram).

Figure 5.9 Example of a data model (class diagram).

Figure 5.10 Example of a behavioral model (BMPN business process model).

Figure 5.11 Markov chain – diagrammatic representation.

Figure 5.12 Markov chain usage model of a coffee machine.

Figure 5.13 Event-flow graph for the print functionality of a well-known text editor.

Figure 5.14 Cause–effect graph example.

Figure 5.15 An example of programming code for MBT in C#.

Figure 5.16 Example of TTCN-3 code.

Figure 5.17 Feeding the cats.

6 GOOD MBT MODELING PRACTICES

Figure 6.1 MBT model with obvious “bug” (semantic error in an activity diagram).

Figure 6.2 Model with a simple syntactic error (activity diagram).

Figure 6.3 Behavioral model with deadlock (state diagram).

Figure 6.4 Hierarchical model of a calendar application (activity diagram).

Figure 6.5 Three possibilities to link requirements to model elements.

Figure 6.6 Effort distribution for selected disciplines according to Ref. [34].

Figure 6.7 RUP in practice.

Figure 6.8 MBT model before and after changes due to project constraints (activity diagrams).

Figure 6.9 Special naming convention for preconditions (example using an activity diagram).

Figure 6.10 Independent models for system and test design.

Figure 6.11 Details of the “Repair electric shutters” use case (activity diagram).

Figure 6.12 Difference between analysis or design models and MBT models

Figure 6.13 Reuse of models from analysis and design

7 HOW MBT RELATES TO TEST DESIGN TECHNIQUES?

Figure 7.1 Taking age restrictions for movies into account (equivalence partitions in an activity diagram; we assume that 120 years is a reasonable upper limit for the spectators age).

Figure 7.2 MBT model containing concrete data values (boundary values in an activity diagram).

Figure 7.3 Decision table for movie ticket price.

Figure 7.4 MBT model combining a behavioral diagram with a decision table for movie ticket price (If you start wondering about the content, please do the exercise first).

Figure 7.5 Smart card lifecycle according to the GlobalPlatform™ card specification (state diagram).

Figure 7.6 Use cases for an automated teller machine (ATM).

Figure 7.7 Activity diagram for ATM cash withdrawal use case (example).

8 DERIVING TESTS FROM AN MBT MODEL

Figure 8.1 Requirements in an MBT model for a sophisticated TV (state diagram; Note: we deliberately neglected the state “Programmed” between “Stopped” and “Recording” to improve readability).

Figure 8.2 Coverage items for structural model coverage.

Figure 8.3 Example diagram for structural model coverage (activity diagram).

Figure 8.4 Equivalence partitioning and boundary value analysis for the spectator's age.

Figure 8.5 Data items set outside the model.

Figure 8.6 Random test case selection (usage profile mapped on an activity diagram).

Figure 8.7 Example of scenario-based test case selection (state diagram and sequence chart).

Figure 8.8 Example of a test pattern.

Figure 8.9 Example model including information on the duration (activity diagram).

Figure 8.10 MBT model for the Microsoft calculator with all subdiagrams (activity diagram).

Figure 8.11 MBT model for the Microsoft calculator avoiding test case explosion (activity diagram).

Figure 8.12 Business process model describing the loan decision process of a bank.

Figure 8.13 Updated business process model describing the bank's loan decision process.

Figure 8.14 States of a rechargeable battery (state diagram).

Figure 8.15 Data domains of the car configurator.

Figure 8.16 Excerpt of the test cases generated for the car configurator with pairwise test selection.

Figure 8.17 MBT pseudocode example.

Figure 8.18 States on an article during an auction (state diagram).

Figure 8.19 Possible combinations of test selection criteria [13].

9 EXECUTING MODEL-BASED TESTS

Figure 9.1 MBT model for a simplified consumer credit application (activity diagram).

Figure 9.2 Model-based testing generated test case (in the open-source test management tool TestLink).

Figure 9.3 Artifacts, roles, and tool in the MBT process for automated test execution.

Figure 9.4 Online model-based testing.

Figure 9.5 Combination of offline and online MBT.

Figure 9.6 Three test adaptation approaches

Figure 9.7 MBT artifacts for test automation.

Figure 9.8 MBT for test execution automation – reuse of existing keyword implementations.

Figure 9.9 Three dimensions of changes.

Figure 9.10 Change of a subdiagram seen in its context (activity diagrams).

Figure 9.11 Change bars in Microsoft Word.

Figure 9.12 Impact of one edge (indicated by the bold gray line) on test case selection.

10 INTRODUCING MBT IN YOUR COMPANY

Figure 10.1 Maslow's hierarchy of needs [48].

Figure 10.2 Some possible measures for metrics in the MBT context.

Figure 10.3 A typical tool chain embedding MBT [13].

Figure 10.4 MBT tools in their context.

11 CASE STUDIES

Figure 11.1 The main OrangeHRM use cases considered in this case study.

Figure 11.2 MBT process for OrangeHRM.

Figure 11.3 Requirements for the recruitment module (excerpt).

Figure 11.4 Top-level diagram of the recruitment process MBT model.

Figure 11.5 MBT model for subprocess “Process Job Application.”

Figure 11.6 MBT model for subprocess “Complete Hiring Process.”

Figure 11.7 Details of a generated test case after publication into the test management tool.

Figure 11.8 Set of executed tests for OrangeHRM as displayed in the test management tool.

Figure 11.9 Use cases for “BugEx” before discussing it with the stakeholders.

Figure 11.10 Use cases for “BugEx” after discussion with stakeholders.

Figure 11.11 MBT approach of the tool validation case study.

Figure 11.12 The “BugEx” state diagram.

Figure 11.13 Actions (TestStep) and checks (verification point, VP) for the state NEW (activity diagram).

Figure 11.14 Actions (TestStep) and checks (verification point, VP) for the state OPEN.

Figure 11.15 Actions (TestStep) and checks (verification point, VP) for the state CLOSED.

Figure 11.16 Bug in the test procedure for state VERIFIED.

Figure 11.17 Actions (TestStep) and checks (verification point, VP) for the state FIXED.

Figure 11.18 General high-level environment of security components.

Figure 11.19 MBT process used in the PKCS#11 case study.

Figure 11.20 SoftHSM 2.0.0b2 test environment.

Figure 11.21 PKCS#11 MBT model – simplified class diagram.

Figure 11.22 OCL postcondition for C_SignInit.

Figure 11.23 Example of function description, C_SignInit.

Figure 11.24 Excerpt of the PKCS#11 initial state definition.

Figure 11.25 Test case view.

Figure 11.26 HTML export of generated test case.

Figure 11.27 Simple abstract test case for PKCS#11.

Figure 11.28 Concrete C++ tests for SoftHSM.

Figure 11.29 Excerpt of PKCS11 test adaptation layer.

12 CONCLUSIONS

Figure 12.1 Fulfillment of expectations

Figure 12.2 Overall rating by MBT practitioners

APPENDIX A SOLUTIONS OF EXERCISES

Figure A.1 Solution of Exercise 1 (test cases 1–6).

Figure A.3 Solution of Exercise 1 (test cases 13–18).

Figure A.4 Solution of Exercise 5.

Figure A.5 Solution of Exercise 6.

Figure A.6 Solution of Exercise 16.

Figure A.7 Solution of Exercise 18.

APPENDIX C TAXONOMY OF MBT APPROACHES

Figure C.1 Taxonomy of basic MBT characteristics.

List of Tables

1 INTRODUCTION TO MODEL-BASED TESTING

Table 1.1 Our First Test Case Generated from a Model

Table 1.2 Our First Test Case with Detailed Test Instructions

Table 1.3 Number of Possible Variations of the Messages in the Chat Room

5 MODELING LANGUAGES – THE AGONY OF CHOICE

Table 5.1 Markov Chain – Square Array Representation

Table 5.2 Examples of Domain-Specific Aspects Requiring Support by the Modeling Language

Table 5.3 Examples of Test-Specific Aspects Influencing the Selection of the Modeling Language

8 DERIVING TESTS FROM AN MBT MODEL

Table 8.1 Number of Generated Test Cases with Pattern-based Test Selection

Table 8.2 Examples of Test Selection Criteria Applied on Business Process Models or Activity Diagrams

Table 8.3 Examples of Test Selection Criteria Applied on State Diagrams

Table 8.4 Examples of Test Selection Criteria Applied on Textual Models

9 EXECUTING MODEL-BASED TESTS

Table 9.1 Abstract Test Case – Consumer Credit Accepted for a Registered Customer

Table 9.2 Concrete Test Case – Consumer Credit Accepted for a Registered Customer

Table 9.3 Test Description for the Activity “Define Credit Characteristics”

Table 9.4 Test Adaptation in Different MBT Approaches

10 INTRODUCING MBT IN YOUR COMPANY

Table 10.1 Initial and Running Costs, as Specified in Ref. [13]

11 CASE STUDIES

Table 11.1 Decision Table Associated with Task “Record Interview Result”

Table 11.2 Mapping Between Abstract and Concrete Data Values

Table 11.3 Test Description of

selectInterviewResult

Table 11.4 Test Procedure Specification for Test Case “Not a Bug”

Table 11.5 Test Objective Charter Example for PKCS#11

APPENDIX C TAXONOMY OF MBT APPROACHES

Table C 1 Mapping Between the Classification of Basic MBT Characteristics and the Sections of This Book

MODEL-BASED TESTING ESSENTIALS

Guide to the ISTQB® Certified Model-Based Tester Foundation Level

Anne Kramer

Bruno Legeard

Copyright © 2016 by John Wiley & Sons, Inc. All rights reserved

Published by John Wiley & Sons, Inc., Hoboken, New Jersey\hb

Published simultaneously in Canada

All names of products and companies used in this book may be trademarks of their corresponding owners. In particular, the following terms and abbreviations are registered trademarks: BPMN™, Hewlett-Packard®, HP®, ISTQB®, IQBBA®, iSQI®, MATLAB Simulink®, Microsoft® Windows®, Model Driven Architecture®, MOF®, Object Management Group®, OMG®, RUP®, Sparx® Systems Enterprise Architect®, Stateflow®, UML®, Unified Modeling Language™, XMI®.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, [phone], fax [phone], or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, [phone], fax [phone], or online at http://www.wiley.com/go/permission.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.

For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at [phone], outside the United States at [phone] or fax [phone].

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic formats. For more information about Wiley products, visit our web site at www.wiley.com.

Library of Congress Cataloging-in-Publication Data

Names: Kramer, Anne, (Software engineer) author. | Legeard, Bruno, author.

Title: Model-based testing essentials : guide to the ISTQB certified

model-based tester foundation level / Anne Kramer, Bruno Legeard.

Description: Hoboken, New Jersey : John Wiley & Sons Inc., [2016] | Includes

bibliographical references and index.

Identifiers: LCCN 2015035751 | ISBN 9781119130017 (cloth)

Subjects: LCSH: Computer software--Testing--Examinations--Study guides. |

Model-based reasoning--Examinations--Study guides. | International

Software Testing Qualifications Board--Examinations--Study guides. |

Electronic data processing personnel--Certification.

Classification: LCC QA76.76.T48 K73 2016 | DDC 005.3028/7--dc23 LC record available at http://lccn.loc.gov/2015035751