Description
Book SynopsisPresents a comprehensive treatment of issues related to the inception, design, implementation and reporting of large-scale education assessments.
In recent years many countries have decided to become involved in international educational assessments to allow them to ascertain the strengths and weaknesses of their student populations. Assessments such as the OECD''s Programme for International Student Assessment (PISA), the IEA''s Trends in Mathematics and Science Study (TIMSS) and Progress in International Reading Literacy (PIRLS) have provided opportunities for comparison between students of different countries on a common international scale.
This book is designed to give researchers, policy makers and practitioners a well-grounded knowledge in the design, implementation, analysis and reporting of international assessments. Readers will be able to gain a more detailed insight into the scientific principles employed in such studies allowing them to make better
Table of Contents
Notes on Contributors xv
Foreword xvii
Acknowledgements xx
Abbreviations xxi
1 Implementation of Large]Scale Education Assessments 1
Petra Lietz, John C. Cresswell, Keith F. Rust and Raymond J. Adams
1.1 Introduction 1
1.2 International, Regional and National Assessment Programmes in Education 3
1.3 Purposes of LSAs in Education 4
1.4 Key Areas for the Implementation of LSAs in Education 10
1.5 Summary and Outlook 16
Appendix 1.A 18
References 22
2 Test Design and Objectives 26
Dara Ramalingam
2.1 Introduction 26
2.2 PISA 27
2.3 TIMSS 34
2.4 PIRLS and Pre]PIRLS 41
2.5 ASER 45
2.6 SACMEQ 52
2.7 Conclusion 56
References 58
3 Test Development 63
Juliette Mendelovits
3.1 Introduction 63
3.2 Developing an Assessment Framework: A Collaborative and Iterative Process 65
3.3 Generating and Collecting Test Material 68
3.4 Refinement of Test Material 72
3.5 Beyond Professional Test Development: External Qualitative Review of Test Material 81
3.6 Introducing Innovation 86
3.7 Conclusion 90
References 90
4 Design, Development and Implementation of Contextual Questionnaires in Large]Scale Assessments 92
Petra Lietz
4.1 Introduction 92
4.2 The Role of Questionnaires in LSAs 93
4.3 Steps in Questionnaire Design and Implementation 95
4.4 Questions and Response Options in LSAs 115
4.5 Alternative Item Formats 119
4.6 Computer]Based/Online Questionnaire Instruments 128
4.7 Conclusion and Future Perspectives 131
Acknowledgements 132
References 132
5 Sample Design, Weighting, and Calculation of Sampling Variance 137
Keith F. Rust, Sheila Krawchuk and Christian Monseur
5.1 Introduction 137
5.2 Target Population 138
5.3 Sample Design 144
5.4 Weighting 148
5.5 Sampling Adjudication Standards 153
5.6 Estimation of Sampling Variance 156
References 166
6 Translation and Cultural Appropriateness of Survey Material in Large]Scale Assessments 168
Steve Dept, Andrea Ferrari and Béatrice Halleux
6.1 Introduction 168
6.2 Overview of Translation/Adaptation and Verification Approaches Used in Current Multilingual Comparative Surveys 169
6.3 Step]by]Step Breakdown of a Sophisticated Localisation Design 174
6.4 Measuring the Benefits of a Good Localisation Design 184
6.5 Checklist of Requirements for a Robust Localisation Design 190
References 191
7 Quality Assurance 193
John C. Cresswell
7.1 Introduction 193
7.2 The Development and Agreement of Standardised Implementation Procedures 194
7.3 The Production of Manuals which Reflect Agreed Procedures 196
7.4 The Recruitment and Training of Personnel in Administration and Organisation: Especially the Test Administrator and the School Coordinator 197
7.5 The Quality Monitoring Processes: Recruiting and Training Quality Monitors to Visit National Centres and Schools 198
7.6 Other Quality Monitoring Procedures 201
7.7 Conclusion 204
Reference 204
8 Processing Responses to Open]Ended Survey Questions 205
Ross Turner
8.1 Introduction 205
8.2 The Fundamental Objective 207
8.3 Contextual Factors: Survey Respondents and Items 207
8.4 Administration of the Coding Process 214
8.5 Quality Assurance and Control: Ensuring Consistent and Reliable Coding 221
8.6 Conclusion 229
References 229
9 Computer]Based Delivery of Cognitive Assessment and Questionnaires 231
Maurice Walker
9.1 Introduction 231
9.2 Why Implement Computer]Based Assessments? 232
9.3 Implementation of International Comparative Computer]Based Assessments 238
9.4 Assessment Architecture 244
9.5 Item Design Issues 247
9.6 State]of]the]Art and Emerging Technologies 250
9.7 Summary and Conclusion 250
References 251
10 Data Management Procedures 253
Falk Brese and Mark Cockle
10.1 Introduction 253
10.2 Historical Review: From Data Entry and Data Cleaning to Integration into the Entire Study Process 254
10.3 The Life Cycle of a LSA Study 255
10.4 Standards for Data Management 256
10.5 The Data Management Process 258
10.6 Outlook 272
References 274
11 Test Implementation in the Field: The Case of PASEC 276
Oswald Koussihouèdé, Antoine Marivin and Vanessa Sy
11.1 Introduction 276
11.2 Test Implementation 278
11.3 Data Entry 294
11.4 Data Cleaning 295
11.5 Data Analysis 295
11.6 Governance and Financial Management of the Assessments 295
Acknowledgments 296
References 297
12 Test Implementation in the Field: The Experience of Chile in International Large]Scale Assessments 298
Ema Lagos Campos
12.1 Introduction 298
12.2 International Studies in Chile 302
Annex A 321
References 321
13 Why Large]Scale Assessments Use Scaling and Item Response Theory 323
Alla Berezner and Raymond J. Adams
13.1 Introduction 323
13.2 Item Response Theory 325
13.3 Test Development and Construct Validation 329
13.4 Rotated Test Booklets 345
13.5 Comparability of Scales Across Settings and Over Time 347
13.6 Construction of Performance Indicators 349
13.7 Conclusion 354
References 354
14 Describing Learning Growth 357
Ross Turner and Raymond J. Adams
14.1 Background 357
14.2 Terminology: The Elements of a Learning Metric 358
14.3 Example of a Learning Metric 360
14.4 Issues for Consideration 360
14.5 PISA Described Proficiency Scales 365
14.6 Defining and Interpreting Proficiency Levels 374
14.7 Use of Learning Metrics 379
Acknowledgement 380
References 381
15 Scaling of Questionnaire Data in International Large]Scale Assessments 384
Wolfram Schulz
15.1 Introduction 384
15.2 Methodologies for Construct Validation and Scaling 386
15.3 Classical Item Analysis 387
15.4 Exploratory Factor Analysis 388
15.5 Confirmatory Factor Analysis 389
15.6 IRT Scaling 392
15.7 Described IRT Questionnaire Scales 396
15.8 Deriving Composite Measures of Socio]economic Status 399
15.9 Conclusion and Future Perspectives 404
References 405
16 Database Production for Large]Scale Educational Assessments 411
Eveline Gebhardt and Alla Berezner
16.1 Introduction 411
16.2 Data Collection 412
16.3 Cleaning, Recoding and Scaling 416
16.4 Database Construction 418
16.5 Assistance 421
References 423
17 Dissemination and Reporting 424
John C. Cresswell
17.1 Introduction 424
17.2 Frameworks 425
17.3 Sample Items 426
17.4 Questionnaires 427
17.5 Video 427
17.6 Regional and International Reports 428
17.7 National Reports 428
17.8 Thematic Reports 429
17.9 Summary Reports 429
17.10 Analytical Services and Support 430
17.11 Policy Papers 430
17.12 Web]Based Interactive Display 431
17.13 Capacity]Building Workshops 432
17.14 Manuals 432
17.15 Technical Reports 432
17.16 Conclusion 433
References 433
Index 436
