Anthony williams c++ concurrency in action

MANNING
Anthony Williams
Practical Multithreading
IN ACTION
C++ Concurrency in Action

C++ Concurrency
in Action
P
RACTICAL
M
ULTITHREADING
ANTHONY WILLIAMS
MANNING
S
HELTER
I
SLAND
For online information and ordering of this and other Manning books, please visit
www.manning.com. The publisher offers discounts on this book when ordered in quantity.
For more information, please contact
Special Sales Department
Manning Publications Co.
20 Baldwin Road
PO Box 261
Shelter Island, NY 11964
Email:
©2012 by Manning Publications Co. All rights reserved.
No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in
any form or by means electronic, mechanical, photocopying, or otherwise, without prior written
permission of the publisher.

Many of the designations used by manufacturers and sellers to distinguish their products are
claimed as trademarks. Where those designations appear in the book, and Manning
Publications was aware of a trademark claim, the designations have been printed in initial caps
or all caps.
Recognizing the importance of preserving what has been written, it is Manning’s policy to have
the books we publish printed on acid-free paper, and we exert our best efforts to that end.
Recognizing also our responsibility to conserve the resources of our planet, Manning books
are printed on paper that is at least 15 percent recycled and processed without the use of
elemental chlorine.
Manning Publications Co. Development editor: Cynthia Kane
20 Baldwin Road Technical proofreader: Jonathan Wakely
PO Box 261 Copyeditor: Linda Recktenwald
Shelter Island, NY 11964 Proofreader: Katie Tennant
Typesetter: Dennis Dalinnik
Cover designer: Marija Tudor
ISBN: 9781933988771
Printed in the United States of America
1 2 3 4 5 6 7 8 9 10 – MAL – 18 17 16 15 14 13 12
To Kim, Hugh, and Erin


vii
brief contents
1
■
Hello, world of concurrency in C++! 1
2
■
Managing threads 15
3

■
Sharing data between threads 33
4
■
Synchronizing concurrent operations 67
5
■
The C++ memory model and operations on atomic types 103
6
■
Designing lock-based concurrent data structures 148
7
■
Designing lock-free concurrent data structures 180
8
■
Designing concurrent code 224
9
■
Advanced thread management 273
10
■
Testing and debugging multithreaded applications 300


ix
contents
preface xv
acknowledgments xvii
about this book xix

about the cover illustration xxii
1
Hello, world of concurrency in C++! 1
1.1 What is concurrency? 2
Concurrency in computer systems 2
Approaches to concurrency 4
1.2 Why use concurrency? 6
Using concurrency for separation of concerns 6
Using concurrency for performance 7
■
When not
to use concurrency 8
1.3 Concurrency and multithreading in C++ 9
History of multithreading in C++ 10
■
Concurrency support
in the new standard 10
■
Efficiency in the C++
Thread Library 11
■
Platform-specific facilities 12
1.4 Getting started 13
Hello, Concurrent World 13
1.5 Summary 14

CONTENTS
x
2
Managing threads 15

2.1 Basic thread management 16
Launching a thread 16
■
Waiting for a thread to complete 18
Waiting in exceptional circumstances 19
■
Running threads
in the background 21
2.2 Passing arguments to a thread function 23
2.3 Transferring ownership of a thread 25
2.4 Choosing the number of threads at runtime 28
2.5 Identifying threads 31
2.6 Summary 32
3
Sharing data between threads 33
3.1 Problems with sharing data between threads 34
Race conditions 35
■
Avoiding problematic race conditions 36
3.2 Protecting shared data with mutexes 37
Using mutexes in C++ 38
■
Structuring code for protecting
shared data 39
■
Spotting race conditions inherent
in interfaces 40
■
Deadlock: the problem and a solution 47
Further guidelines for avoiding deadlock 49

■
Flexible locking
with std::unique_lock 54
■
Transferring mutex ownership
between scopes 55
■
Locking at an appropriate granularity 57
3.3 Alternative facilities for protecting shared data 59
Protecting shared data during initialization 59
■
Protecting rarely
updated data structures 63
■
Recursive locking 64
3.4 Summary 65
4
Synchronizing concurrent operations 67
4.1 Waiting for an event or other condition 68
Waiting for a condition with condition variables 69
Building a thread-safe queue with condition variables 71
4.2 Waiting for one-off events with futures 76
Returning values from background tasks 77
■
Associating a task
with a future 79
■
Making (std::)promises 81
■
Saving an

exception for the future 83
■
Waiting from multiple threads 85
4.3 Waiting with a time limit 87
Clocks 87
■
Durations 88
■
Time points 89
Functions that accept timeouts 91

CONTENTS
xi
4.4 Using synchronization of operations to simplify code 93
Functional programming with futures 93
■
Synchronizing
operations with message passing 97
4.5 Summary 102
5
The C++ memory model and operations on atomic types 103
5.1 Memory model basics 104
Objects and memory locations 104
■
Objects, memory locations,
and concurrency 105
■
Modification orders 106
5.2 Atomic operations and types in C++ 107
The standard atomic types 107

■
Operations on
std::atomic_flag 110
■
Operations on std::atomic<bool> 112
Operations on std::atomic<T*>: pointer arithmetic 114
Operations on standard atomic integral types 116
The std::atomic<> primary class template 116
■
Free functions
for atomic operations 117
5.3 Synchronizing operations and enforcing ordering 119
The synchronizes-with relationship 121
■
The happens-before
relationship 122
■
Memory ordering for atomic operations 123
Release sequences and synchronizes-with 141
■
Fences 143
Ordering nonatomic operations with atomics 145
5.4 Summary 147
6
Designing lock-based concurrent data structures 148
6.1 What does it mean to design for concurrency? 149
Guidelines for designing data structures for concurrency 149
6.2 Lock-based concurrent data structures 151
A thread-safe stack using locks 151
■

A thread-safe queue using
locks and condition variables 154
■
A thread-safe queue using
fine-grained locks and condition variables 158
6.3 Designing more complex lock-based data structures 169
Writing a thread-safe lookup table using locks 169
■
Writing a
thread-safe list using locks 175
6.4 Summary 179
7
Designing lock-free concurrent data structures 180
7.1 Definitions and consequences 181
Types of nonblocking data structures 181
■
Lock-free
data structures 182
■
Wait-free data structures 182
The pros and cons of lock-free data structures 183

CONTENTS
xii
7.2 Examples of lock-free data structures 184
Writing a thread-safe stack without locks 184
■
Stopping those
pesky leaks: managing memory in lock-free data structures 188
Detecting nodes that can’t be reclaimed using hazard pointers 193

Detecting nodes in use with reference counting 200
■
Applying the
memory model to the lock-free stack 205
■
Writing a thread-safe
queue without locks 209
7.3 Guidelines for writing lock-free data structures 221
Guideline: use std::memory_order_seq_cst for prototyping 221
Guideline: use a lock-free memory reclamation scheme 221
Guideline: watch out for the ABA problem 222
Guideline: identify busy-wait loops and help the other thread 222
7.4 Summary 223
8
Designing concurrent code 224
8.1 Techniques for dividing work between threads 225
Dividing data between threads before processing begins 226
Dividing data recursively 227
■
Dividing work by task type 231
8.2 Factors affecting the performance of concurrent code 233
How many processors? 234
■
Data contention and cache
ping-pong 235
■
False sharing 237
■
How close is
your data? 238

■
Oversubscription and excessive
task switching 239
8.3 Designing data structures for multithreaded
performance 239
Dividing array elements for complex operations 240
Data access patterns in other data structures 242
8.4 Additional considerations when designing for
concurrency 243
Exception safety in parallel algorithms 243
■
Scalability and
Amdahl’s law 250
■
Hiding latency with multiple threads 252
Improving responsiveness with concurrency 253
8.5 Designing concurrent code in practice 255
A parallel implementation of std::for_each 255
■
A parallel
implementation of std::find 257
■
A parallel implementation
of std::partial_sum 263
8.6 Summary 272

CONTENTS
xiii
9
Advanced thread management 273

9.1 Thread pools 274
The simplest possible thread pool 274
■
Waiting for tasks
submitted to a thread pool 276
■
Tasks that wait for other
tasks 280
■
Avoiding contention on the work queue 283
Work stealing 284
9.2 Interrupting threads 289
Launching and interrupting another thread 289
■
Detecting that
a thread has been interrupted 291
■
Interrupting a condition
variable wait 291
■
Interrupting a wait on
std::condition_variable_any 294
■
Interrupting other
blocking calls 296
■
Handling interruptions 297
Interrupting background tasks on application exit 298
9.3 Summary 299
10

Testing and debugging multithreaded applications 300
10.1 Types of concurrency-related bugs 301
Unwanted blocking 301
■
Race conditions 302
10.2 Techniques for locating concurrency-related bugs 303
Reviewing code to locate potential bugs 303
Locating concurrency-related bugs by testing 305
Designing for testability 307
■
Multithreaded testing
techniques 308
■
Structuring multithreaded test code 311
Testing the performance of multithreaded code 314
10.3 Summary 314
appendix A Brief reference for some C++11 language features 315
appendix B Brief comparison of concurrency libraries 340
appendix C A message-passing framework and complete ATM example 342
appendix D C++ Thread Library reference 360
resources 487
index 489


xv
preface
I encountered the concept of multithreaded code while working at my first job after I
left college. We were writing a data processing application that had to populate a data-
base with incoming data records. There was a lot of data, but each record was inde-
pendent and required a reasonable amount of processing before it could be inserted

into the database. To take full advantage of the power of our
10-CPU
Ultra
SPARC
, we
ran the code in multiple threads, each thread processing its own set of incoming
records. We wrote the code in C++, using
POSIX
threads, and made a fair number of
mistakes—multithreading was new to all of us—but we got there in the end. It was also
while working on this project that I first became aware of the C++ Standards Commit-
tee and the freshly published C++ Standard.
I have had a keen interest in multithreading and concurrency ever since. Where
others saw it as difficult, complex, and a source of problems, I saw it as a powerful tool
that could enable your code to take advantage of the available hardware to run faster.
Later on I would learn how it could be used to improve the responsiveness and perfor-
mance of applications even on single-core hardware, by using multiple threads to hide
the latency of time-consuming operations such as
I/O
. I also learned how it worked at
the
OS
level and how Intel
CPU
s handled task switching.
Meanwhile, my interest in C++ brought me in contact with the
ACCU
and then the
C++ Standards panel at
BSI

, as well as Boost. I followed the initial development of
the Boost Thread Library with interest, and when it was abandoned by the original
developer, I jumped at the chance to get involved. I have been the primary developer
and maintainer of the Boost Thread Library ever since.

PREFACE
xvi
As the work of the C++ Standards Committee shifted from fixing defects in the exist-
ing standard to writing proposals for the next standard (named C++0x in the hope
that it would be finished by 2009, and now officially C++11, because it was finally pub-
lished in 2011), I got more involved with
BSI
and started drafting proposals of my own.
Once it became clear that multithreading was on the agenda, I jumped in with both
feet and authored or coauthored many of the multithreading and concurrency-
related proposals that shaped this part of the new standard. I feel privileged to have
had the opportunity to combine two of my major computer-related interests—C++
and multithreading—in this way.
This book draws on all my experience with both C++ and multithreading and aims
to teach other C++ developers how to use the C++11 Thread Library safely and effi-
ciently. I also hope to impart some of my enthusiasm for the subject along the way.

xvii
acknowledgments
I will start by saying a big “Thank you” to my wife, Kim, for all the love and support she
has given me while writing this book. It has occupied a significant part of my spare
time for the last four years, and without her patience, support, and understanding, I
couldn’t have managed it.
Second, I would like to thank the team at Manning who have made this book possi-
ble: Marjan Bace, publisher; Michael Stephens, associate publisher; Cynthia Kane, my

development editor; Karen Tegtmeyer, review editor; Linda Recktenwald, my copy-
editor; Katie Tennant, my proofreader; and Mary Piergies, the production manager.
Without their efforts you would not be reading this book right now.
I would also like to thank the other members of the C++ Standards Committee
who wrote committee papers on the multithreading facilities: Andrei Alexandrescu,
Pete Becker, Bob Blainer, Hans Boehm, Beman Dawes, Lawrence Crowl, Peter Dimov,
Jeff Garland, Kevlin Henney, Howard Hinnant, Ben Hutchings, Jan Kristofferson, Doug
Lea, Paul McKenney, Nick McLaren, Clark Nelson, Bill Pugh, Raul Silvera, Herb Sutter,
Detlef Vollmann, and Michael Wong, plus all those who commented on the papers, dis-
cussed them at the committee meetings, and otherwise helped shaped the multithread-
ing and concurrency support in C++11.
Finally, I would like to thank the following people, whose suggestions have greatly
improved this book: Dr. Jamie Allsop, Peter Dimov, Howard Hinnant, Rick Molloy,
Jonathan Wakely, and Dr. Russel Winder, with special thanks to Russel for his detailed
reviews and to Jonathan who, as technical proofreader, painstakingly checked all the
content for outright errors in the final manuscript during production. (Any remaining

ACKNOWLEDGMENTS
xviii
mistakes are of course all mine.) In addition I’d like to thank my panel of reviewers:
Ryan Stephens, Neil Horlock, John Taylor Jr., Ezra Jivan, Joshua Heyer, Keith S. Kim,
Michele Galli, Mike Tian-Jian Jiang, David Strong, Roger Orr, Wagner Rick, Mike Buksas,
and Bas Vodde. Also, thanks to the readers of the MEAP edition who took the time to
point out errors or highlight areas that needed clarifying.

xix
about this book
This book is an in-depth guide to the concurrency and multithreading facilities from the
new C++ Standard, from the basic usage of
std::thread

,
std::mutex
, and
std::async
,
to the complexities of atomic operations and the memory model.
Roadmap
The first four chapters introduce the various library facilities provided by the library
and show how they can be used.
Chapter 5 covers the low-level nitty-gritty of the memory model and atomic opera-
tions, including how atomic operations can be used to impose ordering constraints on
other code, and marks the end of the introductory chapters.
Chapters 6 and 7 start the coverage of higher-level topics, with some examples of
how to use the basic facilities to build more complex data structures—lock-based data
structures in chapter 6, and lock-free data structures in chapter 7.
Chapter 8 continues the higher-level topics, with guidelines for designing multi-
threaded code, coverage of the issues that affect performance, and example imple-
mentations of various parallel algorithms.
Chapter 9 covers thread management—thread pools, work queues, and interrupt-
ing operations.
Chapter 10 covers testing and debugging—types of bugs, techniques for locating
them, how to test for them, and so forth.
The appendixes include a brief description of some of the new language facili-
ties introduced with the new standard that are relevant to multithreading, the

ABOUT THIS BOOK
xx
implementation details of the message-passing library mentioned in chapter 4, and a
complete reference to the C++11 Thread Library.
Who should read this book

If you're writing multithreaded code in C++, you should read this book. If you're using
the new multithreading facilities from the C++ Standard Library, this book is an essen-
tial guide. If you’re using alternative thread libraries, the guidelines and techniques
from the later chapters should still prove useful.
A good working knowledge of C++ is assumed, though familiarity with the new lan-
guage features is not—these are covered in appendix A. Prior knowledge or experience
of multithreaded programming is not assumed, though it may be useful.
How to use this book
If you’ve never written multithreaded code before, I suggest reading this book sequen-
tially from beginning to end, though possibly skipping the more detailed parts of
chapter 5. Chapter 7 relies heavily on the material in chapter 5, so if you skipped chap-
ter 5, you should save chapter 7 until you’ve read it.
If you’ve not used the new C++11 language facilities before, it might be worth
skimming appendix A before you start to ensure that you’re up to speed with the
examples in the book. The uses of the new language facilities are highlighted in
the text, though, and you can always flip to the appendix if you encounter something
you’ve not seen before.
If you have extensive experience with writing multithreaded code in other environ-
ments, the beginning chapters are probably still worth skimming so you can see how
the facilities you know map onto the new standard C++ ones. If you’re going to be
doing any low-level work with atomic variables, chapter 5 is a must. Chapter 8 is worth
reviewing to ensure that you’re familiar with things like exception safety in multi-
threaded C++. If you have a particular task in mind, the index and table of contents
should help you find a relevant section quickly.
Once you’re up to speed on the use of the C++ Thread Library, appendix D should
continue to be useful, such as for looking up the exact details of each class and func-
tion call. You may also like to dip back into the main chapters from time to time to
refresh your use of a particular construct or look at the sample code.
Code conventions and downloads
All source code in listings or in text is in a

fixed-width

font

like

this
to separate it
from ordinary text. Code annotations accompany many of the listings, highlighting
important concepts. In some cases, numbered bullets link to explanations that follow
the listing.
Source code for all working examples in this book is available for download from
the publisher’s website at www.manning.com/
CP
lusPlusConcurrencyinAction.

ABOUT THIS BOOK
xxi
Software requirements
To use the code from this book unchanged, you’ll need a recent C++ compiler that
supports the new C++11 language features used in the examples (see appendix A),
and you’ll need a copy of the C++ Standard Thread Library.
At the time of writing, g++ is the only compiler I’m aware of that ships with an
implementation of the Standard Thread Library, although the Microsoft Visual Studio
2011 preview also includes an implementation. The g++ implementation of the
Thread Library was first introduced in a basic form in g++ 4.3 and extended in subse-
quent releases. g++ 4.3 also introduced the first support for some of the new C++11
language features; more of the new language features are supported in each subse-
quent release. See the g++ C++11 status page for details.
1

Microsoft Visual Studio 2010 provides some of the new C++11 language features,
such as rvalue references and lambda functions, but doesn't ship with an implementa-
tion of the Thread Library.
My company, Just Software Solutions Ltd, sells a complete implementation of the
C++11 Standard Thread Library for Microsoft Visual Studio 2005, Microsoft Visual
Studio 2008, Microsoft Visual Studio 2010, and various versions of g++.
2
This imple-
mentation has been used for testing the examples in this book.
The Boost Thread Library
3
provides an
API
that’s based on the C++11 Standard
Thread Library proposals and is portable to many platforms. Most of the examples
from the book can be modified to work with the Boost Thread Library by judicious
replacement of
std::
with
boost::
and use of the appropriate
#include
directives.
There are a few facilities that are either not supported (such as
std::async
) or have
different names (such as
boost::unique_future
) in the Boost Thread Library.
Author Online

Purchase of C++ Concurrency in Action includes free access to a private web forum run by
Manning Publications where you can make comments about the book, ask technical ques-
tions, and receive help from the author and from other users. To access the forum and
subscribe to it, point your web browser to www.manning.com/
CP
lusPlusConcurrencyin-
Action. This page provides information on how to get on the forum once you’re regis-
tered, what kind of help is available, and the rules of conduct on the forum.
Manning’s commitment to our readers is to provide a venue where a meaningful
dialogue between individual readers and between readers and the author can take
place. It’s not a commitment to any specific amount of participation on the part of the
author, whose contribution to the book’s forum remains voluntary (and unpaid). We
suggest you try asking the author some challenging questions, lest his interest stray!
The Author Online forum and the archives of previous discussions will be accessi-
ble from the publisher’s website as long as the book is in print.
1
GNU Compiler Collection C++0x/C++11 status page, />2
The
just::thread
implementation of the C++ Standard Thread Library, .
3
The Boost C++ library collection, .

xxii
about the cover illustration
The illustration on the cover of C++ Concurrency in Action is captioned “Habit of a
Lady of Japan.” The image is taken from the four-volume Collection of the Dress of
Different Nations by Thomas Jefferys, published in London between 1757 and 1772. The
collection includes beautiful hand-colored copperplate engravings of costumes from
around the world and has influenced theatrical costume design since its publication.

The diversity of the drawings in the compendium speaks vividly of the richness of the
costumes presented on the London stage over 200 years ago. The costumes, both his-
torical and contemporaneous, offered a glimpse into the dress customs of people liv-
ing in different times and in different countries, making them come alive for London
theater audiences.
Dress codes have changed in the last century and the diversity by region, so rich in
the past, has faded away. It’s now often hard to tell the inhabitant of one continent
from another. Perhaps, trying to view it optimistically, we’ve traded a cultural and
visual diversity for a more varied personal life—or a more varied and interesting intel-
lectual and technical life.
We at Manning celebrate the inventiveness, the initiative, and the fun of the com-
puter business with book covers based on the rich diversity of regional and theatrical
life of two centuries ago, brought back to life by the pictures from this collection.

1
Hello, world of
concurrency in C++!
These are exciting times for C++ users. Thirteen years after the original C++ Stan-
dard was published in 1998, the C++ Standards Committee is giving the language
and its supporting library a major overhaul. The new C++ Standard (referred to as
C++11 or C++0x) was published in 2011 and brings with it a whole swathe of
changes that will make working with C++ easier and more productive.
One of the most significant new features in the C++11 Standard is the support of
multithreaded programs. For the first time, the C++ Standard will acknowledge the
existence of multithreaded applications in the language and provide components in
the library for writing multithreaded applications. This will make it possible to write
This chapter covers
■
What is meant by concurrency and
multithreading

■
Why you might want to use concurrency and
multithreading in your applications
■
Some of the history of the support for
concurrency in C++
■
What a simple multithreaded C++ program
looks like

2
C
HAPTER
1
Hello, world of concurrency in C++!
multithreaded C++ programs without relying on platform-specific extensions and thus
allow writing portable multithreaded code with guaranteed behavior. It also comes at a
time when programmers are increasingly looking to concurrency in general, and multi-
threaded programming in particular, to improve application performance.
This book is about writing programs in C++ using multiple threads for concur-
rency and the C++ language features and library facilities that make that possible. I’ll
start by explaining what I mean by concurrency and multithreading and why you
would want to use concurrency in your applications. After a quick detour into why
you might not want to use it in your applications, I’ll give an overview of the concur-
rency support in C++, and I’ll round off this chapter with a simple example of C++
concurrency in action. Readers experienced with developing multithreaded applica-
tions may wish to skip the early sections. In subsequent chapters I’ll cover more
extensive examples and look at the library facilities in more depth. The book will fin-
ish with an in-depth reference to all the C++ Standard Library facilities for multi-
threading and concurrency.

So, what do I mean by concurrency and multithreading?
1.1 What is concurrency?
At the simplest and most basic level, concurrency is about two or more separate activi-
ties happening at the same time. We encounter concurrency as a natural part of life;
we can walk and talk at the same time or perform different actions with each hand,
and of course we each go about our lives independently of each other—you can watch
football while I go swimming, and so on.
1.1.1 Concurrency in computer systems
When we talk about concurrency in terms of computers, we mean a single system per-
forming multiple independent activities in parallel, rather than sequentially, or one
after the other. It isn’t a new phenomenon: multitasking operating systems that allow
a single computer to run multiple applications at the same time through task switch-
ing have been commonplace for many years, and high-end server machines with mul-
tiple processors that enable genuine concurrency have been available for even longer.
What is new is the increased prevalence of computers that can genuinely run multiple
tasks in parallel rather than just giving the illusion of doing so.
Historically, most computers have had one processor, with a single processing
unit or core, and this remains true for many desktop machines today. Such a
machine can really only perform one task at a time, but it can switch between tasks
many times per second. By doing a bit of one task and then a bit of another and so
on, it appears that the tasks are happening concurrently. This is called task switching.
We still talk about concurrency with such systems; because the task switches are so fast,
you can’t tell at which point a task may be suspended as the processor switches to
another one. The task switching provides an illusion of concurrency to both the user
and the applications themselves. Because there is only an illusion of concurrency, the