Python 3 object oriented programming unleash the power of python 3 objects 2nd edition

Python 3 Object-oriented
Programming
Second Edition

Unleash the power of Python 3 objects

Dusty Phillips

BIRMINGHAM - MUMBAI


Python 3 Object-oriented Programming
Second Edition

Copyright © 2015 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval
system, or transmitted in any form or by any means, without the prior written
permission of the publisher, except in the case of brief quotations embedded in
critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy
of the information presented. However, the information contained in this book is
sold without warranty, either express or implied. Neither the author, nor Packt
Publishing, and its dealers and distributors will be held liable for any damages
caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the
companies and products mentioned in this book by the appropriate use of capitals.
However, Packt Publishing cannot guarantee the accuracy of this information.

First published: July 2010

Second edition: August 2015

Production reference: 1130815

Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham B3 2PB, UK.
ISBN 978-1-78439-878-1
www.packtpub.com


Credits
Author
Dusty Phillips
Reviewers
AMahdy AbdElAziz
Grigoriy Beziuk
Krishna Bharadwaj
Justin Cano
Anthony Petitbois
Claudio Rodriguez
Commissioning Editor
Edward Gordon

Copy Editor
Janbal Dharmaraj
Project Coordinator
Neha Bhatnagar
Proofreader

Safis Editing
Indexer
Hemangini Bari
Graphics
Sheetal Aute
Jason Monteiro

Acquisition Editors
Indrajit Das
Rebecca Pedley
Greg Wild
Content Development Editors
Divij Kotian
Arvind Koul
Anila Vincent
Technical Editor
Siddhi Rane

Production Coordinator
Komal Ramchandani
Cover Work
Komal Ramchandani


About the Author
Dusty Phillips is a Canadian software developer and author currently living in

Seattle, Washington. He has been active in the open source community for a decade
and a half and programming in Python for nearly all of it. He cofounded the popular
Puget Sound Programming Python meetup group; drop by and say hi if you're in

the area.
Python 3 Object Oriented Programming, Packt Publishing, was the first of his books. He
has also written Creating Apps In Kivy, O'Reilly, the mobile Python library, and selfpublished Hacking Happy, a journey to mental wellness for the technically inclined.
He was hospitalized for suicidal tendencies shortly after the first edition of this book
was published and has been an outspoken proponent for positive mental health
ever since.


About the Reviewers
AMahdy AbdElAziz has more than 8 years of experience in software engineering

using several languages and frameworks. Over the last 5 years, he has focused
on Android and mobile development, including cross-platform tools, and
Android internals, such as building custom ROMs and customizing AOSP
for embedded devices.

He is currently teaching Python at Information Technology Institution. You can
visit his website, to find out more about him.

Grigoriy Beziuk is a former CIO of Crowdage Foundation, acting as an

independent software developer as this book was being written. He has worked
with a wide variety of programming languages and technologies, including
different versions of Python in different environments, ranging from purely
scientific ones to modern production-scale web development issues.
I would like to thank my mom, Larisa Beziuk, for giving me the
gift of life; all of my teachers and friends for making my life more
interesting; and all of my beloved ones, former and present for…
well, everything.

Krishna Bharadwaj is the cofounder of SMERGERS (rgers.

com/), a Fintech start-up helping small and medium businesses raise capital from
investors and different financial institutions. In the past, he has worked with early
stage start-ups such as BlockBeacon (Santa Monica) and PricePoint (CA) and large
organizations such as National Instruments, Bangalore, and Google, New York.
Krishna got introduced to Python and FOSS during his college days and has continued
to use it extensively in his personal projects and also professionally at work. Because
of his liking for teaching and mentoring, he visits different universities, conducting
workshops whenever he gets an opportunity.

He holds a master's degree in computer science from the University of Southern
California, Los Angeles, and a bachelor's degree in information science and
engineering from the BMS College of Engineering, Bangalore. He can be reached
through his e-mail, , or his website,
o/.

Justin Cano is a recent graduate from the University of California, Riverside,
with a BS in computer engineering and is currently working as a software engineer
in the Silicon Valley area with hopes of moving to a big tech company such as
Google or Apple.
He first started programming in the sixth grade, creating small, primitive websites
in HTML and CSS. He started to learn computer science theory and C++ in his first
year at UC Riverside and then started learning Python in his third year.
Justin admits that at first, he wasn't immediately attracted to Python, since abstractions
between C++ and Python are very different. It wasn't until he began to express more of
an interest in coding contests and challenges that he began to show interest in Python,
mainly because he feels that the readability and elegance of the Python syntax allows
him to quickly and more naturally turn ideas and thought processes into Python

code. He now writes Python code regularly, often to create mock-ups or prototypes of
software applications before moving on to a more domain-specific language.
I would like to thank the author for taking the time to write this book
as I have received a lot of valuable insights and information on the
Python language and design patterns. This book has strengthened
my understanding of Python, and I believe that I am now a more
knowledgeable Python programmer.


Anthony Petitbois is an online architect in the video game industry with 13 years
of professional experience in operations and development and more than 20 years
of software development experience. He is passionate about new technologies and
loves to take creative approaches to solve complex problems.
In his spare time, he learns new languages and new platforms, plays video games,
and spends time with his family in the beautiful region of British Columbia, Canada,
where he now lives after emigrating from France in 2009.

Claudio Rodriguez started working on PLCs for GE, but his main goal has always
been research and development and turning dreams into reality. This made him
move from automation engineering to software engineering and the structured way
of software, OOD; the remote team working from the comfort of his computer was
just too powerful not to take advantage of. During his master's, he got to learn the
proper place to look for resources and found a friend in books and research papers
and conferences. Eventually, he started working on a system to control an electric
arc furnace, but the needs of his clients moved him into taking further control of
technology. He has a deep love for complex AI and can be seen surrounded by
papers, books, and a computer to test things, but he keeps things real by delivering
beautiful and dynamic applications for his customers.

www.PacktPub.com
Support files, eBooks, discount offers, and more

For support files and downloads related to your book, please visit www.PacktPub.com.
Did you know that Packt offers eBook versions of every book published, with PDF
and ePub files available? You can upgrade to the eBook version at www.PacktPub.
com and as a print book customer, you are entitled to a discount on the eBook copy.
Get in touch with us at for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign
up for a range of free newsletters and receive exclusive discounts and offers on
Packt books and eBooks.
TM

/>
Do you need instant solutions to your IT questions? PacktLib is Packt's online digital
book library. Here, you can search, access, and read Packt's entire library of books.

Why subscribe?

• Fully searchable across every book published by Packt
• Copy and paste, print, and bookmark content
• On demand and accessible via a web browser

Free access for Packt account holders

If you have an account with Packt at www.PacktPub.com, you can use this to access
PacktLib today and view 9 entirely free books. Simply use your login credentials for
immediate access.

Introduction to
the second edition
I have a confession to make. When I wrote the first edition of this book, I didn't
have a clue what I was doing. I thought I knew Python and I thought I knew how
to write. I quickly learned that this was false. Luckily, I became adept at both by
finishing the book!
I was so afraid that people wouldn't like Python 3 Object Oriented Programming that
I skipped Pycon for two years straight. After a couple dozen positive reviews, my
confidence was boosted and I finally attended Pycon 2012 in Santa Clara. I soon
discovered that nobody had ever heard of me or my book. So much for arrogance!
I was also afraid to reread the book after completing it. So while it has received many
accolades, the copy on my shelf has remained firmly shut, save for when I open it for
reference to answer a reader's query. In preparing this second edition, I was finally
forced to face my demons. To my surprise and joy, I discovered that the book I wrote
five years ago was both accurate and enjoyable, just as many reviewers had suggested.
Shortly after that initial rereading, I got my first ever negative review on Amazon.
It would have been devastating had I read it directly after completing the project.
Fortunately, four years of good reviews and my own confidence in the writing
allowed me to ignore the vitriol and take the remainder as constructive feedback.
The truth is many of the flaws the reviewer had pointed out were features at the
time the book was originally published. Python 3 Object Oriented Programming was
showing its age, and it was clearly time for an update. You're holding the result in
your hands (or flipping through it on your e-reader).
I've often wondered why authors describe in detail what has changed between the
editions of a technical book. I mean, seriously, how many people reading this second
edition have read the first one? As with software versions, you safely assume the
latest edition is the best, and you don't really care about the project's history. And
yet, this project has consumed so much of my life over the past year that I can't

leave without a few words about how much better the book has become.


The original book was a little disorganized. Many chapters flowed directly into
the next one, but there were a few key places where the topic change was jarring,
or worse, irrelevant. The two chapters preceding the discussions about design
patterns have been reorganized, reversed, and split into three chapters that flow
cleanly into the next topic.
I've also removed an entire chapter on third-party libraries for Python 3. This chapter
made more sense when both the book and Python 3 were new. There were only a
few libraries that had been ported to Python 3 and it was reasonable to have a best
of breed discussion about each of them. However, I was unable to cover any of those
topics in detail, and frankly, I could write an entire book on any one of them.
Finally, I've added an entire new chapter on concurrency. I struggled with this
chapter and I can freely admit that it's not directly related to object-oriented
programming. However, much like the chapter on unit testing, I think that
understanding concurrency is an integral part of all programming and especially
of object-oriented programming in the Python ecosystem. You are, of course, free
to skip those chapters if you disagree (or until you discover a reason to change
your mind).
Enjoy the book and your journey into the world of object-oriented programming.

Dusty Phillips


Table of Contents
Prefacevii
Chapter 1: Object-oriented Design
1
Introducing object-oriented

1
Objects and classes
3
Specifying attributes and behaviors
5
Data describes objects
6
Behaviors are actions
7
Hiding details and creating the public interface
9
Composition11
Inheritance14
Inheritance provides abstraction
16
Multiple inheritance
17
Case study
18
Exercises25
Summary26

Chapter 2: Objects in Python

27

Creating Python classes
Adding attributes
Making it do something

27
29
30

Talking to yourself
More arguments

30
31

Initializing the object
Explaining yourself
Modules and packages
Organizing the modules

33
35
37
40

Organizing module contents

43

Absolute imports
Relative imports

40
41

[i]


Table of Contents

Who can access my data?
46
Third-party libraries
48
Case study
49
Exercises58
Summary58

Chapter 3: When Objects Are Alike

59

Chapter 4: Expecting the Unexpected

97

Basic inheritance
59
Extending built-ins
62
Overriding and super
63
Multiple inheritance
65

The diamond problem
67
Different sets of arguments
72
Polymorphism75
Abstract base classes
78
Using an abstract base class
78
Creating an abstract base class
79
Demystifying the magic
81
Case study
82
Exercises95
Summary96
Raising exceptions
98
Raising an exception
99
The effects of an exception
101
Handling exceptions
102
The exception hierarchy
108
Defining our own exceptions
109
Case study

114
Exercises123
Summary124

Chapter 5: When to Use Object-oriented Programming
Treat objects as objects
Adding behavior to class data with properties
Properties in detail
Decorators – another way to create properties
Deciding when to use properties
Manager objects
Removing duplicate code
In practice
[ ii ]

125

125
129
132
134
136
138
140
142


Table of Contents

Case study

145
Exercises153
Summary154

Chapter 6: Python Data Structures

155

Empty objects
155
Tuples and named tuples
157
Named tuples
159
Dictionaries160
Dictionary use cases
164
Using defaultdict
164
Counter166

Lists167
Sorting lists
169
Sets173
Extending built-ins
177
Queues182
FIFO queues
183

LIFO queues
185
Priority queues
186
Case study
188
Exercises194
Summary195

Chapter 7: Python Object-oriented Shortcuts

197

Python built-in functions
197
The len() function
198
Reversed198
Enumerate200
File I/O
201
Placing it in context
203
An alternative to method overloading
205
Default arguments
206
Variable argument lists
208
Unpacking arguments

212
Functions are objects too
213
Using functions as attributes
217
Callable objects
218
Case study
219
Exercises226
Summary227

[ iii ]


Table of Contents

Chapter 8: Strings and Serialization

229

Strings229
String manipulation
230
String formatting
232
Escaping braces
Keyword arguments
Container lookups
Object lookups

Making it look right

233
234
235
236
237

Strings are Unicode

239

Mutable byte strings
Regular expressions
Matching patterns

243
244
245

Converting bytes to text
Converting text to bytes

240
241

Matching a selection of characters
Escaping characters
Matching multiple characters
Grouping patterns together

Getting information from regular expressions
Making repeated regular expressions efficient

246
247
248
249

250

252

Serializing objects
252
Customizing pickles
254
Serializing web objects
257
Case study
260
Exercises265
Summary267

Chapter 9: The Iterator Pattern

269

Design patterns in brief
269

Iterators270
The iterator protocol
271
Comprehensions273
List comprehensions
273
Set and dictionary comprehensions
276
Generator expressions
277
Generators279
Yield items from another iterable
281
Coroutines284
Back to log parsing
287
Closing coroutines and throwing exceptions
289
The relationship between coroutines, generators, and functions
290

[ iv ]


Table of Contents

Case study
291
Exercises298
Summary299

Chapter 10: Python Design Patterns I

301

Chapter 11: Python Design Patterns II

331

Chapter 12: Testing Object-oriented Programs

357

The decorator pattern
301
A decorator example
302
Decorators in Python
305
The observer pattern
307
An observer example
308
The strategy pattern
310
A strategy example
311
Strategy in Python
313
The state pattern

313
A state example
314
State versus strategy
320
State transition as coroutines
320
The singleton pattern
320
Singleton implementation
321
The template pattern
325
A template example
325
Exercises329
Summary329
The adapter pattern
331
The facade pattern
335
The flyweight pattern
337
The command pattern
341
The abstract factory pattern
346
The composite pattern
351
Exercises355

Summary356
Why test?
Test-driven development
Unit testing
Assertion methods
Reducing boilerplate and cleaning up
Organizing and running tests
Ignoring broken tests

[v]

357
359
360
362
364
365
366


Table of Contents

Testing with py.test
368
One way to do setup and cleanup
370
A completely different way to set up variables
373
Skipping tests with py.test
377

Imitating expensive objects
378
How much testing is enough?
382
Case study
385
Implementing it
386
Exercises391
Summary392

Chapter 13: Concurrency393

Threads394
The many problems with threads
397
Shared memory
The global interpreter lock

397
398

Thread overhead
399
Multiprocessing399
Multiprocessing pools
401
Queues404
The problems with multiprocessing
406

Futures406
AsyncIO409
AsyncIO in action
410
Reading an AsyncIO future
411
AsyncIO for networking
412
Using executors to wrap blocking code
415
Streams417
Executors417

Case study
418
Exercises425
Summary426

Index427

[ vi ]


Preface
This book introduces the terminology of the object-oriented paradigm. It focuses
on object-oriented design with step-by-step examples. It guides us from simple
inheritance, one of the most useful tools in the object-oriented programmer's toolbox
through exception handling to design patterns, an object-oriented way of looking
at object-oriented concepts.
Along the way, we'll learn to integrate the object-oriented and not-so-object-oriented

aspects of the Python programming language. We will learn the complexities of
string and file manipulation, emphasizing (as Python 3 does) the difference between
binary and textual data.
We'll then cover the joys of unit testing, using not one, but two unit testing
frameworks. Finally, we'll explore, through Python's various concurrency
paradigms, how to make objects work well together at the same time.

What this book covers

This book is loosely divided into four major parts. In the first four chapters, we will
dive into the formal principles of object-oriented programming and how Python
leverages them. In chapters 5 through 8, we will cover some of Python's idiosyncratic
applications of these principles by learning how they are applied to a variety of
Python's built-in functions. Chapters 9 through 11 cover design patterns, and the
final two chapters discuss two bonus topics related to Python programming that
may be of interest.
Chapter 1, Object-oriented Design, covers important object-oriented concepts. It deals
mainly with terminology such as abstraction, classes, encapsulation, and inheritance.
We also briefly look at UML to model our classes and objects.

[ vii ]


Preface

Chapter 2, Objects in Python, discusses classes and objects and how they are used in
Python. We will learn about attributes and behaviors on Python objects, and also
the organization of classes into packages and modules. Lastly, we will see how to
protect our data.
Chapter 3, When Objects Are Alike, gives us a more in-depth look into inheritance. It

covers multiple inheritance and shows us how to extend built-ins. This chapter also
covers how polymorphism and duck typing work in Python.
Chapter 4, Expecting the Unexpected, looks into exceptions and exception handling.
We will learn how to create our own exceptions and how to use exceptions for
program flow control.
Chapter 5, When to Use Object-oriented Programming, deals with creating and using
objects. We will see how to wrap data using properties and restrict data access.
This chapter also discusses the DRY principle and how not to repeat code.
Chapter 6, Python Data Structures, covers the object-oriented features of Python's
built-in classes. We'll cover tuples, dictionaries, lists, and sets, as well as a few
more advanced collections. We'll also see how to extend these standard objects.
Chapter 7, Python Object-oriented Shortcuts, as the name suggests, deals with
time-savers in Python. We will look at many useful built-in functions such as
method overloading using default arguments. We'll also see that functions
themselves are objects and how this is useful.
Chapter 8, Strings and Serialization, looks at strings, files, and formatting. We'll
discuss the difference between strings, bytes, and bytearrays, as well as various ways
to serialize textual, object, and binary data to several canonical representations.
Chapter 9, The Iterator Pattern, introduces us to the concept of design patterns and
covers Python's iconic implementation of the iterator pattern. We'll learn about list,
set, and dictionary comprehensions. We'll also demystify generators and coroutines.
Chapter 10, Python Design Patterns I, covers several design patterns, including the
decorator, observer, strategy, state, singleton, and template patterns. Each pattern is
discussed with suitable examples and programs implemented in Python.
Chapter 11, Python Design Patterns II, wraps up our discussion of design patterns
with coverage of the adapter, facade, flyweight, command, abstract, and composite
patterns. More examples of how idiomatic Python code differs from canonical
implementations are provided.

[ viii ]

Preface

Chapter 12, Testing Object-oriented Programs, opens with why testing is so important
in Python applications. It emphasizes test-driven development and introduces two
different testing suites: unittest and py.test. Finally, it discusses mocking test objects
and code coverage.
Chapter 13, Concurrency, is a whirlwind tour of Python's support (and lack thereof)
of concurrency patterns. It discusses threads, multiprocessing, futures, and the new
AsyncIO library.
Each chapter includes relevant examples and a case study that collects the chapter's
contents into a working (if not complete) program.

What you need for this book

All the examples in this book rely on the Python 3 interpreter. Make sure you are not
using Python 2.7 or earlier. At the time of writing, Python 3.4 was the latest release
of Python. Most examples will work on earlier revisions of Python 3, but you are
encouraged to use the latest version to minimize frustration.
All of the examples should run on any operating system supported by Python.
If this is not the case, please report it as a bug.
Some of the examples need a working Internet connection. You'll probably want
to have one of these for extracurricular research and debugging anyway!
In addition, some of the examples in this book rely on third-party libraries that do
not ship with Python. These are introduced within the book at the time they are
used, so you do not need to install them in advance. However, for completeness,
here is a list:
• pip
• requests

• pillow
• bitarray

[ ix ]


Preface

Who this book is for

This book specifically targets people who are new to object-oriented programming.
It assumes you have basic Python skills. You'll learn object-oriented principles in
depth. It is particularly useful for system administrator types who have used
Python as a "glue" language and would like to improve their programming skills.
If you are familiar with object-oriented programming in other languages, then this
book will help you understand the idiomatic ways to apply your knowledge in the
Python ecosystem.

Conventions

This book uses a variety of text styles to distinguish between different kinds
of information. Here are some examples of these styles, and an explanation of
their meaning.
Code words in text, database table names, folder names, filenames, file extensions,
pathnames, dummy URLs, user input, and Twitter handles are shown as follows: "We
look up the class in the dictionary and store it in a variable named PropertyClass."
A block of code is set as follows:
def add_property(self):
property_type = get_valid_input(
"What type of property? ",

("house", "apartment")).lower()
payment_type = get_valid_input(
"What payment type? ",
("purchase", "rental")).lower()

When we wish to draw your attention to a particular part of a code block, the
relevant lines or items are set in bold:
def add_property(self):
property_type = get_valid_input(
"What type of property? ",
("house", "apartment")).lower()
payment_type = get_valid_input(
"What payment type? ",
("purchase", "rental")).lower()

[x]


Preface

Any command-line input or output is written as follows:
>>> c1 = Contact("John A", "")
>>> c2 = Contact("John B", "")
>>> c3 = Contact("Jenna C", "")
>>> [c.name for c in Contact.all_contacts.search('John')]
['John A', 'John B']

New terms and important words are shown in bold. Words that you see on the
screen, in menus or dialog boxes for example, appear in the text like this: "It will
fail with a not enough arguments error similar to the one we received earlier

when we forgot the self argument."
Warnings or important notes appear in a box like this.

Tips and tricks appear like this.

Reader feedback

Feedback from our readers is always welcome. Let us know what you think about
this book—what you liked or disliked. Reader feedback is important for us as it
helps us develop titles that you will really get the most out of.
To send us general feedback, simply e-mail , and mention
the book's title in the subject of your message.
If there is a topic that you have expertise in and you are interested in either writing
or contributing to a book, see our author guide at www.packtpub.com/authors.

Customer support

Now that you are the proud owner of a Packt book, we have a number of things to
help you to get the most from your purchase.

[ xi ]


Preface

Downloading the example code

You can download the example code files from your account at http://www.
packtpub.com for all the Packt Publishing books you have purchased. If you
purchased this book elsewhere, you can visit />and register to have the files e-mailed directly to you.

Errata

Although we have taken every care to ensure the accuracy of our content, mistakes
do happen. If you find a mistake in one of our books—maybe a mistake in the text or
the code—we would be grateful if you could report this to us. By doing so, you can
save other readers from frustration and help us improve subsequent versions of this
book. If you find any errata, please report them by visiting ktpub.
com/submit-errata, selecting your book, clicking on the Errata Submission Form
link, and entering the details of your errata. Once your errata are verified, your
submission will be accepted and the errata will be uploaded to our website or added
to any list of existing errata under the Errata section of that title.
To view the previously submitted errata, go to />content/support and enter the name of the book in the search field. The required
information will appear under the Errata section.

Piracy

Piracy of copyrighted material on the Internet is an ongoing problem across all
media. At Packt, we take the protection of our copyright and licenses very seriously.
If you come across any illegal copies of our works in any form on the Internet, please
provide us with the location address or website name immediately so that we can
pursue a remedy.
Please contact us at with a link to the suspected
pirated material.
We appreciate your help in protecting our authors and our ability to bring you
valuable content.

Questions

If you have a problem with any aspect of this book, you can contact us at

, and we will do our best to address the problem.

[ xii ]