Python journey from novice to expert learn core concepts of python and unleash its power to script highest quality python programs

Python: Journey from
Novice to Expert

Learn core concepts of Python and unleash its power
to script highest quality Python programs.

A course in three modules

BIRMINGHAM - MUMBAI


Python: Journey from Novice to Expert
Copyright © 2016 Packt Publishing

All rights reserved. No part of this course may be reproduced, stored in a retrieval
system, or transmitted in any form or by any means, without the prior written
permission of the publisher, except in the case of brief quotations embedded in
critical articles or reviews.
Every effort has been made in the preparation of this course to ensure the accuracy
of the information presented. However, the information contained in this course
is sold without warranty, either express or implied. Neither the authors, nor Packt
Publishing, and its dealers and distributors will be held liable for any damages
caused or alleged to be caused directly or indirectly by this course.
Packt Publishing has endeavored to provide trademark information about all of the
companies and products mentioned in this course by the appropriate use of capitals.
However, Packt Publishing cannot guarantee the accuracy of this information.

Published on: August 2016

Published by Packt Publishing Ltd.

Livery Place
35 Livery Street
Birmingham B3 2PB, UK.
ISBN 978-1-78712-076-1
www.packtpub.com


Credits
Authors
Fabrizio Romano

Content Development Editor
Onkar Wani

Dusty Phillips
Rick van Hattem
Reviewers
Simone Burol
Julio Vicente Trigo Guijarro
Veit Heller
AMahdy AbdElAziz
Grigoriy Beziuk
Krishna Bharadwaj
Justin Cano
Anthony Petitbois
Claudio Rodriguez
Randall Degges
Dave de Fijter
I. de Hoogt

Graphics
Abhinash Sahu
Production Coordinator
Melwyn Dsa



Preface
Python is a dynamic programming language. It is known for its high readability
and hence it is often the first language learned by new programmers. Python being
multi-paradigm, it can be used to achieve the same thing in different ways and it is
compatible across different platforms. Coding in Python minimizes development
time and increases productivity in comparison to other languages. Clean,
maintainable code is easy to both read and write using Python's clear, concise syntax.

What this learning path covers

Module 1, Learning Python, This module begins by exploring the essentials of
programming, data structures and teaches you how to manipulate them. It then
moves on to controlling the flow of a program and writing reusable and error proof
code. You will then explore different programming paradigms that will allow you to
find the best approach to any situation, and also learn how to perform performance
optimization as well as effective debugging. Throughout, the module steers you
through the various types of applications, and it concludes with a complete mini
website built upon all the concepts that you learned.
Module 2, Python 3 Object-Oriented Programming, Second Edition, You will learn
how to use the Python programming language to clearly grasp key concepts from the
object-oriented paradigm. This modules fully explains classes, data encapsulation,
inheritance, polymorphism, abstraction, and exceptions with an emphasis on when
you can use each principle to develop a well-designed software. You'll get an indepth analysis of many common object-oriented design patterns that are more

suitable to Python's unique style. This module will not just teach Python syntax,
but will also build your confidence in how to program and create maintainable
applications with higher level design patterns.

[i]


Preface

Module 3, Mastering Python, This module is an authoritative guide that will help
you learn new advanced methods in a clear and contextualized way. It starts off
by creating a project-specific environment using venv, introducing you to different
Pythonic syntax and common pitfalls before moving on to cover the functional
features in Python. It covers how to create different decorators, generators, and
metaclasses. It also introduces you to functools.wraps and coroutines and how
they work. Later on you will learn to use asyncio module for asynchronous clients
and servers. You will also get familiar with different testing systems such as
py.test, doctest, and unittest, and debugging tools such as Python debugger and
faulthandler. You will learn to optimize application performance so that it works
efficiently across multiple machines and Python versions. Finally, it will teach you
how to access C functions with a simple Python call. By the end of the module, you
will be able to write more advanced scripts and take on bigger challenges.

What you need for this learning path
Module 1:

You are encouraged to follow the examples in this module. In order to do so, you
will need a computer, an Internet connection, and a browser. The module is written
in Python 3.4, but it should also work with any Python 3.* version. It has written
instructions on how to install Python on the three main operating systems used

today: Windows, Mac, and Linux. This module also explained how to install all
the extra libraries used in the various examples and provided suggestions if the
reader finds any issues during the installation of any of them. No particular editor is
required to type the code; however, module suggest that those who are interested in
following the examples should consider adopting a proper coding environment.
Module 2:
All the examples in this module rely on the Python 3 interpreter. Make sure you
are not using Python 2.7 or earlier. At the time of writing, Python 3.4 was the latest
release of Python. Most examples will work on earlier revisions of Python 3, but you
are encouraged to use the latest version to minimize frustration. All of the examples
should run on any operating system supported by Python.
If this is not the case, please report it as a bug. Some of the examples need a working
Internet connection. You'll probably want to have one of these for extracurricular
research and debugging anyway! In addition, some of the examples in this module
rely on third-party libraries that do not ship with Python. These are introduced
within the module at the time they are used, so you do not need to install them in
advance. However, for completeness, here is a list:

[ ii ]


Preface

• pip
• requests
• pillow
• bitarray
Module 3:
The only hard requirement for this module is a Python interpreter. A Python 3.5 or
newer interpreter is recommended, but many of the code examples will function

in older Python versions, such as 2.7, with a simple from __future__ import print_
statement added at the top of the file.Additionally, Chapter 14, Extensions in C/
C++, System Calls, and C/C++ Librariesrequires a C/C++ compiler, such as GCC,
Visual Studio, or XCode. A Linux machine is by far the easiest to execute the C/C++
examples, but these should function on Windows and OS X machines without too
much effort as well.

Who this learning path is for

This course is meant for programmes who wants learn Python programming from a
basic to an expert level. The course is mostly self-contained and introduces Python
Programming to a new reader and can help him become an expert in this trade.
Intended for students and practitioners from novice to experts.

Reader feedback

Feedback from our readers is always welcome. Let us know what you think about
this course—what you liked or disliked. Reader feedback is important for us as it
helps us develop titles that you will really get the most out of.
To send us general feedback, simply e-mail , and mention
the course's title in the subject of your message.
If there is a topic that you have expertise in and you are interested in either writing
or contributing to a course, see our author guide at www.packtpub.com/authors.

Customer support

Now that you are the proud owner of a Packt course, we have a number of things to
help you to get the most from your purchase.

[ iii ]

Preface

Downloading the example code

You can download the example code files for this course from your account at
. If you purchased this course elsewhere, you can visit
and register to have the files e-mailed directly
to you.
You can download the code files by following these steps:
1. Log in or register to our website using your e-mail address and password.
2. Hover the mouse pointer on the SUPPORT tab at the top.
3. Click on Code Downloads & Errata.
4. Enter the name of the course in the Search box.
5. Select the course for which you're looking to download the code files.
6. Choose from the drop-down menu where you purchased this course from.
7. Click on Code Download.
You can also download the code files by clicking on the Code Files button on the
course's webpage at the Packt Publishing website. This page can be accessed by
entering the course's name in the Search box. Please note that you need to be logged
in to your Packt account.
Once the file is downloaded, please make sure that you unzip or extract the folder
using the latest version of:
• WinRAR / 7-Zip for Windows
• Zipeg / iZip / UnRarX for Mac
• 7-Zip / PeaZip for Linux
The code bundle for the course is also hosted on GitHub at />PacktPublishing/Python-Journey-from-Novice-to-Expert. We also have
other code bundles from our rich catalog of books and videos available at https://
github.com/PacktPublishing/. Check them out!

[ iv ]


Preface

Errata

Although we have taken every care to ensure the accuracy of our content, mistakes
do happen. If you find a mistake in one of our courses—maybe a mistake in the text
or the code—we would be grateful if you could report this to us. By doing so, you
can save other readers from frustration and help us improve subsequent versions
of this course. If you find any errata, please report them by visiting http://www.
packtpub.com/submit-errata, selecting your course, clicking on the Errata
Submission Form link, and entering the details of your errata. Once your errata are
verified, your submission will be accepted and the errata will be uploaded to our
website or added to any list of existing errata under the Errata section of that title.
To view the previously submitted errata, go to />content/support and enter the name of the course in the search field. The required
information will appear under the Errata section.

Piracy

Piracy of copyrighted material on the Internet is an ongoing problem across all
media. At Packt, we take the protection of our copyright and licenses very seriously.
If you come across any illegal copies of our works in any form on the Internet, please
provide us with the location address or website name immediately so that we can
pursue a remedy.
Please contact us at with a link to the suspected pirated
material.
We appreciate your help in protecting our authors and our ability to bring you

valuable content.

Questions

If you have a problem with any aspect of this course, you can contact us at
, and we will do our best to address the problem.

[v]



Module 1: Learning Python
Chapter 1: Introduction and First Steps – Take a Deep Breath

3

A proper introduction
4
Enter the Python
6
About Python
7
What are the drawbacks?
9
Who is using Python today?
10
Setting up the environment
10
Installing Python
11

How you can run a Python program
19
How is Python code organized
23
Python's execution model
27
Guidelines on how to write good code
35
The Python culture
36
A note on the IDEs
37
Summary38

Chapter 2: Built-in Data Types

39

Everything is an object
39
Mutable or immutable? That is the question
40
Numbers42
Immutable sequences
48
Mutable sequences
52
Set types
57
Mapping types – dictionaries

59
The collections module
64
Final considerations
68
Summary72
i


Table of Contents

Chapter 3: Iterating and Making Decisions

75

Conditional programming
76
Looping80
Putting this all together
93
A quick peek at the itertools module
99
Summary102

Chapter 4: Functions, the Building Blocks of Code

103

Chapter 5: Saving Time and Memory

137

Chapter 6: Advanced Concepts – OOP, Decorators, and Iterators

175

Why use functions?
104
Scopes and name resolution
109
Input parameters
112
Return values
123
A few useful tips
126
Recursive functions
127
Anonymous functions
128
Function attributes
129
Built-in functions
130
One final example
131
Documenting your code
132
Importing objects
133

Summary136
map, zip, and filter
139
Comprehensions144
Generators150
Some performance considerations
161
Don't overdo comprehensions and generators
164
Name localization
169
Generation behavior in built-ins
170
One last example
171
Summary173

Decorators175
Object-oriented programming
184
Writing a custom iterator
212
Summary213

ii


Table of Contents

Chapter 7: Testing, Profiling, and Dealing with Exceptions

215

Chapter 8: The Edges – GUIs and Scripts

249

Chapter 9: Data Science

279

Chapter 10: Web Development Done Right

311

Chapter 11: Debugging and Troubleshooting

343

Chapter 12: Summing Up – A Complete Example

359

Testing your application
216
Test-driven development
235
Exceptions237
Profiling Python
243

Summary247
First approach – scripting
252
Second approach – a GUI application
260
Where do we go from here?
275
Summary278

IPython and Jupyter notebook
280
Dealing with data
283
Where do we go from here?
309
Summary310
What is the Web?
311
How does the Web work?
312
The Django web framework
313
A regex website
316
The future of web development
338
Summary342
Debugging techniques
344
Troubleshooting guidelines

357
Summary358
The challenge
359
Our implementation
360
Implementing the Django interface
360
Implementing the Falcon API
387
Where do you go from here?
404
Summary405
A word of farewell
406

iii


Table of Contents

Module 2: Python 3 Object-Oriented Programming
Chapter 1: Object-oriented Design

409

Chapter 2: Objects in Python

435

Chapter 3: When Objects Are Alike

467

Chapter 4: Expecting the Unexpected

505

Chapter 5: When to Use Object-oriented Programming

533

Introducing object-oriented
409
Objects and classes
411
Specifying attributes and behaviors
413
Hiding details and creating the
public interface
417
Composition419
Inheritance422
Case study
426
Exercises433
Summary434
Creating Python classes
435
Modules and packages

445
Organizing module contents
451
Who can access my data?
454
Third-party libraries
456
Case study
457
Exercises466
Summary466
Basic inheritance
467
Multiple inheritance
473
Polymorphism483
Abstract base classes
486
Case study
490
Exercises503
Summary504
Raising exceptions
506
Case study
522
Exercises531
Summary532
Treat objects as objects
Adding behavior to class data with properties

Manager objects
iv

533
537
546


Table of Contents

Case study
553
Exercises561
Summary562

Chapter 6: Python Data Structures

563

Chapter 7: Python Object-oriented Shortcuts

605

Chapter 8: Strings and Serialization

637

Chapter 9: The Iterator Pattern

677

Empty objects
563
Tuples and named tuples
565
Dictionaries568
Lists575
Sets581
Extending built-ins
585
Queues590
Case study
596
Exercises602
Summary603
Python built-in functions
605
An alternative to method overloading
613
Functions are objects too
621
Case study
627
Exercises634
Summary635
Strings637
Regular expressions
652
Serializing objects
660

Case study
668
Exercises673
Summary675

Design patterns in brief
677
Iterators678
Comprehensions681
Generators687
Coroutines692
Case study
699
Exercises706
Summary707

v


Table of Contents

Chapter 10: Python Design Patterns I

709

Chapter 11: Python Design Patterns II

739

Chapter 12: Testing Object-oriented Programs

765

The decorator pattern
709
The observer pattern
715
The strategy pattern
718
The state pattern
721
The singleton pattern
728
The template pattern
733
Exercises737
Summary737
The adapter pattern
739
The facade pattern
743
The flyweight pattern
745
The command pattern
749
The abstract factory pattern
754
The composite pattern
759
Exercises763

Summary764
Why test?
765
Unit testing
768
Testing with py.test
776
Imitating expensive objects
786
How much testing is enough?
790
Case study
793
Exercises799
Summary800

Chapter 13: Concurrency801

Threads802
Multiprocessing807
Futures814
AsyncIO817
Case study
826
Exercises833
Summary834

vi

Table of Contents

Module 3: Mastering Python
Chapter 1: Getting Started – One Environment per Project

837

Chapter 2: Pythonic Syntax, Common Pitfalls, and Style Guide

849

Chapter 3: Containers and Collections – Storing Data the Right Way

885

Chapter 4: Functional Programming – Readability Versus Brevity

917

Chapter 5: Decorators – Enabling Code Reuse by Decorating

939

Chapter 6: Generators and Coroutines – Infinity, One Step at a Time

977

Creating a virtual Python environment using venv
838
Bootstrapping pip using ensurepip

843
Installing C/C++ packages
844
Summary847
Code style – or what is Pythonic code?
850
Common pitfalls
871
Summary884
Time complexity – the big O notation
886
Core collections
887
Advanced collections
898
Summary915
Functional programming
918
list comprehensions
918
dict comprehensions
921
set comprehensions
922
lambda functions
922
functools925
itertools931
Summary938

Decorating functions
940
Decorating class functions
952
Decorating classes
961
Useful decorators
966
Summary976
What are generators?
978
Coroutines990
Summary1002

vii


Table of Contents

Chapter 7: Async IO – Multithreading without Threads

1003

Chapter 8: Metaclasses – Making Classes (Not Instances) Smarter

1025

Introducing the asyncio library
1004
Summary1024

Dynamically creating classes
1026
Abstract classes using collections.abc
1030
Automatically registering a plugin system
1037
Order of operations when instantiating classes
1043
Storing class attributes in definition order
1048
Summary1051

Chapter 9: Documentation – How to Use Sphinx and reStructuredText 1053

The reStructuredText syntax
1054
The Sphinx documentation generator
1069
Documenting code
1085
Summary1091

Chapter 10: Testing and Logging – Preparing for Bugs

1093

Chapter 11: Debugging – Solving the Bugs

1155

Using examples as tests with doctest
1094
Testing with py.test
1110
Mock objects
1138
Logging1141
Summary1153
Non-interactive debugging
1156
Interactive debugging
1168
Summary1180

Chapter 12: Performance – Tracking and Reducing Your Memory and CPU
Usage1181
What is performance?
1182
Timeit – comparing code snippet performance
1183
cProfile – finding the slowest components
1187
Line profiler
1197
Improving performance
1199
Memory usage
1205
Performance monitoring
1218

Summary1219

viii


Table of Contents

Chapter 13: Multiprocessing – When a Single CPU Core Is Not Enough1221

Multithreading versus multiprocessing
1221
Hyper-threading versus physical CPU cores
1224
Creating a pool of workers
1226
Sharing data between processes
1228
Remote processes
1229
Summary1238

Chapter 14: Extensions in C/C++, System Calls, and C/C++ Libraries 1239

Introduction1239
Calling C/C++ with ctypes
1242
CFFI1249
Native C/C++ extensions
1252
Summary1263

Chapter 15: Packaging – Creating Your Own Libraries or Applications 1265

Installing packages
1265
Setup parameters
1266
Packages1270
Entry points
1270
Package data
1274
Testing packages
1275
C/C++ extensions
1279
Wheels – the new eggs
1282
Summary1285

Bibliography1287

ix



Module 1

Learning Python
Learn to code like a professional with Python – an open source, versatile and powerful

programming language



Introduction and First Steps –
Take a Deep Breath
"Give a man a fish and you feed him for a day. Teach a man to fish and you feed
him for a lifetime."
– Chinese proverb
According to Wikipedia, computer programming is:
"...a process that leads from an original formulation of a computing problem to
executable computer programs. Programming involves activities such as analysis,
developing understanding, generating algorithms, verification of requirements
of algorithms including their correctness and resources consumption, and
implementation (commonly referred to as coding) of algorithms in a target
programming language".
In a nutshell, coding is telling a computer to do something using a language
it understands.
Computers are very powerful tools, but unfortunately, they can't think for
themselves. So they need to be told everything. They need to be told how to perform
a task, how to evaluate a condition to decide which path to follow, how to handle
data that comes from a device such as the network or a disk, and how to react
when something unforeseen happens, say, something is broken or missing.
You can code in many different styles and languages. Is it hard? I would say
"yes" and "no". It's a bit like writing. Everybody can learn how to write, and you
can too. But what if you wanted to become a poet? Then writing alone is not
enough. You have to acquire a whole other set of skills and this will take a
longer and greater effort.
[3]

Introduction and First Steps – Take a Deep Breath

In the end, it all comes down to how far you want to go down the road. Coding is
not just putting together some instructions that work. It is so much more!
Good code is short, fast, elegant, easy to read and understand, simple, easy to modify
and extend, easy to scale and refactor, and easy to test. It takes time to be able to write
code that has all these qualities at the same time, but the good news is that you're
taking the first step towards it at this very moment by reading this book. And I have
no doubt you can do it. Anyone can, in fact, we all program all the time, only we aren't
aware of it.
Would you like an example?
Say you want to make instant coffee. You have to get a mug, the instant coffee jar,
a teaspoon, water, and the kettle. Even if you're not aware of it, you're evaluating a
lot of data. You're making sure that there is water in the kettle as well as the kettle
is plugged-in, that the mug is clean, and that there is enough coffee in the jar. Then,
you boil the water and maybe in the meantime you put some coffee in the mug.
When the water is ready, you pour it into the cup, and stir.
So, how is this programming?
Well, we gathered resources (the kettle, coffee, water, teaspoon, and mug) and we
verified some conditions on them (kettle is plugged-in, mug is clean, there is enough
coffee). Then we started two actions (boiling the water and putting coffee in the
mug), and when both of them were completed, we finally ended the procedure by
pouring water in the mug and stirring.
Can you see it? I have just described the high-level functionality of a coffee program.
It wasn't that hard because this is what the brain does all day long: evaluate
conditions, decide to take actions, carry out tasks, repeat some of them, and stop at
some point. Clean objects, put them back, and so on.
All you need now is to learn how to deconstruct all those actions you do automatically
in real life so that a computer can actually make some sense of them. And you need to

learn a language as well, to instruct it.
So this is what this book is for. I'll tell you how to do it and I'll try to do that by
means of many simple but focused examples (my favorite kind).

A proper introduction

I love to make references to the real world when I teach coding; I believe they help
people retain the concepts better. However, now is the time to be a bit more rigorous
and see what coding is from a more technical perspective.
[4]