Hacking ebook nostarch sampler learnyousomecode

More From

No Starch Press



Read sample chapters from
th e s e N o S ta r c h b o o k s !
I mp r a c t i c a l P y t h o n p r o j e c t s
Lee Vaughan

•

Serious Python
Julien Danjou

•

M at h A dv e n t u r e s w i t h p y t h o n
Peter Farrell

•

practical sql
Anthony DeBarros

•

the rust programming language
Steve Klabnik and Carol Nichols,
with contributions from the rust community

•

p r a c t i c a l b i n a ry a n a ly s i s
Dennis Andriesse

•

mission python
Sean McManus

•

c o di n g w i t h m i n e c r a f t
Al Sweigart



7

Br e e ding Gi a n t R at s w it h
Gene tic Algorithms

Genetic algorithms are general-purpose
­optimization programs designed to solve
complex problems. Invented in the 1970s,
they belong to the class of evolutionary algorithms,
so named because they mimic the Darwinian process
of natural selection. They are especially useful when
little is known about a problem, when you’re dealing

with a nonlinear problem, or when searching for bruteforce-type solutions in a large search space. Best of all,
they are easy algorithms to grasp and implement.
In this chapter, you’ll use genetic algorithms to breed a race of superrats that can terrorize the world. After that, you’ll switch sides and help
James Bond crack a high-tech safe in a matter of seconds. These two projects should give you a good appreciation for the mechanics and power of
genetic algorithms.


Finding the Best of All Possible Solutions
Genetic algorithms optimize, which means that they select the best solution
(with regard to some criteria) from a set of available alternatives. For example, if you’re looking for the fastest route to drive from New York to Los
Angeles, a genetic algorithm will never suggest you fly. It can choose only
from within an allowed set of conditions that you provide. As ­optimizers,
these algorithms are faster than traditional methods and can avoid premature convergence to a suboptimal answer. In other words, they efficiently
search the solution space yet do so thoroughly enough to avoid picking a
good answer when a better one is available.
Unlike exhaustive search engines, which use pure brute force, genetic
algorithms don’t try every possible solution. Instead, they continuously
grade solutions and then use them to make “informed guesses” going forward. A simple example is the “warmer-colder” game, where you search
for a hidden item as someone tells you whether you are getting warmer or
colder based on your proximity or search direction. Genetic algorithms use
a fitness function, analogous to natural selection, to discard “colder” solutions and build on the “warmer” ones. The basic process is as follows:
1.
2.
3.
4.

Randomly generate a population of solutions.
Measure the fitness of each solution.
Select the best (warmest) solutions and discard the rest.
Cross over (recombine) elements in the best solutions to make new

solutions.
5. Mutate a small number of elements in the solutions by changing their
value.
6. Return to step 2 and repeat.
The select–cross over–mutate loop continues until it reaches a
stop ­condition, like finding a known answer, finding a “good enough”
answer (based on a minimum threshold), completing a set number
of iterations, or reaching a time deadline. Because these steps closely
resemble the p
­ rocess of evolution, complete with survival of the fittest,
the terminology used with genetic algorithms is often more biological
than computational.

Project #13: Breeding an Army of Super-Rats
Here’s your chance to be a mad scientist with a secret lab full of boiling
beakers, bubbling test tubes, and machines that go “BZZZTTT.” So pull
on some black rubber gloves and get busy turning 100 nimble trash-eating
scavengers into massive man-eating monsters.

126   Chapter 7


The Objec ti v e
Use a genetic algorithm to simulate breeding rats to an average weight of 110 pounds.

Strategy
Your dream is to breed a race of rats the size of bullmastiffs (we’ve already
established that you’re mad). You’ll start with Rattus norvegicus, the brown
rat, then add some artificial sweeteners, some atomic radiation from the
1950s, a lot of patience, and a pinch of Python, but no genetic engineering—

you’re old-school, baby! The rats will grow from less than a pound to a terrifying 110 pounds, about the size of a female bullmastiff (see Figure 7-1).

Figure 7-1: Size comparison of a brown rat, a female bullmastiff,
and a human

Before you embark on such a huge undertaking, it’s prudent to simulate the results in Python. And you’ve drawn up something better than a
plan—you’ve drawn some graphical pseudocode (see Figure 7-2).

Breeding Giant Rats with Genetic Algorithms   127


Mode

Min

Max

Populate: Establish
initial population and
range of weights.

Loop
Grade: Evaluate
fitness by comparing
mean population weight
to a target weight.

Select: Cull smallest
males and females.

Min

Max

Breed: Repopulate
with random weights
based on weight range
of the selected rats.

Mutate: Randomly
alter weights on a few
rats. Most outcomes
reduce weight.

Figure 7-2: Genetic algorithm approach to breeding super-rats

The process shown in Figure 7-2 outlines how a genetic algorithm
works. Your goal is to produce a population of rats with an average weight
of 110 pounds from an initial population weighing much less than that.
Going forward, each population (or generation) of rats represents a candidate solution to the problem. Like any animal breeder, you cull undesirable
males and females, which you humanely send to—for you Austin Powers
fans—an evil petting zoo. You then mate and breed the remaining rats, a
process known as crossover in genetic programming.
128   Chapter 7


The offspring of the remaining rats will be essentially the same size
as their parents, so you need to mutate a few. While mutation is rare and
usually results in a neutral-to-nonbeneficial trait (low weight, in this case),
sometimes you’ll successfully produce a bigger rat.

The whole process then becomes a big repeating loop, whether done
organically or programmatically, making me wonder whether we really are
just virtual beings in an alien simulation. At any rate, the end of the loop—
the stop condition—is when the rats reach the desired size or you just can’t
stand dealing with rats anymore.
For input to your simulation, you’ll need some statistics. Use the metric
system since you’re a scientist, mad or not. You already know that the average weight of a female bullmastiff is 50,000 grams, and you can find useful
rat statistics in Table 7-1.
Table 7-1: Brown Rat Weight and Breeding Statistics

Parameter

Published values

Minimum weight

200 grams

Average weight (female)

250 grams

Average weight (male)

300–350 grams

Maximum weight

600 grams*

Number of pups per litter

8–12

Litters per year

4–13

Life span (wild, captivity)

1–3 years, 4–6 years

*Exceptional individuals may reach 1,000 grams in captivity.

Because both domestic and wild brown rats exist, there may be wide
variation in some of the stats. Rats in captivity tend to be better cared for
than wild rats, so they weigh more, breed more, and have more pups. So
you can choose from the higher end when a range is available. For this project, start with the assumptions in Table 7-2.
Table 7-2: Input Assumptions for the Super-Rats Genetic Algorithm

Variable and value

Comments

GOAL = 50000

Target weight in grams (female bullmastiff)

NUM_RATS = 20

Total number of adult rats your lab can support

INITIAL_MIN_WT = 200

Minimum weight of adult rat, in grams, in initial population

INITIAL_MAX_WT = 600

Maximum weight of adult rat, in grams, in initial population

INITIAL_MODE_WT = 300

Most common adult rat weight, in grams, in initial population

MUTATE_ODDS = 0.01

Probability of a mutation occurring in a rat

MUTATE_MIN = 0.5

Scalar on rat weight of least beneficial mutation

MUTATE_MAX = 1.2

Scalar on rat weight of most beneficial mutation

LITTER_SIZE = 8

Number of pups per pair of mating rats

LITTERS_PER_YEAR = 10

Number of litters per year per pair of mating rats

GENERATION_LIMIT = 500

Generational cutoff to stop breeding program
Breeding Giant Rats with Genetic Algorithms   129



7

Me t hods a nd Decor ators

Python’s decorators are a handy way
to modify functions. Decorators were
first introduced in Python 2.2, with the
classmethod() and staticmethod() decorators,
but were overhauled to become more flexible and
readable. Along with these two original decorators,
Python now provides a few right out of the box and
supports the simple creation of custom decorators.
But it seems as though most developers do not understand how they work behind the scenes.
This chapter aims to change that—we’ll cover what a decorator is and
how to use it, as well as how to create your own decorators. Then we’ll look
at using decorators to create static, class, and abstract methods and take a
close look at the super function, which allows you to place implementable
code inside an abstract method.

Decorators and When to Use Them
A decorator is a function that takes another function as an argument and
replaces it with a new, modified function. The primary use case for decorators is in factoring common code that needs to be called before, after, or
around multiple functions. If you’ve ever written Emacs Lisp code, you may
have used the defadvice decorator, which allows you to define code called
around a function. If you’ve used method combinations in the Common
Lisp Object System (CLOS), Python decorators follow the same concepts.1
We’ll look at some simple decorator definitions, and then we’ll examine
some common situations in which you’d use decorators.

Creating Decorators
The odds are good that you’ve already used decorators to make your own
wrapper functions. The dullest possible decorator and simplest example is
the identity() function, which does nothing except return the original function. Here is its definition:
def identity(f):
return f

You would then use your decorator like this:
@identity
def foo():
return 'bar'

Here we give the decorator preceded by an @ symbol and then the function we want to use it on. This is the same as writing:
def foo():
return 'bar'
foo = identity(foo)

This decorator is useless and does nothing, but it works. Let’s look at
another, more useful example in Listing 7-1.

_functions = {}
def register(f):
global _functions
_functions[f.__name__] = f
return f
@register
def foo():
return 'bar'
Listing 7-1: A decorator to organize functions in a dictionary

1.

64   Chapter 7


In Listing 7-1, the register decorator stores the decorated function
name into a dictionary. The _functions dictionary can then be used and
accessed using the function name to retrieve a function: _functions['foo']
points to the foo function.
In the following sections, I will explain how to write your own decorators. Then I’ll cover how the built-in decorators provided by Python work
and explain how (and when) to use them.

Writing Decorators
As mentioned, decorators are often used when refactoring repeated code
around functions. Consider the following set of functions that need to
check whether the username that they receive as an argument is the admin
or not and, if the user is not an admin, raise an exception:
class Store(object):
def get_food(self, username, food):
if username != 'admin':

raise Exception("This user is not allowed to get food")
return self.storage.get(food)
def put_food(self, username, food):
if username != 'admin':
raise Exception("This user is not allowed to put food")
self.storage.put(food)

We can see there’s some repeated code here. The obvious first step to
making this code more efficient is to factor the code that checks for admin
status:
u def check_is_admin(username):
if username != 'admin':
raise Exception("This user is not allowed to get or put food")
class Store(object):
def get_food(self, username, food):
check_is_admin(username)
return self.storage.get(food)
def put_food(self, username, food):
check_is_admin(username)
self.storage.put(food)

We’ve moved the checking code into its own function u. Now our code
looks a bit cleaner, but we can do even better if we use a decorator, as shown
in Listing 7-2.
def check_is_admin(f):
u def wrapper(*args, **kwargs):
if kwargs.get('username') != 'admin':
raise Exception("This user is not allowed to get or put food")
return f(*args, **kwargs)
Methods and Decorators   65

5

Tr an s fo r m i n g S hapes
with   G eo m etry
In the teahouse one day Nasrudin announced he was
selling his house. When the other patrons asked him to describe
it, he brought out a brick. “It’s just a collection of these.”
—Idries Shah

In geometry class, everything you learn
about involves dimensions in space using
shapes. You typically start by examining
one-dimensional lines and two-dimensional
circles, squares, or triangles, then move on to threedimensional objects like spheres and cubes. These
days, creating geometric shapes is easy with technology
and free software, though manipulating and changing
the shapes you create can be more of a challenge.
In this chapter, you’ll learn how to manipulate and transform geometric shapes using the Processing graphics package. You’ll start with basic
shapes like circles and triangles, which will allow you to work with complicated shapes like fractals and cellular automata in later chapters. You will
also learn how to break down some complicated-looking designs into simple
components.


Drawing a Circle
Let’s start with a simple one-dimensional circle. Open a new sketch in
Processing and save it as geometry.pyde. Then enter the code in Listing 5-1
to create a circle on the screen.

geometry.pyde def setup():
size(600,600)
def draw():
ellipse(200,100,20,20)
Listing 5-1: Drawing a circle

Before we draw the shape, we first define the size of our sketchbook,
known as the coordinate plane. In this example, we use the size() function to
say that our grid will be 600 pixels wide and 600 pixels tall.
With our coordinate plane set up, we then use the drawing function
ellipse() to create our circle on this plane. The first two parameters, 200
and 100, show where the center of the circle is located. Here, 200 is the
x-coordinate and the second number, 100, is the y-coordinate of this circle’s
center, which places it at (200,100) on the plane.
The last two parameters determine the width and height of the shape in
pixels. In the example, the shape is 20 pixels wide and 20 pixels tall. Because
the two parameters are the same, it means that the points on the circumference are equidistant from the center, forming a perfectly round circle.
Click the Run button (it looks like a play symbol), and a new window
with a small circle should open, like in Figure 5-1.

Figure 5-1: The output of Listing 5-1 showing a small circle

78   Chapter 5


Processing has a number of functions you can use to draw shapes.
Check out the full list at to explore other
shape functions.
Now that you know how to draw a circle in Processing, you’re almost
ready to use these simple shapes to create dynamic, interactive graphics.

In order to do that, you’ll first need to learn about location and transformations. Let’s start with location.

Specifying Location Using Coordinates
In Listing 5-1, we used the first two parameters of the ellipse() function
to specify our circle’s location on the grid. Likewise, each shape we create
using Processing needs a location that we specify with the coordinate system, where each point on the graph is represented by two numbers: (x,y).
In traditional math graphs, the origin (where x=0 and y=0) is at the center
of the graph, as shown in Figure 5-2.
(100,100)

(0,0)

Figure 5-2: A traditional coordinate
system with the origin in the center

In computer graphics, however, the coordinate system is a little different.
Its origin is in the top-left corner of the screen so that x and y increase as you
move right and down, respectively, as you can see in Figure 5-3.
(0,0)

(100,100)

Figure 5-3: The coordinate system for computer
graphics, with the origin in the top-left corner
Transforming Shapes with Geometry   79



1

Cr e at ing Your F irs t
Data ba se a nd Ta ble

SQL is more than just a means for extracting knowledge from data. It’s also a language for defining the structures that hold
data so we can organize relationships in the data.
Chief among those structures is the table.
A table is a grid of rows and columns that store data. Each row holds a
collection of columns, and each column contains data of a specified type:
most commonly, numbers, characters, and dates. We use SQL to define the
structure of a table and how each table might relate to other tables in the
database. We also use SQL to extract, or query, data from tables.
Understanding tables is fundamental to understanding the data in your
database. Whenever I start working with a fresh database, the first thing I
do is look at the tables within. I look for clues in the table names and their
column structure. Do the tables contain text, numbers, or both? How many
rows are in each table?
Next, I look at how many tables are in the database. The simplest
database might have a single table. A full-bore application that handles


customer data or tracks air travel might have dozens or hundreds. The
number of tables tells me not only how much data I’ll need to analyze, but
also hints that I should explore relationships among the data in each table.
Before you dig into SQL, let’s look at an example of what the contents
of tables might look like. We’ll use a hypothetical database for managing a
school’s class enrollment; within that database are several tables that track
students and their classes. The first table, called student_enrollment, shows
the students that are signed up for each class section:
student_id
---------CHRISPA004

DAVISHE010
ABRILDA002
DAVISHE010
RILEYPH002

class_id
---------COMPSCI101
COMPSCI101
ENG101
ENG101
ENG101

class_section
------------3
3
40
40
40

semester
--------Fall 2017
Fall 2017
Fall 2017
Fall 2017
Fall 2017

This table shows that two students have signed up for COMPSCI101, and
three have signed up for ENG101. But where are the details about each student and class? In this example, these details are stored in separate tables
called students and classes, and each table relates to this one. This is where
the power of a relational database begins to show itself.

The first several rows of the students table include the following:
student_id
---------ABRILDA002
CHRISPA004
DAVISHE010
RILEYPH002

first_name
---------Abril
Chris
Davis
Riley

last_name
--------Davis
Park
Hernandez
Phelps

dob
---------1999-01-10
1996-04-10
1987-09-14
1996-06-15

The students table contains details on each student, using the value in the
student_id column to identify each one. That value acts as a unique key that
connects both tables, giving you the ability to create rows such as the following with the class_id column from student_enrollment and the first_name and
last_name columns from students:
class_id

---------COMPSCI101
COMPSCI101
ENG101
ENG101
ENG101

first_name
---------Davis
Chris
Abril
Davis
Riley

last_name
--------Hernandez
Park
Davis
Hernandez
Phelps

The classes table would work the same way, with a class_id column and
several columns of detail about the class. Database builders prefer to organize data using separate tables for each main entity the database manages
in order to reduce redundant data. In the example, we store each student’s
name and date of birth just once. Even if the student signs up for multiple
2   Chapter 1


classes—as Davis Hernandez did—we don’t waste database space entering
his name next to each class in the student_enrollment table. We just include his
student ID.

Given that tables are a core building block of every database, in this
chapter you’ll start your SQL coding adventure by creating a table inside a
new database. Then you’ll load data into the table and view the completed
table.

Creating a Database
The PostgreSQL program you downloaded in the Introduction is a database
management system, a software package that allows you to define, manage,
and query databases. When you installed PostgreSQL, it created a database
server—an instance of the application running on your computer—that
includes a default database called postgres. The database is a collection
of objects that includes tables, functions, user roles, and much more.
According to the PostgreSQL documentation, the default database is
“meant for use by users, utilities and third party applications” (see https://
www.postgresql.org/docs/current/static/app-initdb.html). In the exercises in this
chapter, we’ll leave the default as is and instead create a new one. We’ll do
this to keep objects related to a particular topic or application organized
together.
To create a database, you use just one line of SQL, shown in Listing 1-1.
This code, along with all the examples in this book, is available for download via the resources at />CREATE DATABASE analysis;
Listing 1-1: Creating a database named analysis

This statement creates a database on your server named analysis using
default PostgreSQL settings. Note that the code consists of two keywords—
CREATE and DATABASE—followed by the name of the new database. The statement ends with a semicolon, which signals the end of the command. The
semicolon ends all PostgreSQL statements and is part of the ANSI SQL
standard. Sometimes you can omit the semicolon, but not always, and particularly not when running multiple statements in the admin. So, using the
semicolon is a good habit to form.

Executing SQL in pgAdmin

As part of the Introduction to this book, you also installed the graphical
administrative tool pgAdmin (if you didn’t, go ahead and do that now). For
much of our work, you’ll use pgAdmin to run (or execute) the SQL statements we write. Later in the book in Chapter 16, I’ll show you how to run
SQL statements in a terminal window using the PostgreSQL command line
program psql, but getting started is a bit easier with a graphical interface.

Creating Your First Database and Table

   3



2

Progr amming a
Guessing Game

Let’s jump into Rust by working through
a hands-on project together! This chapter introduces you to a few common Rust
concepts by showing you how to use them in a
real program. You’ll learn about let, match, methods,
associated functions, external crates, and more!
The following chapters will explore these ideas
in more detail. In this chapter, you’ll practice the
fundamentals.
We’ll implement a classic beginner programming problem: a guessing
game. Here’s how it works: the program will generate a random integer
between 1 and 100. It will then prompt the player to enter a guess. After a
guess is entered, the program will indicate whether the guess is too low or
too high. If the guess is correct, the game will print a congratulatory message

and exit.


Setting Up a New Project
To set up a new project, go to the projects directory that you created in
Chapter 1 and make a new project using Cargo, like so:
$ cargo new guessing_game --bin
$ cd guessing_game

The first command, cargo new, takes the name of the project (guessing​
_game) as the first argument. The --bin flag tells Cargo to make a binary
project, like the one in Chapter 1. The second command changes to the
new project’s directory.
Look at the generated Cargo.toml file:
Cargo.toml

[package]
name = "guessing_game"
version = "0.1.0"
authors = ["Your Name <>"]
[dependencies]

If the author information that Cargo obtained from your environment
is not correct, fix that in the file and save it again.
As you saw in Chapter 1, cargo new generates a “Hello, world!” program
for you. Check out the src/main.rs file:
src/main.rs

fn main() {
println!("Hello, world!");

}

Now let’s compile this “Hello, world!” program and run it in the same
step using the cargo run command:
$ cargo run
Compiling guessing_game v0.1.0 (file:///projects/guessing_game)
Finished dev [unoptimized + debuginfo] target(s) in 1.50 secs
Running `target/debug/guessing_game`
Hello, world!

The run command comes in handy when you need to rapidly iterate on a
project, as we’ll do in this game, quickly testing each iteration before moving
on to the next one.
Reopen the src/main.rs file. You’ll be writing all the code in this file.

Processing a Guess
The first part of the guessing game program will ask for user input, process
that input, and check that the input is in the expected form. To start, we’ll
allow the player to input a guess. Enter the code in Listing 2-1 into src/main.rs.
14   Chapter 2


src/main.rs

use std::io;
fn main() {
println!("Guess the number!");
println!("Please input your guess.");
let mut guess = String::new();
io::stdin().read_line(&mut guess)

.expect("Failed to read line");
println!("You guessed: {}", guess);
}
Listing 2-1: Code that gets a guess from the user and prints it

This code contains a lot of information, so let’s go over it line by line.
To obtain user input and then print the result as output, we need to bring
the io (input/output) library into scope. The io library comes from the
standard library (which is known as std):
use std::io;

By default, Rust brings only a few types into the scope of every program
in the prelude. If a type you want to use isn’t in the prelude, you have to bring
that type into scope explicitly with a use statement. Using the std::io library
provides you with a number of useful features, including the ability to accept
user input.
As you saw in Chapter 1, the main function is the entry point into the
program:
fn main() {

The fn syntax declares a new function, the parentheses, (), indicate there
are no parameters, and the curly bracket, {, starts the body of the function.
As you also learned in Chapter 1, println! is a macro that prints a string
to the screen:
println!("Guess the number!");
println!("Please input your guess.");

This code is printing a prompt stating what the game is and requesting
input from the user.

Storing Values with Variables
Next, we’ll create a place to store the user input, like this:
let mut guess = String::new();
Programming a Guessing Game

   15