PostreSQL 10 administration cookbook over 165 effective recipes for database management and maintenance in PostreSQL 10

PostgreSQL 10 Administration Cookbook

Over 165 effective recipes for database management and maintenance in PostgreSQL 10

Simon Riggs
Gianni Ciolli


BIRMINGHAM - MUMBAI



PostgreSQL 10 Administration Cookbook
Copyright © 2018 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written
permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is
sold without warranty, either express or implied. Neither the author(s), nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused
or alleged to have been caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals.
However, Packt Publishing cannot guarantee the accuracy of this information.
Commissioning Editor: Amey Varangaonkar
Acquisition Editor: Namrata Patil
Content Development Editor: Amrita Noronha
Technical Editor: Sneha Hanchate
Copy Editor: Safis
Project Coordinator: Shweta Birwatkar
Proofreader: Safis Editing
Indexer: Pratik Shirodkar
Graphics: Jisha Chirayil

Production Coordinator: Shraddha Falebhai
First published: April 2017
Production reference: 1070518
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham
B3 2PB, UK.
ISBN 978-1-78847-492-4
www.packtpub.com


mapt.io

Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as
industry leading tools to help you plan your personal development and advance your career. For more
information, please visit our website.


Why subscribe?
Spend less time learning and more time coding with practical eBooks and Videos from over
4,000 industry professionals
Improve your learning with Skill Plans built especially for you
Get a free eBook or video every month
Mapt is fully searchable
Copy and paste, print, and bookmark content


PacktPub.com
Did you know that Packt offers eBook versions of every book published, with PDF and ePub files

available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you
are entitled to a discount on the eBook copy. Get in touch with us at for more
details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free
newsletters, and receive exclusive discounts and offers on Packt books and eBooks.


Contributors


About the authors
Simon Riggs is the CTO of 2ndQuadrant, having contributed to PostgreSQL as a major developer and
committer for 14 years. He has written and designed features for replication, performance,
BI, management, and security. Under his guidance, 2ndQuadrant is now a leading developer of open
source PostgreSQL, serving hundreds of clients in USA, Europe, and worldwide. Simon is a frequent
speaker at many conferences on PostgreSQL Futures. He has worked as a database architect for 30
years.

Gianni Ciolli is the head of professional services at 2ndQuadrant. PostgreSQL consultant, trainer, and
speaker at many PostgreSQL conferences in Europe and abroad since the last 10 years. He has a PhD
in Mathematics from the University of Florence. He has worked with free and open source software
since the 1990s and is active in the community (Prato Linux User Group and Italian PostgreSQL Users
Group). He lives in London with his son. His other interests include music, drama, poetry, and
athletics.


About the reviewer
Sheldon Strauch is a 23-year veteran of software consulting at IBM, Sears, Ernst & Young, and Kraft
Foods. Having a bachelor's degree in business administration, he leverages his technical skills to
improve business' self-awareness. He is interested in data gathering, management, and mining; maps

and mapping; business intelligence; and application of data analysis. He is currently focusing on the
development of data management and mining at Enova International at Chicago. At his leisure, he
enjoys performing arts, music, and traveling with his wife, Marilyn.


Packt is searching for authors like you
If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today.
We have worked with thousands of developers and tech professionals, just like you, to help them
share their insight with the global tech community. You can make a general application, apply for a
specific hot topic that we are recruiting an author for, or submit your own idea.


Table of Contents
Title Page
Copyright and Credits
PostgreSQL 10 Administration Cookbook
Packt Upsell
Why subscribe?
PacktPub.com
Contributors
About the authors
About the reviewer
Packt is searching for authors like you
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Download the color images
Conventions used

Sections
Getting ready
How to do it…
How it works…
There's more…
See also
Get in touch
Reviews

1.

First Steps
Introduction
Introducing PostgreSQL 10
What makes PostgreSQL different?
Robustness
Security
Ease of use
Extensibility
Performance and concurrency
Scalability
SQL and NoSQL
Popularity
Commercial support
Research and development funding
Getting PostgreSQL
How to do it...
How it works...
There's more…
Connecting to the PostgreSQL server

Getting ready
How to do it…


How it works…
There's more…
See also
Enabling access for network/remote users
How to do it…
How it works…
There's more…
See also
Using graphical administration tools
How to do it…
How it works…
OmniDB
See also
Using the psql query and scripting tool
Getting ready
How to do it…
How it works…
There's more…
See also
Changing your password securely
How to do it…
How it works…
Avoiding hardcoding your password
Getting ready
How to do it…
How it works…

There's more…
Using a connection service file
How to do it…
How it works…
Troubleshooting a failed connection
How to do it…
There's more…

2.

Exploring the Database
Introduction
What version is the server?
How to do it…
How it works…
There's more…
What is the server uptime?
How to do it…
How it works...
See also
Locating the database server files
Getting ready
How to do it...
How it works...
There's more…
Locating the database server's message log


Getting ready
How to do it...

How it works...
There's more...
Locating the database's system identifier
Getting ready
How to do it…
How it works…
Listing databases on this database server
How to do it…
How it works...
There's more...
How many tables are there in a database?
How to do it...
How it works…
There's more…
How much disk space does a database use?
How to do it...
How it works...
How much disk space does a table use?
How to do it…
How it works…
There's more…
Which are my biggest tables?
How to do it...
How it works…
How many rows are there in a table?
How to do it…
How it works...
Quickly estimating the number of rows in a table
How to do it…
How it works…

There's more…
Function 1 – Estimating the number of rows
Function 2 – Computing the size of a table without locks
Listing extensions in this database
Getting ready
How to do it…
How it works…
There's more…
Understanding object dependencies
Getting ready
How to do it…
How it works…
There's more…

3.

Configuration
Introduction
Reading the fine manual
How to do it…


How it works…
There's more…
Planning a new database
Getting ready
How to do it…
How it works…
There's more…
Changing parameters in your programs

How to do it…
How it works…
There's more…
Finding the current configuration settings
How to do it…
There's more…
How it works…
Which parameters are at non-default settings?
How to do it…
How it works...
There's more...
Updating the parameter file
Getting ready
How to do it…
How it works…
There's more…
Setting parameters for particular groups of users
How to do it…
How it works…
The basic server configuration checklist
Getting ready
How to do it…
There's more…
Adding an external module to PostgreSQL
Getting ready
How to do it…
Installing modules using a software installer
Installing modules from PGXN
Installing modules from a manually downloaded package
Installing modules from source code

How it works...
Using an installed module
Getting ready
How to do it…
How it works...
Managing installed extensions
How to do it…
How it works…
There's more…

4.

Server Control
Introduction


Starting the database server manually
Getting ready
How to do it…
How it works…
Stopping the server safely and quickly
How to do it…
How it works…
See also
Stopping the server in an emergency
How to do it…
How it works…
Reloading the server configuration files
How to do it…
How it works…

There's more…
Restarting the server quickly
How to do it…
There's more…
Preventing new connections
How to do it…
How it works…
Restricting users to only one session each
How to do it…
How it works…
Pushing users off the system
How to do it…
How it works…
Deciding on a design for multitenancy
How to do it…
How it works…
Using multiple schemas
Getting ready
How to do it…
How it works…
Giving users their own private database
Getting ready
How to do it…
How it works…
There's more…
See also
Running multiple servers on one system
Getting ready
How to do it…
How it works…

Setting up a connection pool
Getting ready
How to do it…
How it works…
There's more…
Accessing multiple servers using the same host and port


Getting ready
How to do it…
There's more…

5.

Tables and Data
Choosing good names for database objects
Getting ready
How to do it…
There's more…
Handling objects with quoted names
Getting ready
How to do it...
How it works…
There's more…
Enforcing the same name and definition for columns
Getting ready
How to do it...
How it works…
There's more…
Identifying and removing duplicates

Getting ready
How to do it…
How it works…
There's more…
Preventing duplicate rows
Getting ready
How to do it…
How it works…
There's more...
Duplicate indexes
Uniqueness without indexes
Real-world example – IP address range allocation
Real-world example – range of time
Real-world example – prefix ranges
Finding a unique key for a set of data
Getting ready
How to do it…
How it works…
Generating test data
How to do it...
How it works…
There's more…
See also
Randomly sampling data
How to do it…
How it works...
Loading data from a spreadsheet
Getting ready
How to do it...
How it works...

There's more...
Loading data from flat files
Getting ready
How to do it...
How it works…
There's more…

6.

Security
Introduction
Typical user role
The PostgreSQL superuser
How to do it…
How it works…
There's more…
Other superuser-like attributes
Attributes are never inherited
See also
Revoking user access to a table
Getting ready
How to do it…
How it works…
There's more…
Database creation scripts
Default search path
Securing views
Granting user access to a table

Getting ready
How to do it…
How it works...
There's more…
Access to the schema
Granting access to a table through a group role
Granting access to all objects in a schema
Granting user access to specific columns
Getting ready
How to do it…
How it works…
There's more…
Granting user access to specific rows
Getting ready
How to do it…
How it works…
There's more...
Creating a new user
Getting ready
How to do it...
How it works…
There's more…
Temporarily preventing a user from connecting
Getting ready


How to do it…
How it works...
There's more…
Limiting the number of concurrent connections by a user

Forcing NOLOGIN users to disconnect
Removing a user without dropping their data
Getting ready
How to do it…
How it works…
Checking whether all users have a secure password
How to do it…
How it works…
Giving limited superuser powers to specific users
Getting ready
How to do it…
How it works…
There's more…
Writing a debugging_info function for developers
Auditing database access
Getting ready
Auditing SQL
Auditing table access
Managing the audit log
Auditing data changes
Always knowing which user is logged in
Getting ready
How to do it…
How it works…
There's more…
Not inheriting user attributes
Integrating with LDAP
Getting ready
How to do it…
How it works…

There's more…
Setting up the client to use LDAP
Replacement for the User Name Map feature
See also
Connecting using SSL
Getting ready
How to do it…
How it works…
There's more…
Getting the SSL key and certificate
Setting up a client to use SSL
Checking server authenticity
Using SSL certificates to authenticate
Getting ready
How to do it…
How it works…


There's more…
Avoiding duplicate SSL connection attempts
Using multiple client certificates
Using the client certificate to select the database user
See also
Mapping external usernames to database roles
Getting ready
How to do it…
How it works…
There's more…
Encrypting sensitive data
Getting ready

How to do it…
How it works…
There's more…
For really sensitive data
For really, really, really sensitive data!
See also

7.

Database Administration
Introduction
Writing a script that either succeeds entirely or fails entirely
How to do it…
How it works…
There's more…
Writing a psql script that exits on the first error
Getting ready
How to do it…
How it works…
There's more…
Using psql variables
Getting ready
How to do it…
How it works…
There’s more…
Placing query output into psql variables
Getting ready
How to do it…
How it works…
There’s more…

Writing a conditional psql script
Getting ready
How to do it…
How it works…
There’s more…
Investigating a psql error
Getting ready
How to do it…
There's more…
Performing actions on many tables


Getting ready
How to do it…
How it works…
There's more…
Adding/removing columns on a table
How to do it…
How it works…
There's more…
Changing the data type of a column
Getting ready
How to do it…
How it works…
There's more…
Changing the definition of a data type
Getting ready
How to do it…
How it works…
There's more…

Adding/removing schemas
How to do it…
There's more…
Using schema-level privileges
Moving objects between schemas
How to do it…
How it works…
There's more…
Adding/removing tablespaces
Getting ready
How to do it…
How it works…
There's more…
Putting pg_wal on a separate device
Tablespace-level tuning
Moving objects between tablespaces
Getting ready
How to do it…
How it works…
There's more…
Accessing objects in other PostgreSQL databases
Getting ready
How to do it…
How it works…
There's more…
There's more…
Accessing objects in other foreign databases
Getting ready
How to do it…
How it works…

There's more…
Updatable views
Getting ready
How to do it…
How it works…
There's more…
Using materialized views
Getting ready
How to do it…
How it works…
There's more…

8.

Monitoring and Diagnosis
Providing PostgreSQL information to monitoring tools
Finding more information about generic monitoring tools
Real-time viewing using pgAdmin or OmniDB
Getting ready
How to do it… (with pgAdmin)
How to do it… (with OmniDB)
Checking whether a user is connected
Getting ready
How to do it…
How it works…
There's more…
Checking whether a computer is connected
How to do it…

There's more…
Repeatedly executing a query in psql
How to do it…
There's more…
Checking which queries are running
Getting ready
How to do it…
How it works…
There's more…
Catching queries that only run for a few milliseconds
Watching the longest queries
Watching queries from ps
See also
Checking which queries are active or blocked
Getting ready
How to do it…
How it works…
There's more…
No need for the = true part
Do we catch all queries waiting on locks?
Knowing who is blocking a query
Getting ready
How to do it…
How it works…


Killing a specific session
How to do it…
How it works…
There's more…

Try to cancel the query first
What if the backend won't terminate?
Using statement_timeout to clean up queries that take too long to run
Killing idle in transaction queries
Killing the backend from the command line
Detecting an in-doubt prepared transaction
How to do it…
Knowing whether anybody is using a specific table
Getting ready
How to do it…
How it works…
There's more...
The quick-and-dirty way
Collecting daily usage statistics
Knowing when a table was last used
Getting ready
How to do it…
How it works...
There's more…
Usage of disk space by temporary data
Getting ready
How to do it…
How it works…
There's more…
Finding out whether a temporary file is in use any more
Logging temporary file usage
Understanding why queries slow down
Getting ready
How to do it…
How it works…

There's more…
Do the queries return significantly more data than they did earlier?
Do the queries also run slowly when they are run alone?
Is the second run of the same query also slow?
Table and index bloat
See also
Investigating and reporting a bug
Getting ready
How to do it…
How it works…
Producing a daily summary of log file errors
Getting ready
How to do it…
How it works…
There's more…


Analyzing the real-time performance of your queries
Getting ready
How to do it…
How it works…
There's more…

9.

Regular Maintenance
Controlling automatic database maintenance
Getting ready
How to do it…
How it works…

There's more…
See also
Avoiding auto-freezing and page corruptions
How to do it…
Removing issues that cause bloat
Getting ready
How to do it…
How it works…
There's more…
Removing old prepared transactions
Getting ready
How to do it…
How it works…
There's more…
Actions for heavy users of temporary tables
How to do it…
How it works…
Identifying and fixing bloated tables and indexes
How to do it…
How it works…
There's more…
Monitoring and tuning vacuum
Getting ready
How to do it…
How it works…
There's more…
Maintaining indexes
Getting ready
How to do it…
How it works…

There's more…
Adding a constraint without checking existing rows
Getting ready
How to do it…
How it works…
Finding unused indexes
How to do it…
How it works…
Carefully removing unwanted indexes