1593275277 {9f0810a5} ruby under a microscope an illustrated guide to ruby internals shaughnessy 2013 11 22

Ruby is a powerful programming language with
a focus on simplicity, but beneath its elegant
syntax it performs countless unseen tasks.
Ruby Under a Microscope gives you a
hands-on look at Ruby’s core, using extensive
diagrams and thorough explanations to show
you how Ruby is implemented (no C skills
required). Author Pat Shaughnessy takes
a scientific approach, laying out a series of
experiments with Ruby code to take you behind
the scenes of how programming languages
work. You’ll even find information on JRuby
and Rubinius (two alternative implementations
of Ruby), as well as in-depth explorations of
Ruby’s garbage collection algorithm.
Ruby Under a Microscope will teach you:
How

a few computer science concepts
underpin Ruby’s complex implementation

How

Ruby executes your code using a
virtual machine

How

classes and modules are the same
inside Ruby

How

Ruby employs algorithms originally
developed for Lisp

How

Ruby uses grammar rules to parse
and understand your code

How

your Ruby code is translated into a
different language by a compiler

No programming language needs to be a
black box. Whether you’re already intrigued by
language implementation or just want to dig
deeper into Ruby, you’ll find Ruby Under a
Microscope a fascinating way to become
a better programmer.

About the Author
Well known for his coding expertise and
passion for the Ruby programming language,
Pat Shaughnessy blogs and writes tutorials
at He also
develops Ruby applications at management
consulting firm McKinsey & Co. Shaughnessy
is a regular presenter on the Ruby conference

circuit, and his articles and presentations have
been featured in the Ruby Weekly newsletter,
the Ruby5 podcast, and The Ruby Show.

“I LIE FLAT.” This book uses RepKover — a durable binding that won’t snap shut.

T H E F I N E ST I N G E E K E N T E RTA I N M E N T ™

w w w.nostarch.com

$39.95 ($41.95 CDN)

Shelve In: Programming Languages/Ruby

Shaughnessy

Covers Ruby 2.x, 1.9, and 1.8

Ruby Under a Microscope

How Ruby Works
Under the Hood

Ruby Under a
Microscope
An Illustrated Guide
to Ruby Internals
Pat Shaughnessy

s

e
m
i
t
.
0
1
n
s
t
pu
end

|
n
|
do


Advance Praise for Ruby Under a Microscope
“Many people have dug into the Ruby source code, but few make it back
out and tell the tale as elegantly as Pat does in Ruby Under a Microscope!
I particularly love the diagrams—and there are lots of them—as they
make many opaque implementation topics a lot easier to understand,
especially when coupled with Pat’s gentle narrative. This book is a delight
for language implementation geeks and Rubyists with a penchant for digging into the guts of their tools.”
—Peter Cooper (@ peterc), Editor of Ruby Inside and Ruby Weekly
“Man, this book was missing in the Ruby landscape—awesome content.”
—X avier Noria (@ fxn), Ruby Hero, Ruby on R ails Core Team Member
“Pat Shaughnessy did a tremendous job writing THE book about Ruby

internals. Definitely a must read—you won’t find information like this
anywhere else.”
—Santiago Pastorino (@ spastorino), W yeWorks Co -Founder,
Ruby on R ails Core Team Member
“I really enjoyed the book and now have a far better understanding of both
Ruby and CS. The writing made very complex topics (at least for me) very
accessible, and I found the book hard to put down. Diagrams were awesome
and are already popping in my head as I code. This is by far one of my top 3
favourite Ruby books.”
—Vlad Ivanovic (@ vladiim), Digital Strategist at Holler S ydney
“While I’m not usually digging into Ruby Internals, this book was an
absolutely awesome read.”
—David Deryl Downey (@ daviddwdowney), Founder of C yber Space
Technologies Group



Ruby Under a
Microscope
An Illustrated Guide
to Ruby Internals
Pat Shaughnessy


Ruby Under a Microscope. Copyright © 2014 by Patrick Shaughnessy.
All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system,
without the prior written permission of the copyright owner and the publisher.
Printed in USA
First printing
17 16 15 14 13   1 2 3 4 5 6 7 8 9

ISBN-10: 1-59327-527-7
ISBN-13: 978-1-59327-527-3
Publisher: William Pollock
Production Editor: Riley Hoffman
Cover Illustration: Charlie Wylie
Interior Design: Octopod Studios
Developmental Editor: William Pollock
Technical Reviewer: Aaron Patterson
Copyeditor: Julianne Jigour
Compositors: Susan Glinert Stevens and Riley Hoffman
Proofreader: Elaine Merrill
For information on distribution, translations, or bulk sales, please contact No Starch Press, Inc. directly:
No Starch Press, Inc.
245 8th Street, San Francisco, CA 94103
phone: 415.863.9900; fax: 415.863.9950; ; www.nostarch.com
Library of Congress Cataloging-in-Publication Data
Shaughnessy, Pat.
Ruby under a microscope : an illustrated guide to Ruby internals / by Pat Shaughnessy.
pages cm
Summary: "An under-the-hood look at how the Ruby programming language runs code. Extensively illustrated with
complete explanations and hands-on experiments. Covers Ruby 2.x"-- Provided by publisher.
ISBN 978-1-59327-527-3 (paperback) -- ISBN 1-59327-527-7 (paperback)
1. Ruby (Computer program language) I. Title.
QA76.73.R83S53 2013
005.1'17--dc23
2013030614

No Starch Press and the No Starch Press logo are registered trademarks of No Starch Press, Inc. Other product
and company names mentioned herein may be the trademarks of their respective owners. Rather than use a
trademark symbol with every occurrence of a trademarked name, we are using the names only in an editorial

fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark.
The information in this book is distributed on an “As Is” basis, without warranty. While every precaution has been
taken in the preparation of this work, neither the author nor No Starch Press, Inc. shall have any liability to any
person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the
information contained in it.


To my wife, Cristina; my daughter, Ana; and my son, Liam—
thanks for supporting me all along.


About the Author

Pat Shaughnessy is a Ruby developer working at McKinsey & Co., a
management consulting firm. Pat was originally trained as a physicist
at MIT, but later spent more than 20 years working as a software developer
using C, Java, PHP, and Ruby, among other languages. Writing Ruby Under
a Microscope has given him an excuse to reuse bits of his scientific training
while studying Ruby. A fluent Spanish speaker, Pat frequently visits his wife’s
family in northern Spain. He lives outside of Boston with his wife and two
children.


Brief Contents

Foreword by Aaron Patterson . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix
Chapter 1: Tokenization and Parsing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Chapter 2: Compilation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

Chapter 3: How Ruby Executes Your Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Chapter 4: Control Structures and Method Dispatch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
Chapter 5: Objects and Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
Chapter 6: Method Lookup and Constant Lookup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
Chapter 7: The Hash Table: The Workhorse of Ruby Internals . . . . . . . . . . . . . . . . . . . . . . 167
Chapter 8: How Ruby Borrowed a Decades-Old Idea from Lisp . . . . . . . . . . . . . . . . . . . . 191
Chapter 9: Metaprogramming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
Chapter 10: JRuby: Ruby on the JVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
Chapter 11: Rubinius: Ruby Implemented with Ruby . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
Chapter 12: Garbage Collection in MRI, JRuby, and Rubinius . . . . . . . . . . . . . . . . . . . . . 295
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327



Con t e n t s in De ta il
Foreword by Aaron Patterson

xv

Acknowledgments

xvii

Introduction

xix

Who This Book Is For . . . . . . . .
Using Ruby to Test Itself . . . . . . .
Which Implementation of Ruby? .

Overview . . . . . . . . . . . . . . . . .

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.

.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.

.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.

.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.

.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.

.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.

.

.
.
.
.

.
.
.
.

.
.
.
.

1
Tokenization and Parsing

. xx
xx
xxi
xxi

3

Tokens: The Words That Make Up the Ruby Language . . . . . . . . . . . . . . . . . . . . . . . . . . 4
The parser_yylex Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Experiment 1-1: Using Ripper to Tokenize Different Ruby Scripts . . . . . . . . . . . . . . . . . . . . 9

Parsing: How Ruby Understands Your Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
Understanding the LALR Parse Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Some Actual Ruby Grammar Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Reading a Bison Grammar Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Experiment 1-2: Using Ripper to Parse Different Ruby Scripts . . . . . . . . . . . . . . . . . . . . . 23
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

2
Compilation31
No Compiler for Ruby 1.8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Ruby 1.9 and 2.0 Introduce a Compiler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
How Ruby Compiles a Simple Script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Compiling a Call to a Block . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
How Ruby Iterates Through the AST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Experiment 2-1: Displaying YARV Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The Local Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Compiling Optional Arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Compiling Keyword Arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Experiment 2-2: Displaying the Local Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3
How Ruby Executes Your Code
YARV’s Internal Stack and Your Ruby Stack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Stepping Through How Ruby Executes a Simple Script . . . . . . . . . . . . . . . . . .
Executing a Call to a Block . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Taking a Close Look at a YARV Instruction . . . . . . . . . . . . . . . . . . . . . . . . . .
Experiment 3-1: Benchmarking Ruby 2.0 and Ruby 1.9 vs. Ruby 1.8 . . . . . . . . . . . . . . . .

32

33
34
38
42
44
46
48
49
51
53

55
56
58
61
63
65


Local and Dynamic Access of Ruby Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Local Variable Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Method Arguments Are Treated Like Local Variables . . . . . . . . . . . . . . . . . . . .
Dynamic Variable Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Climbing the Environment Pointer Ladder in C . . . . . . . . . . . . . . . . . . . . . . . .
Experiment 3-2: Exploring Special Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A Definitive List of Special Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4
Control Structures and Method Dispatch

67
67
70
71
74
75
79
81

83

How Ruby Executes an if Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
Jumping from One Scope to Another . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
Catch Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
Other Uses for Catch Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
Experiment 4-1: Testing How Ruby Implements for Loops Internally . . . . . . . . . . . . . . . . . 90
The send Instruction: Ruby’s Most Complex Control Structure . . . . . . . . . . . . . . . . . . . . 92
Method Lookup and Method Dispatch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
Eleven Types of Ruby Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Calling Normal Ruby Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Preparing Arguments for Normal Ruby Methods . . . . . . . . . . . . . . . . . . . . . . 95
Calling Built-In Ruby Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
Calling attr_reader and attr_writer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
Method Dispatch Optimizes attr_reader and attr_writer . . . . . . . . . . . . . . . . . 98
Experiment 4-2: Exploring How Ruby Implements Keyword Arguments . . . . . . . . . . . . . . 99
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

5
Objects and Classes

Inside a Ruby Object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Inspecting klass and ivptr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Visualizing Two Instances of One Class . . . . . . . . . . . . . . . . . . . . . . . . . . .
Generic Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Simple Ruby Values Don’t Require a Structure at All . . . . . . . . . . . . . . . . . . .
Do Generic Objects Have Instance Variables? . . . . . . . . . . . . . . . . . . . . . . .
Reading the RBasic and RObject C Structure Definitions . . . . . . . . . . . . . . . .
Where Does Ruby Save Instance Variables for Generic Objects? . . . . . . . . . .
Experiment 5-1: How Long Does It Take to Save a New Instance Variable? . . . . . . . . . .
What’s Inside the RClass Structure? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Inheritance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Class Instance Variables vs. Class Variables . . . . . . . . . . . . . . . . . . . . . . . .
Getting and Setting Class Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Constants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The Actual RClass Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Reading the RClass C Structure Definition . . . . . . . . . . . . . . . . . . . . . . . . . .
Experiment 5-2: Where Does Ruby Save Class Methods? . . . . . . . . . . . . . . . . . . . . . . . .
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

x 

Contents in Detail

105
106
107
108
109
110
111

112
113
113
115
118
120
122
124
125
127
127
131


6
Method Lookup and Constant Lookup
How Ruby Implements Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Modules Are Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Including a Module into a Class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Ruby’s Method Lookup Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A Method Lookup Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The Method Lookup Algorithm in Action . . . . . . . . . . . . . . . . . . . . . . . . . . .
Multiple Inheritance in Ruby . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The Global Method Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The Inline Method Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Clearing Ruby’s Method Caches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Including Two Modules into One Class . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Including One Module into Another . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A Module#prepend Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
How Ruby Implements Module#prepend . . . . . . . . . . . . . . . . . . . . . . . . . . .

Experiment 6-1: Modifying a Module After Including It . . . . . . . . . . . . . . . . . . . . . . . . .
Classes See Methods Added to a Module Later . . . . . . . . . . . . . . . . . . . . . .
Classes Don’t See Submodules Included Later . . . . . . . . . . . . . . . . . . . . . . .
Included Classes Share the Method Table with the Original Module . . . . . . . .
A Close Look at How Ruby Copies Modules . . . . . . . . . . . . . . . . . . . . . . . .
Constant Lookup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Finding a Constant in a Superclass . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
How Does Ruby Find a Constant in the Parent Namespace? . . . . . . . . . . . . .
Lexical Scope in Ruby . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Creating a Constant for a New Class or Module . . . . . . . . . . . . . . . . . . . . .
Finding a Constant in the Parent Namespace Using Lexical Scope . . . . . . . . .
Ruby’s Constant Lookup Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Experiment 6-2: Which Constant Will Ruby Find First? . . . . . . . . . . . . . . . . . . . . . . . . . .
Ruby’s Actual Constant Lookup Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . .
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7
The Hash Table: The Workhorse of Ruby Internals
Hash Tables in Ruby . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Saving a Value in a Hash Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Retrieving a Value from a Hash Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Experiment 7-1: Retrieving a Value from Hashes of Varying Sizes . . . . . . . . . . . . . . . . .
How Hash Tables Expand to Accommodate More Values . . . . . . . . . . . . . . . . . . . . .
Hash Collisions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Rehashing Entries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
How Does Ruby Rehash Entries in a Hash Table? . . . . . . . . . . . . . . . . . . . . .
Experiment 7-2: Inserting One New Element into Hashes of Varying Sizes . . . . . . . . . . .
Where Do the Magic Numbers 57 and 67 Come From? . . . . . . . . . . . . . . .
How Ruby Implements Hash Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Experiment 7-3: Using Objects as Keys in a Hash . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Hash Optimization in Ruby 2.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

133
135
135
136
138
139
140
141
142
143
143
144
145
146
150
151
152
152
153
154
155
156
157
158
159
160
162

162
163
165

167
169
169
171
172
174
174
175
176
177
180
181
183
187
189

Contents in Detail 

xi


8
How Ruby Borrowed a Decades-Old Idea from Lisp
Blocks: Closures in Ruby . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Stepping Through How Ruby Calls a Block . . . . . . . . . . . . . . . . . . . . . . . . .
Borrowing an Idea from 1975 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

The rb_block_t and rb_control_frame_t Structures . . . . . . . . . . . . . . . . . . . . .
Experiment 8-1: Which Is Faster: A while Loop or Passing a Block to each? . . . . . . . . . .
Lambdas and Procs: Treating a Function as a First-Class Citizen . . . . . . . . . . . . . . . . .
Stack vs. Heap Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A Closer Look at How Ruby Saves a String Value . . . . . . . . . . . . . . . . . . . . .
How Ruby Creates a Lambda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
How Ruby Calls a Lambda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The Proc Object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Experiment 8-2: Changing Local Variables After Calling lambda . . . . . . . . . . . . . . . . . .
Calling lambda More Than Once in the Same Scope . . . . . . . . . . . . . . . . . .
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

191
192
194
196
198
200
203
204
204
207
209
211
214
216
217

9
Metaprogramming219

Alternative Ways to Define Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Ruby’s Normal Method Definition Process . . . . . . . . . . . . . . . . . . . . . . . . . .
Defining Class Methods Using an Object Prefix . . . . . . . . . . . . . . . . . . . . . .
Defining Class Methods Using a New Lexical Scope . . . . . . . . . . . . . . . . . .
Defining Methods Using Singleton Classes . . . . . . . . . . . . . . . . . . . . . . . . .
Defining Methods Using Singleton Classes in a Lexical Scope . . . . . . . . . . . .
Creating Refinements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Using Refinements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Experiment 9-1: Who Am I? How self Changes with Lexical Scope . . . . . . . . . . . . . . . . .
self in the Top Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
self in a Class Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
self in a Metaclass Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
self Inside a Class Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Metaprogramming and Closures: eval, instance_eval, and binding . . . . . . . . . . . . . . .
Code That Writes Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Calling eval with binding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
An instance_eval Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Another Important Part of Ruby Closures . . . . . . . . . . . . . . . . . . . . . . . . . . .
instance_eval Changes self to the Receiver . . . . . . . . . . . . . . . . . . . . . . . . .
instance_eval Creates a Singleton Class for a New Lexical Scope . . . . . . . . .
How Ruby Keeps Track of Lexical Scope for Blocks . . . . . . . . . . . . . . . . . . .
Experiment 9-2: Using a Closure to Define a Method . . . . . . . . . . . . . . . . . . . . . . . . . . .
Using define_method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Methods Acting as Closures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

xii 

Contents in Detail

221
221
223
224
226
227
228
229
231
231
232
233
234
236
236
238
240
241
242
243
244
246
246
247
248


10
JRuby: Ruby on the JVM

251

Running Programs with MRI and JRuby . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
How JRuby Parses and Compiles Your Code . . . . . . . . . . . . . . . . . . . . . . . .
How JRuby Executes Your Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Implementing Ruby Classes with Java Classes . . . . . . . . . . . . . . . . . . . . . . .
Experiment 10-1: Monitoring JRuby’s Just-in-Time Compiler . . . . . . . . . . . . . . . . . . . . .
Experiment Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Using the -J-XX:+PrintCompilation Option . . . . . . . . . . . . . . . . . . . . . . . . . .
Does JIT Speed Up Your JRuby Program? . . . . . . . . . . . . . . . . . . . . . . . . . .
Strings in JRuby and MRI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
How JRuby and MRI Save String Data . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Copy-on-Write . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Experiment 10-2: Measuring Copy-on-Write Performance . . . . . . . . . . . . . . . . . . . . . . .
Creating a Unique, Nonshared String . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Experiment Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Visualizing Copy-on-Write . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Modifying a Shared String Is Slower . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11
Rubinius: Ruby Implemented with Ruby

273

The Rubinius Kernel and Virtual Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Tokenization and Parsing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Using Ruby to Compile Ruby . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Rubinius Bytecode Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Ruby and C++ Working Together . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Implementing Ruby Objects with C++ Objects . . . . . . . . . . . . . . . . . . . . . . .
Experiment 11-1: Comparing Backtraces in MRI and Rubinius . . . . . . . . . . . . . . . . . . . .
Backtraces in Rubinius . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Arrays in Rubinius and MRI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Arrays Inside of MRI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The RArray C Structure Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Arrays Inside of Rubinius . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Experiment 11-2: Exploring the Rubinius Implementation of Array#shift . . . . . . . . . . . .
Reading Array#shift . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Modifying Array#shift . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

12
Garbage Collection in MRI, JRuby, and Rubinius
Garbage Collectors Solve Three Problems . . . .
Garbage Collection in MRI: Mark and Sweep .
The Free List . . . . . . . . . . . . . . . . . .
MRI’s Use of Multiple Free Lists . . . . .
Marking . . . . . . . . . . . . . . . . . . . . .
How Does MRI Mark Live Objects? . .

.
.
.
.
.
.

.
.

.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.

.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

252
254
255
257
260
260

261
262
263
264
265
267
267
268
269
270
271

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.

.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

274
276
277
278
279
280
281
282

284
285
286
286
288
288
289
292

295
.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.

.
.
.
.
.

.
.
.
.
.
.

297
297
297
298
299
299

Contents in Detail 

xiii


Sweeping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Lazy Sweeping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The RVALUE Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Disadvantages of Mark and Sweep . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Experiment 12-1: Seeing MRI Garbage Collection in Action . . . . . . . . . . . . . . . . . . . . . .

Seeing MRI Perform a Lazy Sweep . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Seeing MRI Perform a Full Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Interpreting a GC Profile Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Garbage Collection in JRuby and Rubinius . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Copying Garbage Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Bump Allocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The Semi-Space Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The Eden Heap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Generational Garbage Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The Weak Generational Hypothesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Using the Semi-Space Algorithm for Young Objects . . . . . . . . . . . . . . . . . . .
Promoting Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Garbage Collection for Mature Objects . . . . . . . . . . . . . . . . . . . . . . . . . . .
References Between Generations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Concurrent Garbage Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Marking While the Object Graph Changes . . . . . . . . . . . . . . . . . . . . . . . . .
Tricolor Marking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Three Garbage Collectors in the JVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Experiment 12-2: Using Verbose GC Mode in JRuby . . . . . . . . . . . . . . . . . . . . . . . . . . .
Triggering Major Collections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

300
300
301
302
302
303
304

305
309
309
310
311
312
313
313
314
314
315
316
317
317
319
320
321
323
324
325

Index327

xiv 

Contents in Detail


Fore word

Oh, hi! I didn’t see you come in. I don’t want to be too forward, but let me
preface this by saying you should buy this book!
My name is Aaron Patterson, but my Internet friends call me “tenderlove.”
I am on both the Ruby core team and the Ruby on Rails core team, and I
did the technical review of this book. Does that mean you should listen
to me? No. Well, maybe.
Actually, when Pat approached me to do the technical review of this
book, I was so excited that my top hat fell off and I dropped my monocle in
my coffee! I knew about Pat’s previous work on Ruby Under a Microscope, and
the idea of making an updated and print version available made me really
happy. I think many developers are intimidated by Ruby’s internals and are
afraid to dive in. Quite often people ask me how they can learn about how
Ruby works under the hood or where to get started hacking on Ruby internals. Unfortunately I didn’t have a good answer for people—until now.
Pat’s style of writing, in combination with experimentation, makes Ruby
internals very approachable. The experiments are combined with explanations of Ruby’s internals such that you can easily understand why Ruby acts
the way it does with regard to behavior and performance. Next time you
encounter some behavior in your Ruby code, whether it be with performance, local variables and your environment, or even garbage collection,
this book won’t just tell you why your code behaves the way it does, but will
even tell you how.
If you’re someone who wants to start hacking on Ruby’s internals, or if
you just want to understand why Ruby acts the way it does without any handwaving, this is the book for you. I enjoyed this book, and I hope you will too.
Aaron Patterson
<3 <3 <3 <3



Acknowledgments

I could never have finished a project like this without
the support of many different people!

First of all, thanks to Satty Bhens and everyone else at McKinsey for
giving me the flexibility to write a book and keep my day job at a great
company. Alex Rothenberg and Daniel Higginbotham gave me invaluable advice, suffered through reading many early drafts, and helped me
throughout the process. Special thanks to Xavier Noria, who took an
interest in the project early on, gave me fantastic feedback on the entire
rough draft, and was also the inspiration behind Experiment 6-1. Santiago
Pastorino reviewed the rough draft as well. Jill Caporrimo, Prajakta Thakur,
Yvannova Montalvo, Divya Ganesh, and Yanwing Wong were my “proofreading SWAT team.” Self-publishing would have been much harder without
your help. Finally, without the constant encouragement and support Peter
Cooper has given me this year, I probably never would have attempted to
write this book. Thank you, Peter.
Thanks to everyone at No Starch Press for helping me bring an
expanded, updated version of Ruby Under a Microscope to print. The result is
a book I’m proud of and one the Ruby internals topic deserves. Thanks to
Julianne Jigour, my copyeditor. My writing has never been so clear and easy
to follow. Thank you, Riley Hoffman and Alison Law, for your editing advice
and for beautifully reproducing hundreds of diagrams for print. You’ve been
a pleasure to work with. Thanks to Charles Nutter for the technical help and
advice on JVM garbage collection. Special thanks to Aaron Patterson: This
is a more interesting and accurate book because of your great suggestions
and technical review. Finally, thanks to Bill Pollock for reading and editing every single line of text in the book. Your guidance and expertise have
allowed me to write a book I could never have dreamed of writing on my own.


What seems complex from
a distance is often quite
simple when you look
closely enough.

Introduction

At first glance, learning how to use Ruby can seem
fairly simple. Developers around the world find Ruby’s
syntax to be graceful and straightforward. You can
express algorithms in a very natural way, and then it’s
just a matter of typing ruby at the command line and
pressing enter, and your Ruby script is running.
However, Ruby’s syntax is deceptively simple; in fact, Ruby employs
sophisticated ideas from complex languages like Lisp and Smalltalk.
On top of this, Ruby is dynamic; using metaprogramming, Ruby programs
can inspect and change themselves. Beneath this thin veneer of simplicity,
Ruby is a very complex tool.
By looking very closely at Ruby—by learning how Ruby itself works
internally—you’ll discover that a few important computer science concepts
underpin Ruby’s many features. By studying these, you’ll gain a deeper
understanding of what is happening under the hood as you use the language. In the process, you’ll learn how the team that built Ruby intends for
you to use the language.


Ruby Under a Microscope will show you what happens inside Ruby when
you run a simple program. You’ll learn how Ruby understands and executes
your code, and with the help of extensive diagrams, you’ll build a mental
model of what Ruby does when you create an object or call a block.

Who This Book Is For
Ruby Under a Microscope is not a beginner’s guide to learning Ruby. I assume
you already know how to program in Ruby and that you use it daily. There
are already many great books that teach Ruby basics; the world doesn’t
need another one.

Although Ruby itself is written in C, a confusing, low-level language,
no C programming knowledge is required to read this book. Ruby Under a
Microscope will give you a high-level, conceptual understanding of how Ruby
works without your having to understand how to program in C. Inside this
book, you’ll find hundreds of diagrams that make the low-level details of
Ruby’s internal implementation easy to understand.
Note

Readers familiar with C will find a few snippets of C code that give a more concrete
sense of what’s going on inside Ruby. I’ll also tell you where the code derives from,
making it easier for you to start studying the C code yourself. If you’re not interested
in the C code details, just skip over these sections.

Using Ruby to Test Itself
It doesn’t matter how beautiful your theory is, it doesn’t matter
how smart you are. If it doesn’t agree with experiment, it’s wrong.
—Richard Feynman

Imagine that the entire world functioned like a large computer program.
To explain natural phenomena or experimental results, physicists like
Richard Feynman would simply consult this program. (A scientist’s dream
come true!) But of course, the universe is not so simple.
Fortunately, to discover how Ruby works, all we need to do is read its
internal C source code: a kind of theoretical physics that describes Ruby’s
behavior. Just as Maxwell’s equations explain electricity and magnetism,
Ruby’s internal C source code explains what happens when you pass an
argument to a method or include a module in a class.
Like scientists, however, we need to perform experiments to be sure our
hypotheses are correct. After learning about each part of Ruby’s internal
implementation, we’ll perform an experiment and use Ruby to test itself!

We’ll run small Ruby test scripts to see whether they produce the expected
output or run as quickly or as slowly as we expect. We’ll find out if Ruby
actually behaves the way theory says it should. And since these experiments
are written in Ruby, you can try them yourself.

xx   Introduction


Which Implementation of Ruby?
Ruby was invented by Yukihiro “Matz” Matsumoto in 1993, and the original,
standard version of Ruby is often known as Matz’s Ruby Interpreter (MRI).
Most of this book will discuss how MRI works; essentially, we’ll learn how
Matz implemented his own language.
Over the years many alternative implementations of Ruby have been
written. Some, like RubyMotion, MacRuby, and IronRuby, were designed to
run on specific platforms. Others, like Topaz and JRuby, were built using
programming languages other than C. One version, Rubinius, was built
using Ruby itself. And Matz himself is now working on a smaller version
of Ruby called mruby, designed to run inside another application.
I explore the Ruby implementations JRuby and Rubinius in detail in
Chapters 10, 11, and 12. You’ll learn how they use different technologies
and philosophies to implement the same language. As you study these alternative Rubies, you’ll gain additional perspective on MRI’s implementation.

Overview
In Chapter 1: Tokenization and Parsing, you’ll learn how Ruby parses
your Ruby program. This is one of the most fascinating areas of computer
science: How can a computer language be smart enough to understand the
code you give it? What does this intelligence really consist of?
Chapter 2: Compilation explains how Ruby uses a compiler to convert
your program into a different language before running it.

Chapter 3: How Ruby Executes Your Code looks at the virtual machine
Ruby uses to run your program. What’s inside this machine? How does it
work? We’ll look deep inside this virtual machine to find out.
Chapter 4: Control Structures and Method Dispatch continues the
description of Ruby’s virtual machine, looking at how Ruby implements
control structures such as if...else statements and while...end loops. It also
explores how Ruby implements method calls.
Chapter 5: Objects and Classes discusses Ruby’s implementation of
objects and classes. How are objects and classes related? What would we
find inside a Ruby object?
Chapter 6: Method Lookup and Constant Lookup examines Ruby
modules and their relationship to classes. You’ll learn how Ruby finds
methods and constants in your Ruby code.
Chapter 7: The Hash Table: The Workhorse of Ruby Internals
explores Ruby’s implementation of hash tables. As it turns out, MRI uses
hash tables for much of its internal data, not only for data you save in
Ruby hash objects.
Chapter 8: How Ruby Borrowed a Decades-Old Idea from Lisp reveals
that one of Ruby’s most elegant and useful features, blocks, is based on an
idea originally developed for Lisp.
In Chapter 9: Metaprogramming tackles one of the most difficult
topics for Ruby developers. By studying how Ruby implements metaprogramming internally, you’ll learn how to use metaprogramming effectively.
Introduction   xxi


Chapter 10: JRuby: Ruby on the JVM introduces JRuby, an alternative
version of Ruby implemented with Java. You’ll learn how JRuby uses the Java
Virtual Machine (JVM) to run your Ruby programs faster.
Chapter 11: Rubinius: Ruby Implemented with Ruby looks at one of
the most interesting and innovative implementations of Ruby: Rubinius.

You’ll learn how to locate—and modify—the Ruby code in Rubinius to see
how a particular Ruby method works.
Chapter 12: Garbage Collection in MRI, JRuby, and Rubinius concludes with a look at garbage collection (GC), one of the most mysterious
and confusing topics in computer science. You’ll see how Rubinius and
JRuby use very different GC algorithms from those used by MRI.
By studying all of these aspects of Ruby’s internal implementation,
you’ll acquire a deeper understanding of what happens when you use
Ruby’s complex feature set. Just as Antonie van Leeuwenhoek first saw
microbes and cells looking through early microscopes in the 1600s, by
looking inside of Ruby you’ll discover a wide array of interesting structures and algorithms. Join me on a fascinating behind-the-scenes look at
what brings Ruby to life!

xxii   Introduction



Your code has a long
road to take before
Ruby ever runs it.