The Search Engine Marketing Kit

The Search Engine Marketing Kit
(Chapter 1)
Thank you for downloading this excerpt of Dan Thies’s The Search Engine Marketing Kit.
This excerpt contains the Summary of Contents, Information about the Author, Expert
Reviewers, and SitePoint, Table of Contents, Preface, and a chapter of the kit.
We hope you find this information useful in evaluating The Search Engine Marketing Kit.
For more information on The Search Engine Marketing Kit and to order,
visit sitepoint.com
Summary of Contents of this Excerpt
About This Kit........................................................................... viii
1. Understanding Search Engines ................................................1
Index .........................................................................................254
Summary of Additional Kit Contents
2. Search Engine Optimization Basics .......................................41
3. Advanced SEO And Search Engine-Friendly Design.............82
4. Paying To Play: Pay-Per-Click And Paid Inclusion.............120
5. Running A Search Engine Marketing Business ...................155
6. Interviews .............................................................................196
7. Tools .....................................................................................222
A. Resources..............................................................................234
The Search Engine Marketing Kit
by Dan Thies
The Search Engine Marketing Kit
by Dan Thies
Copyright © 2005 SitePoint Pty. Ltd.
Cover Designer: Julian CarrollManaging Editor: Simon Mackie
Cover Illustrator: Lucas LicataEditor: Georgina Laidlaw
CD-ROM Designer: Alex WalkerExpert Reviewer: Ed Kohler
Expert Reviewer: Jill Whalen
Expert Reviewer: Gord Collins

Printing History:
First Edition: March 2005
Notice of Rights
All rights reserved. No part of this kit may be reproduced, stored in a retrieval system or transmitted in
any form or by any means, without the prior written permission of the publisher, except in the case of
brief quotations embodied in critical articles or reviews.
Notice of Liability
The author and publisher have made every effort to ensure the accuracy of the information herein.
However, the information contained in this kit is sold without warranty, either express or implied. Neither
the authors and SitePoint Pty. Ltd., nor its dealers or distributors will be held liable for any damages to
be caused either directly or indirectly by the instructions contained in this kit, or by the software or
hardware products described herein.
Trademark Notice
Rather than indicating every occurrence of a trademarked name as such, this kit uses the names only in
an editorial fashion and to the benefit of the trademark owner with no intention of infringement of the
trademark.
Published by SitePoint Pty. Ltd.
424 Smith Street Collingwood
VIC Australia 3066.
Web: www.sitepoint.com
Email:
ISBN 0-9752402–5-0
Printed and bound in the United States of America
About The Author
Dan Thies lives in Frisco, Texas with his wife, two sons, two cats, and a very hyperactive mini-
ature pinscher. He has been a student and practitioner of search engine marketing since the
earliest days of the industry. Since 2001, he has been an active writer, speaker, and teacher for
beginners and professionals alike.
He is a member of the Executive Committee of SeoPros, the Organization of Search Engine
Optimization Professionals; a membership committee member and volunteer for SEMPO (the

Search Engine Marketing Professional Organization); and a frequent speaker at Jupiter Media’s
Search Engine Strategies conferences.
About The Expert Reviewers
Ed Kohler is president of Haystack In A Needle, a Web marketing firm based in Minneapolis,
MN, offering pay-per-click advertising, search engine optimization, and email marketing con-
sulting services.
Jill Whalen of High Rankings is an internationally recognized search engine optimization con-
sultant and host of the free weekly High Rankings Advisor search engine marketing newsletter
and forum. She is the author of the handbook The Nitty-gritty of Writing for the Search Engines.
Gord Collins owns Bay Street Search Engine Optimization, an SEO company in Toronto. He
has been an SEO specialist since 1998 and has authored two books on the subject.
About SitePoint
SitePoint specializes in publishing fun, practical, and easy-to-understand content for Web
professionals. Visit to access our books, newsletters, articles and
community forums.
This kit is dedicated to my wife, Gina, and
my sons, Jeremy and Jordan. Without their
support and extreme patience I would never
have been able to complete this work. In fact,
without them, there isn’t much point in getting
out of bed!
Table of Contents
About This Kit ................................................................................................... viii
Who Should Read This Kit? ................................................................................. ix
What’s in This Kit? ............................................................................................... ix
What’s on the CD-ROM? ..................................................................................... xi
Your Feedback ..................................................................................................... xiii
Acknowledgements .............................................................................................. xiii
Getting Started .................................................................................................... xiii
1. Understanding Search Engines ......................................................................... 1

A Brief History of the Search Engine ..................................................................... 1
The Early Days of Web Search ....................................................................... 2
The Great Search Engine Explosion ............................................................... 2
Google Dominates, the Field Narrows ........................................................... 3
Anatomy of a Web Search Portal .......................................................................... 5
Crawler-Based (Organic) Listings ................................................................... 6
Sponsored (Pay-Per-Click) Listings ................................................................ 7
Directory (Human-Edited) Listings ................................................................ 8
Other Listings .................................................................................................. 9
Search Engine Marketing Defined ....................................................................... 10
The Crawling Search Engines .............................................................................. 11
Major Tasks Handled by Search Engines ..................................................... 11
The Crawling Phase: How Spiders Work ..................................................... 12
Scheduling: How Search Engines Set Priorities ........................................... 17
Parsing and Caching ...................................................................................... 18
Results of the Crawling Phase ....................................................................... 19
Indexing: How Content is Analyzed ............................................................. 20
Link Analysis ................................................................................................. 22
How Queries Are Processed .......................................................................... 28
Ranking and Retrieval Strategies .................................................................. 31
Other Considerations .................................................................................... 32
What Search Engines Want ......................................................................... 33
Snapshot of the Search Market .................................................................... 34
The Future of Search ............................................................................................ 37
Localization ................................................................................................... 37
Context and Personalization ......................................................................... 39
Structure and the Semantic Web ................................................................. 39
Summary .............................................................................................................. 40
2. Search Engine Optimization Basics ................................................................. 41
The Process and Craft of SEO ............................................................................. 41

Phase 1: Keyword Strategy .................................................................................. 42
Understanding the Keyword Hierarchy ....................................................... 42
Step 1–Keyword Discovery ........................................................................... 47
Step 2–Keyword Research and Metrics ........................................................ 48
Step 3–Keyword Selection ............................................................................ 57
Phase 2: Site Design and Structure ..................................................................... 58
Mapping Search Terms to Content .............................................................. 58
Crawlability and Site Navigation ................................................................. 61
Phase 3: Optimizing Web Pages .......................................................................... 63
Key Page Elements ........................................................................................ 63
Page Layout ................................................................................................... 66
SEO Copywriting .......................................................................................... 66
Keyword Density and Overdoing It ............................................................. 67
HTML Issues ................................................................................................. 68
Phase 4: Link Building ......................................................................................... 69
Managing the External Profile ...................................................................... 69
Directory Submissions .................................................................................. 71
Other One-Way Links .................................................................................. 72
Link Exchanges and Partnerships ................................................................. 73
Keeping it Relevant ....................................................................................... 73
Local Sites, Local Links ................................................................................. 74
Linking Out ................................................................................................... 74
Phase 5: Getting Indexed ..................................................................................... 74
The Easy Way: Links and Crawlability ........................................................ 75
Submission and Submission “Services” ........................................................ 75
Paid Inclusion Options ................................................................................. 75
For Indexing Problems, Look at the Site ...................................................... 76
Search Engine Spam ............................................................................................. 76
How Search Engines Define Spam ............................................................... 77
Cloaking and Variable Delivery .................................................................... 78

The Rules Have Never Changed ................................................................... 79
Best Practice SEO ......................................................................................... 79
Summary .............................................................................................................. 81
3. Advanced SEO And Search Engine-Friendly Design ......................................... 82
Harmonizing Design and SEO ............................................................................. 82
Designing with Tables ................................................................................... 83
Designing with CSS ...................................................................................... 84
The Blended Approach ................................................................................. 85
Dynamic Text Replacement .......................................................................... 86
Search Engines and Frames .................................................................................. 86
Why Designers Use Frames .......................................................................... 86
How Search Engines Handle Frames ............................................................ 87
Solution: a Self-Referencing Frameset .......................................................... 87
Site Navigation ..................................................................................................... 87
Integrating Text Navigation ......................................................................... 88
Developing a Site Map ................................................................................. 88
“Crawlable” DHTML and JavaScript Menus ............................................... 89
iiiOrder The Search Engine Marketing Kit now and get $150 of PPC advertising credit
The Search Engine Marketing Kit
Pop-Up Windows .......................................................................................... 90
Forced Cookies and Form-Based Navigation ............................................... 91
Working with Flash .............................................................................................. 91
Why Designers Like Flash ............................................................................ 91
Search Engines and Flash .............................................................................. 92
Solution: Mixing Flash with HTML ............................................................. 92
Solution: Using <noembed> ....................................................................... 93
Warning: Heavy Content Ahead! ........................................................................ 93
Duplicate Content: a Definition ................................................................... 94
HTTP Headers: a Peek Under the Hood ..................................................... 95
Dynamic Site Issues and Opportunities .............................................................. 98

Content Management Systems ..................................................................... 99
Shopping Carts ............................................................................................ 100
Link Directories .......................................................................................... 101
Database and Server Error Handling ......................................................... 101
URL Rewriting ............................................................................................ 103
Duplicate Content .............................................................................................. 104
Spider Control and robots.txt ..................................................................... 104
Diagnosing Duplication .............................................................................. 108
Sessions and Cookies .................................................................................. 109
www.example.com vs example.com ............................................................ 109
Checking and Fixing Scripts and Variables ................................................ 110
Empty Pages ................................................................................................ 111
Server and Domain Issues .................................................................................. 112
Redirection .................................................................................................. 112
Custom Error Pages .................................................................................... 114
Managing Multiple Domain Names ........................................................... 114
Moving a Domain ....................................................................................... 117
Watching the Clock .................................................................................... 118
The Importance of Reliable Hosting .......................................................... 119
Summary ............................................................................................................ 119
4. Paying To Play: Pay-Per-Click And Paid Inclusion .......................................... 120
Introduction to Pay-Per-Click ............................................................................ 120
The Pay-Per-Click Marketplace ......................................................................... 122
Major Players: AdWords and Overture ...................................................... 123
Minor Pay-Per-Click Services ..................................................................... 128
The Pay-Per-Click Process ................................................................................. 130
Triggering: Targeting Ad Displays .............................................................. 132
Click-Through: Qualifying and Motivating Visitors .................................. 138
Landing Pages and Landing Zones ............................................................. 144
Interaction: Improving Website Conversion .............................................. 147

Measurement and Reporting ...................................................................... 150
Other Pay-To-Play Programs ............................................................................. 150
Paid Inclusion ............................................................................................. 151
ivOrder The Search Engine Marketing Kit now and get $150 of PPC advertising credit
The Search Engine Marketing Kit
Trusted Feed ............................................................................................... 151
Paid Directories ........................................................................................... 152
The Future of Paid Search ................................................................................. 152
Supply and Demand Issues ........................................................................ 153
Advertisers Demand Greater Control ......................................................... 153
Big Budgets ................................................................................................. 153
Summary ............................................................................................................ 154
5. Running A Search Engine Marketing Business .............................................. 155
Building an SEM Business ................................................................................. 155
Essential Functions and Skills .................................................................... 156
Processes and Tools .................................................................................... 160
People .......................................................................................................... 163
Are You In? ................................................................................................. 165
Getting Business ................................................................................................. 165
Understanding the Selling Cycle ................................................................ 166
What Prospects Look For ........................................................................... 169
Gaining Experience and References ............................................................ 171
Finding Prospects ........................................................................................ 173
Consultative Selling .................................................................................... 175
Effective Proposals ...................................................................................... 176
Doing Business ................................................................................................... 179
Pricing ......................................................................................................... 179
What to Sell ................................................................................................ 184
“Difficult” Clients ....................................................................................... 185
Developing SEM Strategy .................................................................................. 186

Assessment .................................................................................................. 187
Goal Setting ................................................................................................ 188
Planning: Keyword Strategy ....................................................................... 189
Planning: Linking Strategy ......................................................................... 192
Being a Professional ........................................................................................... 193
Lifetime Value: Results Matter ................................................................... 193
Methods Matter .......................................................................................... 194
Lifelong Learning ........................................................................................ 194
What’s Coming Up? .......................................................................................... 194
6. Interviews ................................................................................................... 196
Andy Beal, Keyword Ranking ............................................................................ 197
Greg Jarboe, SEO-PR ......................................................................................... 201
Jill Whalen, High Rankings ............................................................................... 205
John Slade, Overture .......................................................................................... 209
Scottie Claiborne, Karcher Group ..................................................................... 214
7. Tools ........................................................................................................... 222
SEO Tools and Services ..................................................................................... 222
Keyword Discovery ..................................................................................... 222
vOrder The Search Engine Marketing Kit now and get $150 of PPC advertising credit
The Search Engine Marketing Kit
Wordtracker ................................................................................................ 223
Overture Search Term Suggestion Tool ..................................................... 223
Position Technologies ................................................................................. 224
Priority Submit ........................................................................................... 224
Arelis ............................................................................................................ 224
SEO Elite ..................................................................................................... 225
Mozilla Firefox ............................................................................................ 225
PPC Tools and Services ..................................................................................... 226
Atlas OnePoint ............................................................................................ 226
BidRank and BidRank Plus ........................................................................ 227

Findwhat and eSpotting ............................................................................. 227
Overture Keyword Selector Tool ................................................................ 227
Who’s Clicking Who? ................................................................................. 228
Analysis Tools .................................................................................................... 228
Clicktracks ................................................................................................... 229
NetTracker .................................................................................................. 229
Omniture SiteCatalyst ................................................................................ 229
Webtrends ................................................................................................... 230
DigitalPoint ................................................................................................. 230
Advanced Web Ranking ............................................................................. 231
Other Tools and Services ................................................................................... 231
SEO-PR ....................................................................................................... 231
Hitwise and Comscore ................................................................................ 232
eLance.com .................................................................................................. 232
SEO Research Labs ..................................................................................... 233
Alliance Link ............................................................................................... 233
A. Resources .................................................................................................... 234
PageRank in Focus ............................................................................................. 234
PageRank Resources .................................................................................... 235
White Paper: “The Classification of Search Engine Spam” .............................. 236
Abstract ....................................................................................................... 236
Content Spam ............................................................................................. 238
Meta Spam .................................................................................................. 239
Links ............................................................................................................ 241
Redirects ...................................................................................................... 243
Agent-Based Delivery and Agent-Based Spam ........................................... 243
IP Delivery and IP Cloaking ....................................................................... 245
Conclusion .................................................................................................. 246
Resources ............................................................................................................ 247
Recommended Reading .............................................................................. 247

The Big Directory List ....................................................................................... 248
Websites ............................................................................................................. 250
SEO/SEM Forums ...................................................................................... 250
Other Resources .......................................................................................... 251
viOrder The Search Engine Marketing Kit now and get $150 of PPC advertising credit
The Search Engine Marketing Kit
SEM Organizations, Marketplaces, and Directories ......................................... 253
Index .......................................................................................................................... 254
viiOrder The Search Engine Marketing Kit now and get $150 of PPC advertising credit
The Search Engine Marketing Kit
About This Kit
Search engine marketing (SEM) is one of the hottest topics in the marketing world
today. Even traditional “offline” marketing agencies are beginning to understand the
powerful ways in which search engine marketing can help achieve their objectives. For
those in the business of providing Web design, search engine marketing, and Website
promotion services, this is great news, but it does come at a price.
As more individuals and organizations begin to use search engine optimization (SEO)
and pay-per-click advertising (PPC) as part of their Website’s marketing strategy,
competition for space in the search engine results will become increasingly fierce. In
order to compete effectively, search engine marketers-be they individual site owners, or
professional consultants-must increase their knowledge and skills.
This kit is intended to fill a large gap that exists between the many helpful but basic
introductory texts for beginners, and the often expensive conferences, training programs,
and workshops designed for full time search engine marketers. The bottom line is that
nobody has taken the knowledge of the professional search engine marketer, and put
it in writing. That’s what I’ve tried to do here.
While many things will change in the search engine marketing world, and search engines
will continue to adapt their algorithms to deliver more useful information to searchers,
some things remain static. In this kit, I hope to have captured those lasting truths, and
provided a sound reference to an increasingly complex field.

The Search Engine Marketing Kit provides a fantastic road map for your successful journey
into search engine marketing. It provides considerable detailed information that will
enable you to affect your site’s position in search results. What you do with this
knowledge is up to you, but I do hope you’ll pay attention to the strategic (and perhaps
philosophical) aspects of the field as well.
Beyond the important how-to questions, and the technical information, I believe that
it’s important for search engine marketers to better understand both the search engines
and their users. There’s a great deal of conflict between today’s search engine marketers
and the search engines on which they rely, but this doesn’t have to be the case.
Search engine marketing should not be carried out in a vacuum, and those practitioners
who ignore other pertinent aspects of the Web and the user experience will ultimately
fail. The reason is simple: a better Website will generate greater profits, and will ulti-
mately have available more resources with which to compete for search rankings and
traffic.
Those who embrace the concept that the entire Website must be optimized in every
respect will be the real winners in the long run. A Website should be optimized not just
for the benefit of search engines, but for the interests of users and the business or organ-
ization behind the site as well.
The primary mission of search engines is to deliver relevant search results to users.
Relevance is also the goal of the searcher. If you focus your efforts on enhancing the
relevance of your Website, and dedicate your search engine marketing to reaching a
well-targeted audience, then you will experience the success you deserve.
Who Should Read This Kit?
This kit is intended for those who already have some knowledge of Website design and
development. The information presented is, in some cases, very basic; in others, it’s far
more advanced. This is necessary—the kit aims to teach you the skills that professional
search engine marketers use to work their magic. It therefore involves a natural gradu-
ation from the basics to more advanced material.
Although the primary audience is the Web professional (or skilled amateur), even those
who do not participate in the actual design of Websites can learn a lot from this kit.

Different chapters and sections will appeal to different interests—designers, IT folks,
marketing people, and so on. Site owners and Web professionals who must fulfil all of
these roles will find this kit especially useful, because it encompasses so many aspects
of the industry.
Readers who find some aspects heavily technical or overwhelming should take heart,
because the application of even the most basic elements of search engine optimization
to a site can yield substantial results.
The advanced techniques are presented for those readers who require more than the
basics, and those who want to develop their expertise over time. Don’t be afraid to seek
professional help or expert guidance if it’s needed. By the time you finish this kit, you’ll
be an expert yourself-even if you don’t fully grasp the technical details.
What’s in This Kit?
Chapter 1: Understanding Search Engines
Do you think you understand how search engines work? So did I … until I started
doing a little in-depth research for this kit! In the first chapter, we’ll take a revealing
peek under the hood of modern search engines. We’ll see where search results come
from, how search engines crawl the Web, and how Web pages are ranked.
Chapter 2: Search Engine Optimization Basics
Now that we understand how search engines work, it’s time to look at how you can
influence your site’s position in search results. This chapter covers the basics of
ixOrder The Search Engine Marketing Kit now and get $150 of PPC advertising credit
About This Kit
search engine optimization including keyword strategy, optimizing page layout, and
effective site structure.
Chapter 3: Advanced SEO and Search Engine-Friendly Design
It’s time to move beyond the basics! This chapter is a little more technical, but ne-
cessarily so. The understanding you’ll have developed from the first two chapters
will serve you well as we explore such advanced topics as duplicate content, Web
server issues, content management systems, and moving domains.
Chapter 4: Paying To Play: Pay-Per-Click and Paid Inclusion

In Chapter 4, we’ll take an in-depth look at the world of pay-per-click (PPC) advert-
ising and other pay-to-play options. If you feel that you can’t afford to use PPC to
promote your Website, think again! Here, you’ll discover many new ways to optimize
PPC campaigns to deliver a greater return on investment.
Chapter 5: Running A Search Engine Marketing Business
As I mentioned in the introduction to this kit, the current boom in search engine
marketing represents a tremendous business opportunity for Web professionals. In
this chapter, we’ll look at the various elements involved in building your own search
engine marketing business, or integrating search engine marketing services into your
current offering.
Chapter 6: Interviews
In the course of writing this kit, I spoke with dozens of search engine marketing
professionals. In this chapter, I’ve collected six interviews with a range of folks who
provide expert perspectives on topics ranging from SEO strategy and pay-per-click,
to running a successful search engine marketing business.
Chapter 7: Tools
The world of search engine marketing is simply filled with companies offering services,
software, and other tools. No search engine marketer can do his or her job without
a substantial number of these offerings. In this chapter, I’ll review a variety of tools,
focusing on the best that are currently available.
Appendix A: Resources
The Appendix provides references to a range of quality resources—some specifically
related to search engine marketing, others dealing with broader questions of the
Web and its users—that will allow budding search engine marketers to expand their
perspectives and boost their knowledge.
xOrder The Search Engine Marketing Kit now and get $150 of PPC advertising credit
About This Kit
What’s on the CD-ROM?
The CD-ROM included with this kit contains several useful tools for search engine
marketers and professional SEM consultants.

Client Management Form (MS Word)
This form is intended to help professional SEO/SEM consultants manage information
about a client. It includes a contact form for new leads, an intake form for new clients,
and a business assessment form.
SEM Sales Presentation (MS PowerPoint)
Another tool for professionals, this PowerPoint presentation template will allow you
to speak to the value of search engine marketing, the advantages and disadvantages
of SEO and PPC, reasons to hire a professional, and the overall process involved in
an SEM campaign.
SEM Process Flowchart (MS Visio/PDF)
This flowchart provides a big-picture overview of the search engine marketing process
as described in this kit. This can be used by professionals, in-house search engine
marketers, or do-it-yourselfers—anyone who needs to communicate what’s involved
in search engine marketing.
Keyword Research Worksheet (MS Excel)
This is the same keyword research worksheet that my own company delivers to its
clients. The major advantages of this worksheet are that it allows you easily to make
a weighted popularity calculation for search terms based on their actual relevance,
and estimates monthly traffic for the top ten listings on major search engines.
Link Partnership Tracker (MS Excel)
This worksheet represents a very simple and effective tool for tracking link exchanges,
promotions, and partnerships. Keeping this information in Excel allows you to sort
and filter the data quickly, and perform mail merges with Microsoft Outlook.
Directory Submission Tracker (MS Excel)
Another simple Excel tool for tracking directory submissions: the directory, the title
and description used, the date of submission, and any associated costs can be noted
in this tracker. The spreadsheet includes my own seed list of general-purpose direct-
ories.
xiOrder The Search Engine Marketing Kit now and get $150 of PPC advertising credit
About This Kit

Site Review Checklist (PDF)
Intended mainly for the professionals, but useful for all search engine marketers,
this site review checklist covers the main points that you’ll want to address prior to
beginning an SEO/SEM campaign.
SEM Proposal Sample (MS Word)
Professionals are often asked to deliver proposals to prospects and clients, and un-
fortunately, many such documents fall far short of what’s required to sell SEM ser-
vices. This sample proposal exemplifies several key points of effective proposal
writing. It begins by addressing the client’s business issues, maintains negotiating
flexibility, and ties the proposed SEM activities back to business outcomes.
SEM Service Agreement Sample (MS Word)
When you start selling your services to clients, you’ll need an agreement that sets
out the work you’ll be doing, how much you’ll be paid, and the responsibilities of
both parties. This is a basic “bare bones” agreement that you can use to gain ideas
for your own contracts. Be sure to seek professional legal advice before entering into
any agreement.
Rates, Pricing, & ROI Calculator (MS Excel)
This tool is intended for all search engine marketers, to help make realistic assess-
ments of the true value of an SEM campaign. Set hourly rates for each activity, es-
timate the amount of work required for the campaign, and see how different outcomes
affect the overall return on investment (ROI).
SEM Project Planner (MS Excel)
Another tool that all search engine marketers can use, this Excel spreadsheet contains
a simple project planning tool. Identify the tasks involved in each phase of the
campaign, assign responsibilities, and schedule the work. Project planning is especially
important when multiple teams are involved, for instance, when an SEO consultant
works with a site designer.
Web CEO (Application)
Web CEO is suite of software tools, including a keyword researcher, site optimization
tool, and link checker, to help you to promote your site in search engines, analyze

your visitors, and easily maintain your Website at optimal quality. We’ve included
the free version of Web CEO on the CD-ROM so that you can take it for a test-
drive.
xiiOrder The Search Engine Marketing Kit now and get $150 of PPC advertising credit
About This Kit
Your Feedback
If you have questions about any of the information presented in this kit, your best
chance of a quick response is to post your query in the SitePoint Forums.[1] If you have
any feedback, questions, or wish to alert us to mistakes, email
Suggestions for improvement, as well as notices of any mistakes you may find, are espe-
cially welcome.
Acknowledgements
I would like to sincerely thank the folks at SitePoint (Georgina, Matt, and Simon) for
all their help, and for giving me an opportunity to put my knowledge into writing.
Thanks are also due to the kit’s technical editors (Ed, Gord, and Jill) who made so many
valuable contributions to the final product.
Getting Started
I hope you enjoy using this kit! Please note that all the information presented here—from
case studies to documentation, be it printed or in electronic format—is protected under
international copyright laws.
SitePoint Pty. Ltd. reserves all rights to the content presented in The Search Engine
Marketing Kit, which may not be copied, reproduced, or redistributed, in whole, or in
part, under any circumstances, without their express written permission.
Also, while every effort has been made to ensure the accuracy of the information and
documents herein, neither the authors, nor SitePoint Pty. Ltd. will be held liable for
any damages caused by the instructions or documents contained in The Search Engine
Marketing Kit.
What we’re saying here is that it’s up to you to decide what information and resources
suit your business, and to seek professional advice if you’re unsure about any of the
topics covered in The Search Engine Marketing Kit.

That’s the legals out of the way. Let’s get started!
[1] />xiiiOrder The Search Engine Marketing Kit now and get $150 of PPC advertising credit
About This Kit
xiv
Understanding Search Engines
1
Every day, millions of people turn to their computers and look for information on the
Web. And, more often than not, they use a search engine to find that information. It’s
estimated that more than 350 million English language Web searches are conducted
every day!
In this chapter, I’ll offer a brief history of search engines, explaining the different com-
ponents of search portals, and how people use them. We’ll dive into the inner workings
of the major crawling search engines. Finally, we’ll conclude with a review of today’s
search engine landscape, and some thoughts on the future of search engine technology.
You may be tempted to skip right past this chapter to the nitty gritty, but, trust me:
this is required reading. Understanding where search results come from, how search
engines work, and where the industry is headed is essential if you’re to make successful
search engine marketing decisions now and in the future.
In the search engine optimization business, one of the key distinctions between amateurs
and professionals is that a professional truly understands how the system works, and why.
An amateur might learn to tweak a page’s content and call it “optimized,” but a professional
is capable of explaining the rationale behind their every action, and adapting to changing
industry conditions without radically altering their methods.
A Brief History of the Search Engine
The World Wide Web was born in November, 1990, with the launch of the first Web
server (and Web page) hosted at the CERN research facility in Switzerland. Not surpris-
ingly, the purpose of the first Web page was to describe the World Wide Web project.
At the time, no search engine was needed—you could literally read the entire contents
of the World Wide Web in less than an hour.
By early 1993, the stage was set for the Web explosion. In February of that year, the

first (alpha) release of the NCSA Mosaic graphical browser provided a client application
that, by the end of the year, was available on all major desktop computing platforms.
The Netscape browser, based on Mosaic, was released in 1994. By this time, dial-up
Internet access had become readily available and was cheap. The Web was taking off!
The Early Days of Web Search
Even though the combination of cheap dial-up access and the Mosaic browser had made
the Web semi-popular, there was still no way to search the growing collection of hypertext
documents available online. Most Web pages were basically collections of links, and a
popular pastime of Web users was to share their bookmark files.
This isn’t to say that attempts weren’t made to bring order to the swiftly growing chaos.
The first automated Web crawler, or robot, was the World Wide Web Wanderer created
by MIT student Mathew Gray. This crawler did little more than collect URLs, and was
largely seen as a nuisance by the operators of Web servers. Martjin Koster created the
first Web directory, ALIWeb, in late 1993, but it, like the Wanderer, met with limited
success.
In February 1993, six Stanford graduate students began work on a research project
called Architext, using word relationships to search collections of documents. By the
middle of that year, their software was available for site search. More robots had appeared
on the scene by late 1993, but it wasn’t until early 1994 that searching really came into
its own.
The Great Search Engine Explosion
1994 was a big year in the history of Web search. The first hierarchical directory, Galaxy,
was launched in January and, in April, Stanford students David Filo and Jerry Yang
created Yet Another Hierarchical Officious Oracle, better known as Yahoo!.
During that same month, Brian Pinkerton at the University of Washington released
WebCrawler. This, the first true Web search engine, indexed the entire contents of Web
pages, where previous crawlers had indexed little more than page titles, headings, and
URLs. Lycos was launched a few months later.
By the end of 1995, nearly a dozen major search engines were online. Names like
MetaCrawler (the first metasearch engine), Magellan, Infoseek, and Excite (born out

of the Architext project) were released into cyberspace throughout the year. AltaVista
arrived on the scene in December with a stunningly large database and many advanced
features, and Inktomi debuted the following year.
Over the next few years, new search engines would appear every few months, but many
of these differed only slightly from their competitors. Yet the occasional handy innovation
2Order The Search Engine Marketing Kit now and get $150 of PPC advertising credit
Chapter 1: Understanding Search Engines
would find its way into practical use. Here are a few of the most successful ideas from
that time:
❑
GoTo (now Overture) introduced the concept of pay-per-click (PPC) listings in 1997.
Instead of ranking sites based on some arcane formula, GoTo allowed open bidding
for keywords, with the top position going to the highest bidder. All major search
portals now rely on PPC listings for the bulk of their revenues.
❑
Metasearch engines, which combine results from several other search engines, prolif-
erated for a time, driven by the rise of pay-per-click systems and the inconsistency
of results among the major search engines. Today, new metasearch engines are rarely
if ever seen, but those that remain possess a loyal following. The current crop of
metasearch engines display mostly pay-per-click listings.
❑
The Mining Company (now About) launched in February 1997, using human experts
to create a more exclusive directory. Many topic-specific (vertical) directories and
resource sites have been created since, but About remains a leading resource.
❑
DirectHit introduced the concept of user feedback in 1998, allocating a higher
ranking to sites whose listings were clicked by users. DirectHit’s data influenced the
search results on many portals for a long time, but, because of the system’s suscept-
ibility to manipulation, none of today’s search portals openly use this form of feed-
back. DirectHit was later acquired by Ask Jeeves (now Ask), and user behavior may

well be factored into the Ask/Teoma search results we see today.
❑
Pay-to-play was introduced, as search engines and directories sought to capitalize
on the value of their editorial listings. The LookSmart and Yahoo! directories began
to charge fees for the review and inclusion of business Websites. Inktomi launched
“paid inclusion” and “trusted feed,” allowing site owners to ensure their inclusion
(subject to editorial standards) in the Inktomi search engine.
❑
The examination of linking relationships between pages began in earnest, with
AltaVista and other search engines adding “link popularity” to their ranking al-
gorithms. At Stanford University, a research project created the Backrub search en-
gine, which took a novel approach to ranking Web pages.
Google Dominates, the Field Narrows
The Backrub search engine eventually found its way into the public consciousness as
Google. By the time the search engine was officially launched as Google in September
1998, it had already become a very popular player.
The development of search engines since that time has been heavily influenced by
Google’s rise to dominance. More than any other search portal, Google has focused on
the user experience and quality of search results. Even at the time of its launch, Google
3Order The Search Engine Marketing Kit now and get $150 of PPC advertising credit
Chapter 1: Understanding Search Engines
offered users several major improvements, some of which had nothing to do with the
search results offered.
One of the most appealing aspects of Google was its ultra-simple user interface. Advert-
ising was conspicuously absent from Google’s homepage—a great advantage in a market
whose key players typically adorned their pages with multiple banners—and the portal
took only a few seconds to load even on a slow dial-up connection. Users had the option
to search normally, but a second option, called “I’m Feeling Lucky,” took users directly
to the page that ranked at the top of the results for their search.
Like its homepage, Google’s search results took little time to appear and carried no ad-

vertising. By the time Google began to show a few paid listings through the AdWords
service in late 2000, users didn’t mind: Google had successfully established itself as the
leading search portal and, unlike many other search engines, it didn’t attempt to hide
paid advertising among regular Web search results.
Many other search portals recognized the superiority of Google’s search results, and the
loyalty that quality generated. AOL and Yahoo! made arrangements to display Google’s
results on their own pages, as did many minor search portals. By the end of 2003, it
was estimated that three-quarters of all Web searches returned Google-powered results.
Within a few years, the near-monopoly that Google achieved in 2003 will be recognized
as a high water mark, but the development of this search engine is by no means finished.
The years 2001–2003 saw a series of acquisitions that rapidly consolidated the search
industry into a handful of major players. Yahoo! acquired the Inktomi search engine in
March 2003; Overture acquired AltaVista and AllTheWeb a month later; Yahoo! an-
nounced the acquisition of Overture in August 2003.
In 2004, a new balance of power took shape:
❑
Yahoo! released its own search engine powered by a fusion of the AltaVista, Inktomi,
and AllTheWeb technology they acquired in 2003. Yahoo! stopped returning Google
search results in January 2004.
❑
Google’s AdWords and AdSense systems, which deliver pay-per-click listings to
search portals and Websites respectively, grew dramatically. Google filed for an initial
public offering (IPO).
❑
The popularity of the Ask search portal, powered by the innovative Teoma search
engine, steadily increased. Like most portals that Yahoo! doesn’t own, Ask uses
Google’s AdWords for paid listings.
❑
The 800-lb gorilla of the computing world, Microsoft, announced plans for its own
search engine, releasing beta versions for public use in January and June of 2004,

4Order The Search Engine Marketing Kit now and get $150 of PPC advertising credit
Chapter 1: Understanding Search Engines
and formally launching the service in February 2005. Microsoft now offers MSN
search results on the MSN portal.
That’s enough history for now. We’ll take a closer look at the current search engine
landscape a little later in this chapter, when I’ll introduce you to the major players, and
explain how all this will affect your search engine strategy.
Anatomy of a Web Search Portal
Today, what we call a search engine is usually a much more complex Web search portal.
Search portals are designed as starting points for users who need to find information
on the Web. On a search portal, a single site offers many different search options and
services:
❑
AOL’s user interface gives users access to a wide variety of services, including email,
online shopping, chat rooms, and more. Searching the Web is just one of many
choices available.
❑
MSN features Web search, but also shows news, weather, links to dozens of sites
on the MSN network, and offers from affiliated sites like Expedia, ESPN, and others.
❑
Yahoo! still features Web search prominently on its homepage, but also offers a
dazzling array of other services, from news and stock quotes to personal email and
interactive games.
❑
Even Google, the most search-focused portal, offers links to breaking news, Usenet
discussion groups, Froogle shopping search, a proprietary image search system, and
many other options.
In this section, we’ll examine the makeup of a typical search engine results page (SERP).
Every portal delivers search results from different data sources. The ways in which these
sources are combined and presented to the user is what gives each Web search portal

its own unique flavor.
Changes to the way a major portal presents its search results can have a significant impact
on the search engine strategy you craft for your Website. As we look at the different
sources of search results, and the ways in which those results are handled by individual
portals, I’ll offer examples to illustrate this point.
A typical search engine results page has three major components: crawler-based listings,
sponsored listings, and directory listings. Not all SERPs contain all three elements; some
portals incorporate additional data sources depending on the search term used. Figure 1.1,
from Yahoo!, shows a typical SERP:
5Order The Search Engine Marketing Kit now and get $150 of PPC advertising credit
Chapter 1: Understanding Search Engines