The Azure Data Lakehouse Toolkit Building and Scaling Data Lakehouses on Azure with Delta Lake, Apache Spark, Databricks, Synapse Analytics, and Snowflake

Bạn đang xem bản rút gọn của tài liệu. Xem và tải ngay bản đầy đủ của tài liệu tại đây (8.35 MB, 700 trang )

<span class="text_page_counter">Trang 2</span><div class="page_container" data-page="2">

<b>The Azure DataLakehouse Toolkit</b>

<b>Building and Scaling Data</b>

<b>Lakehouses on Azure with DeltaLake, Apache Spark, Databricks,Synapse Analytics, and SnowflakeRon L’Esteve</b>

<i><b>The Azure Data Lakehouse Toolkit: Building and Scaling DataLakehouses on Azure</b></i>

<i><b>with Delta Lake, Apache Spark, Databricks, Synapse Analytics, andSnowflake</b></i>

Ron L’EsteveChicago, IL, USA

ISBN-13 (pbk): 978-1-4842-8232-8

This work is subject to copyright. All rights are reserved by the Publisher,whether the whole or part of the material is concerned, specifically therights of translation, reprinting, reuse of illustrations, recitation,

broadcasting, reproduction on microfilms or in any other physical way, andtransmission or information storage and retrieval, electronic adaptation,

</div><span class="text_page_counter">Trang 3</span><div class="page_container" data-page="3">

computer software, or by similar or dissimilar methodology now known orhereafter developed.

Trademarked names, logos, and images may appear in this book. Ratherthan use a trademark symbol with every occurrence of a trademarked name,logo, or image we use the names, logos, and images only in an editorialfashion and to the benefit of the trademark owner, with no intention ofinfringement of the trademark.

The use in this publication of trade names, trademarks, service marks, andsimilar terms, even if they are not identified as such, is not to be taken as anexpression of opinion as to whether or not they are subject to proprietaryrights.

While the advice and information in this book are believed to be true andaccurate at the date of publication, neither the authors nor the editors northe publisher can accept any legal responsibility for any errors or omissionsthat may be made. The publisher makes no warranty, express or implied,with respect to the material contained herein.

Managing Director, Apress Media LLC: Welmoed SpahrAcquisitions Editor: Jonathan Gennick

Development Editor: Laura BerendsonCoordinating Editor: Jill BalzanoCover designed by eStudioCalamar

Cover image designed by Freepik (www.freepik.com)

Distributed to the book trade worldwide by Springer Science+BusinessMedia New York, 1 New York Plaza, Suite 4600, New York, NY 10004-1562, USA. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@

springer-sbm.com, or visit www.springeronline.com. Apress Media, LLC isa California LLC and the sole member (owner) is Springer Science +

</div><span class="text_page_counter">Trang 4</span><div class="page_container" data-page="4">

Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a

<b>Delaware corporation.</b>

For information on translations, please ; for reprint,paperback, or audio rights, please e-mail

Apress titles may be purchased in bulk for academic, corporate, or

promotional use. eBook versions and licenses are also available for mosttitles. For more information, reference our Print and eBook Bulk Sales webpage at source code or other supplementary material referenced by the authorin this book is available to readers on GitHub via the book’s product page,located at www.apress.com/. For more detailed information, please visit code.

Printed on acid-free paper

<i>For Cayden and Christina</i>

<b>Table of Contents</b>

About the Author

��xv About the Technical Reviewer��xvii

��xix Introduction

</div><span class="text_page_counter">Trang 5</span><div class="page_container" data-page="5">

��xxi Part I: GettingStarted

�� 1Chapter 1: The Data Lakehouse Paradigm

�� 3

�� 4

Ingestion and Processing

�� 6

Data Factory

��

</div><span class="text_page_counter">Trang 6</span><div class="page_container" data-page="6">

�� 6

�� 10

Functions and Logic Apps

�� 11

Synapse Analytics Serverless Pools

�� 12Stream Analytics

Messaging Hubs

��15

</div><span class="text_page_counter">Trang 7</span><div class="page_container" data-page="7">

Storing and Serving

Delta Lake

�� 17

Synapse Analytics Dedicated SQL Pools

�� 19

Relational Database

�� 21

Non-relational Databases

�� 23

�� 26

</div><span class="text_page_counter">Trang 8</span><div class="page_container" data-page="8">

�� 28

Analysis Services

�� 28Power BI

�� 30

Power Apps

�� 33

Table of ConTenTsAdvanced Analytics

�� 33

</div><span class="text_page_counter">Trang 9</span><div class="page_container" data-page="9">

Cognitive Services

�� 34Machine Learning

�� 35Continuous Integration, Deployment, and Governance

�� 36

�� 39

Summary�� 41

</div><span class="text_page_counter">Trang 10</span><div class="page_container" data-page="10">

Part II: Data Platforms

�� 43

Chapter 2: Snowflake

�� 45

�� 46

�� 48

�� 49

Azure Key Vault

��

</div><span class="text_page_counter">Trang 11</span><div class="page_container" data-page="11">

�� 49

Azure Private Link

�� 50Applications

�� 50

Replication and Failover

�� 50

Data Integration with Azure

�� 51

Data Lake Storage Gen2

�� 51

Real-Time Data Loading with ADLS gen2

��

</div><span class="text_page_counter">Trang 12</span><div class="page_container" data-page="12">

�� 53

Data Factory

�� 54

�� 55

Data Transformation

�� 63

Column-Level Security

�� 63

</div><span class="text_page_counter">Trang 13</span><div class="page_container" data-page="13">

Row-Level Security

�� 65

Access History

�� 66

Object Tagging

�� 66

�� 67

Direct Share

�� 69

Data Marketplace

��

</div><span class="text_page_counter">Trang 14</span><div class="page_container" data-page="14">

�� 70Data Exchange

�� 70

Table of ConTenTs

Continuous Integration and Deployment

�� 71Jenkins

�� 71

Azure DevOps

�� 72

��

</div><span class="text_page_counter">Trang 15</span><div class="page_container" data-page="15">

�� 73

Power BI

�� 73

Delta Lake, Machine Learning, and Constraints

�� 78

Delta Lake

�� 78

Machine Learning

�� 79Constraints

�� 80

Summary��

</div><span class="text_page_counter">Trang 16</span><div class="page_container" data-page="16">

�� 80

Chapter 3: Databricks

�� 83

�� 84

Data Science and Engineering

�� 84

Machine Learning

�� 91SQL

�� 93

</div><span class="text_page_counter">Trang 17</span><div class="page_container" data-page="17">

�� 96

�� 100

Mount Data Lake Storage Gen2 Account

�� 101

Delta Lake

�� 114

�� 115

Real-Time Analytics

��

</div><span class="text_page_counter">Trang 18</span><div class="page_container" data-page="18">

�� 117Advanced Analytics

Security and Governance

�� 121

Continuous Integration and Deployment

�� 125

Integration with Synapse Analytics

�� 126

Dynamic Data Encryption

�� 127

Data Profile

��

</div><span class="text_page_counter">Trang 19</span><div class="page_container" data-page="19">

�� 129

Query Profile

�� 129

�� 133

Table of ConTenTs

Delta Live Tables Merge

�� 135

Summary��

</div><span class="text_page_counter">Trang 20</span><div class="page_container" data-page="20">

�� 138

Chapter 4: Synapse Analytics

�� 141

�� 142

�� 144

�� 149

�� 150

</div><span class="text_page_counter">Trang 21</span><div class="page_container" data-page="21">

�� 154

�� 156

Continuous Integration and Deployment

�� 158

Real-Time Analytics

�� 160Structured Streaming

�� 160

</div><span class="text_page_counter">Trang 22</span><div class="page_container" data-page="22">

Synapse Link

�� 160

Advanced Analytics

�� 163

�� 166

Additional Features

Delta Tables

��

</div><span class="text_page_counter">Trang 23</span><div class="page_container" data-page="23">

�� 170

Machine Learning

�� 172

SQL Server Integration Services Integration Runtime (SSIS IR)

�� 172

Map Data Tool

�� 173

Data Sharing

�� 175

SQL Incremental

�� 175Constraints

��

</div><span class="text_page_counter">Trang 24</span><div class="page_container" data-page="24">

�� 176

Summary�� 178

Part III: Apache Spark ELT

�� 183

Chapter 5: Pipelines and Jobs

�� 185

Data Factory

�� 191

viii

</div><span class="text_page_counter">Trang 25</span><div class="page_container" data-page="25">

Table of ConTenTsMapping Data Flows

�� 191

HDInsight Spark Activity

�� 196

Scheduling and

Monitoring�� 200

Synapse Analytics Workspace

�� 202

Summary�� 206

Chapter 6: Notebook Code

�� 209

</div><span class="text_page_counter">Trang 26</span><div class="page_container" data-page="26">

�� 210

�� 211

XML�� 217

�� 221

ZIP�� 225

��

</div><span class="text_page_counter">Trang 27</span><div class="page_container" data-page="27">

�� 227

�� 228

Optimizing Performance

�� 229

Summary�� 232

Part IV: Delta Lake

Chapter 7: Schema

Evolution�� 235Schema Evolution Using Parquet Format

�� 236

</div><span class="text_page_counter">Trang 28</span><div class="page_container" data-page="28">

Schema Evolution Using Delta Format

�� 240

�� 241

Summary�� 243

Chapter 8: Change Data Feed

�� 245

Create Database and Tables

�� 245

</div><span class="text_page_counter">Trang 29</span><div class="page_container" data-page="29">

Insert Data into Tables

�� 248

Change Data Capture

�� 249

Streaming Changes

Summary�� 255

Table of ConTenTsChapter 9: Clones

�� 257

Clones��

</div><span class="text_page_counter">Trang 30</span><div class="page_container" data-page="30">

�� 257

Deep Clones

�� 263

Summary�� 267

Chapter 10: Live Tables

�� 269

Advantages of Delta Live Tables

�� 270

Create a Notebook

��270

</div><span class="text_page_counter">Trang 31</span><div class="page_container" data-page="31">

Create and Run a Pipeline

�� 274

Schedule a

Pipeline�� 278

Explore Event Logs

Summary�� 283

Chapter 11: Sharing

�� 285

��

</div><span class="text_page_counter">Trang 32</span><div class="page_container" data-page="32">

�� 286

Share Data

�� 287

Access Data

�� 288

Sharing Data with Snowflake

�� 291

Summary�� 292

Part V: Optimizing Performance

�� 295

Chapter 12: Dynamic Partition Pruning

��

</div><span class="text_page_counter">Trang 33</span><div class="page_container" data-page="33">

�� 297

�� 299

DPP Commands

Create Cluster

�� 300

Create Notebook and Mount Data Lake

�� 300

</div><span class="text_page_counter">Trang 34</span><div class="page_container" data-page="34">

Create Fact Table

�� 301Verify Fact Table Partitions

�� 304

Create Dimension Table

�� 305

Table of ConTenTs

Join Results Without DPP Filter

�� 306

Join Results with DPP Filter

�� 308

Summary��

</div><span class="text_page_counter">Trang 35</span><div class="page_container" data-page="35">

�� 309

Chapter 13: Z-Ordering and Data Skipping

�� 311

Prepare Data in Delta Lake

�� 312

Verify Data in Delta Lake

�� 314

Create Hive Table

�� 317

Run Optimize and Z-Order

Commands�� 318

Verify Data Skipping

��

</div><span class="text_page_counter">Trang 36</span><div class="page_container" data-page="36">

�� 320Summary�� 325

Chapter 14: Adaptive Query Execution

�� 327

How It Works

�� 327

�� 328

Comparing AQE Performance on Query with Joins

�� 329

Create Datasets

��

</div><span class="text_page_counter">Trang 37</span><div class="page_container" data-page="37">

Disable AQE

�� 332

Enable AQE

�� 334

Summary�� 338

Chapter 15: Bloom Filter Index

�� 339

How a Bloom Filter Index Works

�� 339

Create a Cluster

��

</div><span class="text_page_counter">Trang 38</span><div class="page_container" data-page="38">

�� 340

Create a Notebook and Insert Data

�� 341

Enable Bloom Filter Index

�� 343

Tables�� 344

Create a Bloom Filter Index

�� 346

Optimize Table with Z-Order

�� 348

Verify Performance Improvements

��

</div><span class="text_page_counter">Trang 39</span><div class="page_container" data-page="39">

�� 349

Summary�� 352

Table of ConTenTs

Chapter 16: Hypers pace

�� 354

Create Parquet Files

�� 358Run a Query Without an Index

��

</div><span class="text_page_counter">Trang 40</span><div class="page_container" data-page="40">

�� 360

Import Hyperspace

Read the Parquet Files to a Data Frame

�� 362

Create a Hyperspace Index

�� 362

Rerun the Query with Hyperspace Index

�� 364

Other Hyperspace Management APIs

�� 365

Summary�� 366

</div>

The Azure Data Lakehouse Toolkit Building and Scaling Data Lakehouses on Azure with Delta Lake, Apache Spark, Databricks, Synapse Analytics, and Snowflake

Tài liệu liên quan

Tài liệu bạn tìm kiếm đã sẵn sàng tải về