Cloudera Developer Training for Spark & Hadoop (DSH) – Details

Detaillierter Kursinhalt

Introduction to Apache Hadoop and the Hadoop Ecosystem
  • Introduction to Apache Hadoop and the Hadoop Ecosystem
  • Apache Hadoop Overview
  • Data Ingestion and Storage
  • Data Processing
  • Data Analysis and Exploration
  • Other Ecosystem Tools
  • Introduction to the Hands-On Exercises
Apache Hadoop File Storage
  • Apache Hadoop Cluster Components
  • HDFS Architecture
  • Using HDFS
Distributed Processing on an Apache Hadoop Cluster
  • YARN Architecture
  • Working With YARN
Apache Spark Basics
  • What is Apache Spark?
  • Starting the Spark Shell
  • Using the Spark Shell
  • Getting Started with Datasets and DataFrames
  • DataFrame Operations
Working with DataFrames and Schemas
  • Creating DataFrames from Data Sources
  • Saving DataFrames to Data Sources
  • DataFrame Schemas
  • Eager and Lazy Execution
Analyzing Data with DataFrame Queries
  • Querying DataFrames Using Column Expressions
  • Grouping and Aggregation Queries
  • Joining DataFrames
RDD Overview
  • RDD Overview
  • RDD Data Sources
  • Creating and Saving RDDs
  • RDD Operations
Transforming Data with RDDs
  • Writing and Passing Transformation Functions
  • Transformation Execution
  • Converting Between RDDs and DataFrames
Aggregating Data with Pair RDDs
  • Key-Value Pair RDDs
  • Map-Reduce
  • Other Pair RDD Operations
Querying Tables and Views with Apache Spark SQL
  • Querying Tables in Spark Using SQL
  • Querying Files and Views
  • The Catalog API
  • Comparing Spark SQL, Apache Impala, and Apache Hive-on-Spark
Working with Datasets in Scala
  • Datasets and DataFrames
  • Creating Datasets
  • Loading and Saving Datasets
  • Dataset Operations
Writing, Configuring, and Running Apache Spark Applications
  • Writing a Spark Application
  • Building and Running an Application
  • Application Deployment Mode
  • The Spark Application Web UI
  • Configuring Application Properties
Distributed Processing
  • Review: Apache Spark on a Cluster
  • RDD Partitions
  • Example: Partitioning in Queries
  • Stages and Tasks
  • Job Execution Planning
  • Example: Catalyst Execution Plan
  • Example: RDD Execution Plan
Distributed Data Persistence
  • DataFrame and Dataset Persistence
  • Persistence Storage Levels
  • Viewing Persisted RDDs
Common Patterns in Apache Spark Data Processing
  • Common Apache Spark Use Cases
  • Iterative Algorithms in Apache Spark
  • Machine Learning
  • Example: k-means
Apache Spark Streaming: Introduction to DStreams
  • Apache Spark Streaming Overview
  • Example: Streaming Request Count
  • DStreams
  • Developing Streaming Applications
Apache Spark Streaming: Processing Multiple Batches
  • Multi-Batch Operations
  • Time Slicing
  • State Operations
  • Sliding Window Operations
  • Preview: Structured Streaming
Apache Spark Streaming: Data Sources
  • Streaming Data Source Overview
  • Apache Flume and Apache Kafka Data Sources
  • Example: Using a Kafka Direct Data Source