Big Data Architecture Workshop (BDAW) – Outline

Detailed Course Outline

Introduction

Workshop Application Use Cases

  • Oz Metropolitan
  • Architectural questions
  • Team activity: Analyze Metroz
  • Application Use Cases

Application Vertical Slice

  • Definition
  • Minimizing risk of an unsound architecture
  • Selecting a vertical slice
  • Team activity: Identify an initial vertical slice for Metroz

Application Processing

  • Real time, near real time processing
  • Batch processing
  • Data access patterns
  • Delivery and processing guarantees
  • Machine Learning pipelines
  • Team activity: identify delivery and processing patterns in Metroz, characterize response time requirements, identify Machine Learning pipelines

Application Data

  • Three V’s of Big Data
  • Data Lifecycle
  • Data Formats
  • Transforming Data
  • Team activity: Metroz Data Requirements

Scalable Applications

  • Scale up, scale out, scale to X
  • Determining if an application will scale
  • Poll: scalable airport terminal designs
  • Hadoop and Spark Scalability
  • Team activity: Scaling Metroz

Fault Tolerant Distributed Systems

  • Principles
  • Transparency
  • Hardware vs. Software redundancy
  • Tolerating disasters
  • Stateless functional fault tolerance
  • Stateful fault tolerance
  • Replication and group consistency
  • Fault tolerance in Spark and Map Reduce
  • Application tolerance for failures
  • Team activity: Identify Metroz component failures and requirements

Security and Privacy

  • Principles
  • Privacy
  • Threats
  • Technologies
  • Team activity: identify threats and security mechanisms in Metroz

Deployment

  • Cluster sizing and evolution
  • On-premise vs. Cloud
  • Edge computing
  • Team activity: select deployment for Metroz

Technology Selection

  • HDFS
  • HBase
  • Kudu
  • Relational Database Management Systems
  • Map Reduce
  • Spark, including streaming, SparkSQL and SparkML
  • Hive
  • Impala
  • Cloudera Search
  • Data Sets and Formats
  • Team activity: technologies relevant to Metroz

Software Architecture

  • Architecture artifacts
  • One platform or multiple, lambda architecture
  • Team activity: produce high level architecture, selected technologies, revisit vertical slice
  • Vertical Slice demonstration

Wrap Up