Grover M., Malaska T., Seidman J., Shapira G. Hadoop Application Architectures

O’Reilly Media, 2015 — 400 p. — e-ISBN: 978-1-4919-0004-8, ISBN10: 1-4919-0004-0Get expert guidance on architecting end-to-end data management solutions with Apache Hadoop. While many sources explain how to use various components in the Hadoop ecosystem, this practical book takes you through architectural considerations necessary to tie those components together into a complete tailored application, based on your particular use case.To reinforce those lessons, the book’s second section provides detailed examples of architectures used in some of the most commonly found Hadoop applications. Whether you’re designing a new Hadoop application, or planning to integrate Hadoop into your existing data infrastructure, Hadoop Application Architectures will skillfully guide you through the process.This book covers:Factors to consider when using Hadoop to store and model data
Best practices for moving data in and out of the system
Data processing frameworks, including MapReduce, Spark, and Hive
Common Hadoop processing patterns, such as removing duplicate records and using windowing analytics
Giraph, GraphX, and other tools for large graph processing on Hadoop
Using workflow orchestration and scheduling tools such as Apache Oozie
Near-real-time stream processing with Apache Storm, Apache Spark Streaming, and Apache Flume
Architecture examples for clickstream analysis, fraud detection, and data warehousingDATA MODELING IN HADOOP
HDFS Schema Design
HBase Schema Design
Managing MetadataDATA MOVEMENT
Data Ingestion Considerations
Data Ingestion Options
Data ExtractionPROCESSING DATA IN HADOOP
MapReduce
Spark
Abstractions
Crunch
Cascading
Hive
ImpalaCOMMON HADOOP PROCESSING PATTERNS
Pattern: Removing Duplicate Records by Primary Key
Pattern: Windowing Analysis
Pattern: Time Series ModificationsGRAPH PROCESSING ON HADOOP
What Is a Graph?
What Is Graph Processing?
How Do You Process a Graph in a Distributed System?
Giraph
GraphX
Which Tool to Use?ORCHESTRATION
Why We Need Workflow Orchestration
The Limits of Scripting
The Enterprise Job Scheduler and Hadoop
Orchestration Frameworks in the Hadoop Ecosystem
Oozie Terminology
Oozie Overview
Oozie Workflow
Workflow Patterns
Parameterizing Workflows
Classpath Definition
Scheduling Patterns
Executing WorkflowsNEAR-REAL-TIME PROCESSING WITH HADOOP
Stream Processing
Apache Storm
Trident
Spark Streaming
Flume Interceptors
Which Tool to Use?Case StudiesCLICKSTREAM ANALYSIS
Defining the Use Case
Using Hadoop for Clickstream Analysis
Design Overview
Storage
Ingestion
Processing
Analyzing
OrchestrationFRAUD DETECTION
Continuous Improvement
Taking Action
Architectural Requirements of Fraud Detection Systems
Introducing Our Use Case
High-Level Design
Client Architecture
Profile Storage and Retrieval
Ingest
Near-Real-Time and Exploratory Analytics
Near-Real-Time Processing
Exploratory Analytics
What About Other Architectures?DATA WAREHOUSE
Using Hadoop for Data Warehousing
Defining the Use Case
OLTP Schema
Data Warehouse: Introduction and Terminology
Data Warehousing with Hadoop
High-Level DesignAppendix Joins in Impala
Broadcast Joins
Partitioned Hash Join