White T. Hadoop: The Definitive Guide

2th Edition. — O’Reilly Media / Yahoo Press, 2010. — 628 p. — Print ISBN:978-1-4493-8973-4ISBN 10:1-4493-8973-2.Discover how Apache Hadoop can unleash the power of your data. This comprehensive resource shows you how to build and maintain reliable, scalable, distributed systems with the Hadoop framework -- an open-source implementation of MapReduce, the algorithm on which Google built its empire. Programmers will find details for analyzing datasets of any size, and administrators will learn how to set up and run Hadoop clusters.This revised edition covers recent changes to Hadoop, including new features such as Hive, Sqoop, and Avro. It also provides illuminating case studies that illustrate how Hadoop is used to solve specific problems. Looking to get the most out of your data? This is your book.Use the Hadoop Distributed File System (HDFS) for storing large datasets, then run distributed computations over those datasets with MapReduce
Become familiar with Hadoop’s data and I/O building blocks for compression, data integrity, serialization, and persistence
Discover common pitfalls and advanced features for writing real-world MapReduce programs
Design, build, and administer a dedicated Hadoop cluster, or run Hadoop in the cloud
Use Pig, a high-level query language for large-scale data processing
Analyze datasets with Hive, Hadoop’s data warehousing system
Take advantage of HBase, Hadoop’s database for structured and semi-structured data
Learn ZooKeeper, a toolkit of coordination primitives for building distributed systems
"Now you have the opportunity to learn about Hadoop from a master -- not only of the technology, but also of common sense and plain talk."--Doug Cutting, ClouderaChapter 1 Meet Hadoop
Data!
Data Storage and Analysis
Comparison with Other Systems
A Brief History of Hadoop
Apache Hadoop and the Hadoop Ecosystem
Chapter 2 MapReduce
A Weather Dataset
Analyzing the Data with Unix Tools
Analyzing the Data with Hadoop
Scaling Out
Hadoop Streaming
Hadoop Pipes
Chapter 3 The Hadoop Distributed Filesystem
The Design of HDFS
HDFS Concepts
The Command-Line Interface
Hadoop Filesystems
The Java Interface
Data Flow
Parallel Copying with distcp
Hadoop Archives
Chapter 4 Hadoop I/O
Data Integrity
Compression
Serialization
File-Based Data Structures
Chapter 5 Developing a MapReduce Application
The Configuration API
Configuring the Development Environment
Writing a Unit Test
Running Locally on Test Data
Running on a Cluster
Tuning a Job
MapReduce Workflows
Chapter 6 How MapReduce Works
Anatomy of a MapReduce Job Run
Failures
Job Scheduling
Shuffle and Sort
Task Execution
Chapter 7 MapReduce Types and Formats
MapReduce Types
Input Formats
Output Formats
Chapter 8 MapReduce Features
Counters
Sorting
Joins
Side Data Distribution
MapReduce Library Classes
Chapter 9 Setting Up a Hadoop Cluster
Cluster Specification
Cluster Setup and Installation
SSH Configuration
Hadoop Configuration
Security
Benchmarking a Hadoop Cluster
Hadoop in the Cloud
Chapter 10 Administering Hadoop
HDFS
Monitoring
Maintenance
Chapter 11 Pig
Installing and Running Pig
An Example
Comparison with Databases
Pig Latin
User-Defined Functions
Data Processing Operators
Pig in Practice
Chapter 12 Hive
Installing Hive
An Example
Running Hive
Comparison with Traditional Databases
HiveQL
Tables
Querying Data
User-Defined Functions
Chapter 13 HBase
HBasics
Concepts
Installation
Clients
Example
HBase Versus RDBMS
Praxis
Chapter 14 ZooKeeper
Installing and Running ZooKeeper
An Example
The ZooKeeper Service
Building Applications with ZooKeeper
ZooKeeper in Production
Chapter 15 Sqoop
Getting Sqoop
A Sample Import
Generated Code
Database Imports: A Deeper Look
Working with Imported Data
Importing Large Objects
Performing an Export
Exports: A Deeper Look
Chapter 16 Case Studies
Hadoop Usage at Last.fm
Hadoop and Hive at Facebook
Nutch Search Engine
Log Processing at Rackspace
Cascading
TeraByte Sort on Apache Hadoop
Using Pig and Wukong to Explore Billion-edge Network Graphs
Appendix Installing Apache HadoopPrerequisitesInstallation
Configuration
Appendix Cloudera’s Distribution for Hadoop
Appendix Preparing the NCDC Weather Data
Colophon