Get ready to unlock the power of your data. With the fourth edition of this comprehensive guide, you’ll learn how to build and maintain reliable, scalable, distributed systems with Apache Hadoop. This book is ideal for programmers looking to analyze datasets of any size, and for administrators who want to set up and run Hadoop clusters.
Using Hadoop 2 exclusively, author Tom White presents new chapters on YARN and several Hadoop-related projects such as Parquet, Flume, Crunch, and Spark. You’ll learn about recent changes to Hadoop, and explore new case studies on Hadoop’s role in healthcare systems and genomics data processing.
Learn fundamental components such as MapReduce, HDFS, and YARN
Explore MapReduce in depth, including steps for developing applications with it
Set up and maintain a Hadoop cluster running HDFS and MapReduce on YARN
Learn two data formats: Avro for data serialization and Parquet for nested data
Use data ingestion tools such as Flume (for streaming data) and Sqoop (for bulk data transfer)
Understand how high-level data processing tools like Pig, Hive, Crunch, and Spark work with Hadoop
Learn the HBase distributed database and the ZooKeeper distributed configuration service
- 全面解析Hadoop 2架构，涵盖HDFS、MapReduce及YARN核心原理
- 深入讲解数据完整性、压缩序列化及Hadoop I/O底层机制
- 介绍Spark、Flume等周边生态项目，拓展大数据处理视野
- 希望构建可靠可扩展分布式系统的程序员与系统管理员
- 需要分析任意规模数据集，进行即席分析的技术人员
- 对系统底层设计感兴趣，欲夯实Hadoop基础的新手
- 重点研读前两部分基础与MapReduce，后部分生态项目可略读
- 建议直接阅读英文原版，中文版翻译质量参差不齐，易误导
- 结合具体项目实践阅读，单纯理论阅读较枯燥且难以消化
- Hadoop领域经典权威指南，原理讲解透彻，知识体系清晰
- 内容详实但篇幅较长，适合做工具书查阅或入门系统学习
- 虽MapReduce编程少用，但理解其思想对掌握大数据至关重要
本导读基于书籍简介、目录、原文摘录、短评和书评生成，不等同于全文精读。
- "* The architecture of HDFS is described in “The Hadoop Distributed File System” by Konstantin Shvachko, Hairong Kuang, Sanjay Radia, and Robert Chansler (Proceedings of MSST2010, May 2010, http:// storageconference.org/2010/Papers/MSST/Shvachko.pdf). † “Scaling Hadoop to 4000 nodes at Yahoo!,” http:"
- "在许多情况下，可以视Mapreduce为关系型数据库管理系统的补充。MapReduce比较适合以批处理的方式处理需要分析整个数据集的问题，尤其是即席分析。RDBMS适用于点查询和更新，数据集被索引后，数据库系统能够提供低延迟的数据检索和快速的少量数据更新。MapReduce适合数据一次写入、多次读取的应用，而关系型数据库更适合持续更新数据集."
- "MapReduce is a programming model for data processing. MapReduce works by breaking the processing into two phases: the map phase and the reduce phase. Each phase has key-value pairs as input and output, the types of which may be chosen by the programmer. The programmer also specifies two functions: t"
- "Hadoop divides the input to a MapReduce job into fixed-size pieces called input splits, or just splits. Hadoop creates one map task for each split, which runs the userdefined map function for each record in the split. Hadoop does its best to run the map task on a node where the input data resides in"
- "HDFS is a filesystem designed for storing very large files with streaming data access patterns(write-once, read-many-times pattern), running on clusters of commodity hardware. HDFS blocks(>64M) are large compared to disk blocks, and the reason is to minimize the cost of seeks. Map tasks in MapReduce"
- "One important aspect of this design is that the client contacts datanodes directly to retrieve data and is guided by the namenode to the best datanode for each block. This design allows HDFS to scale to a large number of concurrent clients, since the data traffic is spread across all the datanodes i"
- "As the client writes data (step 3), DFSOutputStream splits it into packets, which it writes to an internal queue, called the data queue . The data queue is consumed by the Data Streamer , whose responsibility it is to ask the namenode to allocate new blocks by picking a list of suitable datanodes to"
- "HDFS Federation, introduced in the 0.23 release series, allows a cluster to scale by adding namenodes, each of which manages a portion of the filesystem namespace. For example, one namenode might manage all the files rooted under /user , say, and a second namenode might handle files under /share . T"
Developing Apps with GPT-4 and ChatGPT
Building Knowledge Graphs
Essential Math for AI
Running Lean (3/e)
Quarkus Cookbook
Learning React 2nd ed.