Setting up a modern Big Data environment has become a full-time job that requires more systems plumbing skills than data engineering. A year ago I wrote wonders about Apache Spark, but the harsh corporate reality is that Spark rarely runs alone; it needs to live inside the monstrous zoo that is the Apache Hadoop ecosystem.
Spinning up a Hadoop 2.x cluster from scratch is a headache to say the least.
Deconstructing the yellow elephant
Hadoop is no longer just one thing, it's a set of overlapping tools. To understand it, you have to separate storage from compute.
The foundation of everything is HDFS (Hadoop Distributed File System). It takes the hard drives of twenty different computers and joins them into a single virtual file system, replicating each data block three times just in case a drive burns out. Moving data here isn't done with Windows Explorer, it's done via terminal:
# Create a directory on the distributed cluster
hadoop fs -mkdir /datos/logs_corporativos/
# Upload a giant log file from my local machine to HDFS
hadoop fs -put /var/log/apache2/access.log /datos/logs_corporativos/
# See how the blocks have been distributed and replicated
hdfs fsck /datos/logs_corporativos/access.log -files -blocks -locations
But HDFS just stores dumb files. How do we process them without stepping on each other? Here comes the great revolution of Hadoop 2.0: YARN (Yet Another Resource Negotiator).
In the old version of Hadoop, the resource manager and MapReduce were coupled. It was a disaster if you wanted to run something else. YARN acts as the "operating system" of the cluster. If I launch a Python script, YARN looks at which nodes in the cluster have free RAM and CPU, and assigns "containers" to execute my code right on the machine that physically has the data on its disk (to avoid clogging the network).
Reflection: Operational madness
The problem with this ecosystem is the operational complexity. If you look at the architecture diagram of a real project, you have HDFS for storing, YARN for managing resources, Spark or MapReduce for computing, Hive for firing SQL, Zookeeper to coordinate which node is alive, and asynchronous messaging tools populating the entire corporate data ecosystem.
Keeping this house of cards updated, ensuring Java versions line up and daemons don't crash, is a DevOps nightmare. This technology democratizes Big Data because it lets you use cheap hardware, but the cost in man-hours for maintenance is extremely high. I wonder if we won't all end up surrendering and paying Amazon or Google to manage these clusters in the cloud for us at the push of a single button, instead of suffering by compiling Apache projects locally. Today, I'd sign up for that.