> For the complete documentation index, see [llms.txt](https://vikram-bajaj.gitbook.io/cs-gy-9223-d-programming-for-big-data/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://vikram-bajaj.gitbook.io/cs-gy-9223-d-programming-for-big-data/hadoop/the-google-file-system.md).

# The Google File System vs HDFS

Google's [paper](https://research.google.com/archive/gfs.html) on their file system describes GFS as a scalable distributed file system for large distributed data-intensive applications. It provides fault tolerance while running on inexpensive commodity hardware, and it delivers high aggregate performance to a large number of clients.

GFS formed the basis for the development of HDFS, which is more or less an open-source implementation of GFS.

## GFS Assumptions

* **Scale OUT, not UP**: prefer using commodity hardware to exotic hardware
* **High component failure rate**: it is assumed that components of commodity hardware fail all the time
* **Modest number of huge files**: it is assumed that files will be multiple-gigabytes in size
* **Files are write-once, mostly appended to**
* **Large streaming reads over random access**: sequential reading is preferred to random access since the high sustained throughput provided by sequential reading is preferred to the low latency provided by random access

## GFS Design Decisions

* **Files stored as chunks:** Fixed size (64MB)
* **Reliability through replication:** Each chunk is replicated across 3+ "chunkservers"
* **Single master to coordinate access, keep metadata**: Simple centralized management
* **No data caching**: Little benefit due to large datasets, streaming reads
* **Simplify the API:** Push some of the issues onto the client (e.g., data layout)

## From GFS to HDFS

As mentioned earlier, HDFS is based on GFS. However, there are a few key differences between the two:

### Terminology differences

* GFS Master = Hadoop NameNode
* GFS chunkservers = Hadoop DataNodes

### Functional differences

* No file appends in HDFS
* HDFS performance is (likely) slower

**The next section introduces the Data Types in Hadoop.**