A distributed file system in Go
This started as a big data course project at USF: build something shaped like HDFS, then run it on the department’s cluster. Mine ran on twelve storage machines named orion01 to orion12, which is why the simulation further down uses those names.
A client cuts a file into 7 MB chunks and sends them to storage nodes. One controller remembers which node holds which chunk, keeps that in a Bitcask store on disk, and makes sure every chunk exists on three nodes. Everything talks protobuf over TCP.
The part I’d show someone first is how the controller picks a home for the next chunk.
func getRouteChunk() (string, error) {
efficiencyConstant := float64(0)
mostEfficientNode := ""
for node, value := range registrationMap {
if value == 1 { // node is active
space := getDiskSpace(node)
requests := getNodeRequests(node)
efficiency := float64(space) / (float64(requests) + 1)
if efficiency > efficiencyConstant || mostEfficientNode == "" {
efficiencyConstant = efficiency
mostEfficientNode = node
}
}
}
…
}
Free disk space divided by the number of requests the node has already been handed, highest wins. A big empty machine doesn’t get everything just because it is big.
Failure handling is blunt. Every node sends a heartbeat each five seconds, carrying its free space and request count. The controller looks at the clock every two seconds, and any node that has been quiet for more than six is written off. For each chunk that node held, a surviving copy is told to send itself somewhere new.
You can watch that happen here. Pick a node, stop it, and wait.