IIIT Hyderabad Publications |
|||||||||
|
Locality-Aware Hybrid Coded MapReduce for Server-Rack ArchitectureAuthors: Sneh Gupta,Lalitha Vadlamani Conference: 2017 IEEE Information Theory Workshop, Kaohsiung, Taiwan Date: 2017-11-06 Report no: IIIT/TR/2017/54 AbstractMapReduce is a widely used framework for dis- tributed computing. Data shuffling between the Map phase and Reduce phase of a job involves a large amount of data transfer across servers, which in turn accounts for increase in job completion time. Recently, Coded MapReduce has been proposed to offer savings with respect to the communication cost incurred in data shuffling. This is achieved by creating coded multicast opportunities for shuffling through repeating Map tasks at multiple servers. We consider a server-rack architecture for MapReduce and in this architecture, propose to divide the total communication cost into two: intra-rack communication cost and cross-rack communication cost. Having noted that cross-rack data transfer operates at lower speed as compared to intra-rack data transfer, we present a scheme termed as Hybrid Coded MapReduce which results in lower cross-rack communication than Coded MapReduce at the cost of increase in intra-rack communication. In addition, we pose the problem of assigning Map tasks to servers to maximize data locality in the framework of Hybrid Coded MapReduce as a constrained integer optimization problem. We show through simulations that data locality can be improved considerably by using the solution of optimization to assign Map tasks to servers. Full paper: pdf Centre for Communications |
||||||||
Copyright © 2009 - IIIT Hyderabad. All Rights Reserved. |