Extending a grid resource management architecture for managing data and computation for parallel applications

Abstract

Grid Resource Procedure Call (GridRPC) systems have been used to solve parallel applications over Grid resources using simple sequential interfaces. In these systems, sequential data from the user is segmented according to the data distribution used in the parallel application, and the data segments are staged to different Grid resources chosen either by the user or a Grid scheduler. The same input data may be reused by the same or different users multiple times, possibly with different data distributions, leading to multiple replicas of parallel data segments across various Grid resources. When a Grid user submits the same data with a data distribution for parallel problem solving, three possibilities exist:

Staging data from the user. Gathering data from replicas. Moving computation to a replica location.

The first part of this thesis addresses solution 2: selecting data segments from multiple replica sites and redistributing them to parallel computational resources. We developed four algorithms for selecting data servers and transferring data segments from multiple servers to multiple client resources for parallel application execution. The basic downloading and fastest algorithms are modifications of those used in multiple server–single client scenarios. The ISD algorithm considers the impact of simultaneous downloads on data transfer, while collective downloads is based on collective I/O optimization used in parallel I/O. We found that the relative performance of the algorithms depends on the network characteristics of the links between servers and clients, and between the clients themselves. While the collective download algorithm performs well when the relative performance difference is large and the number of client machines is small, the ISD algorithm yields the best performance when a large number of client machines are located in a single cluster/site and when the number of replicas is large. In all cases, the collective and ISD algorithms performed 15–30% better than the two basic algorithms. The next part of the thesis extends an existing Grid system, GrADSolve, which stages data from the user to remote parallel resources for problem solving. We created a version of GrADSolve in which the required data is obtained from replica sites using the ISD algorithm, as it produced the best results in most cases. Another version of GrADSolve was developed in which the Grid scheduler selects the best schedule of machines containing the existing replicas and performs computation at the selected data location. In this case, computation is moved to the data location. The thesis quantitatively evaluates and compares all three versions of the GrADSolve architecture and explains their relative performance under various Grid scenarios.