Special Issue: The Best of CCGrid'2007: A Snapshot of an ‘Adolescent’ Area

It is with great pleasure that we introduce you to this special issue with the best work presented in CCGrid'2007, the 7th IEEE International Symposium on Cluster Computing and the Grid, held in Rio de Janeiro, Brazil, on May 14–17, 2007. Over the years, CCGrid has become a premium conference of truly international coverage, bringing together researchers and practitioners, and enabling them to share their insight, results, and experience in the multi-faceted areas of Grid and cluster computing. Overall, 330 people attended the event, a CCGrid all-time record. This is a testimony to the fact that the grid and cluster computing communities continue to grow, and many early projects that were started in the beginning of the decade (when the conference series began) are now maturing and producing significant results. The international coverage and impact of CCGrid are also worth highlighting. After Australia, Germany, Japan, the United States of America, the United Kingdom, and Singapore, CCGrid comes to Brazil without yet repeating a host country. Similarly, these proceedings carry papers from 22 different countries: Algeria, Australia, Austria, Brazil, Canada, China, Denmark, France, Germany, Greece, Holland, Israel, Italy, Korea, Japan, Poland, Singapore, Switzerland, Tunisia, the United States of America, the United Kingdom, and Singapore. But, of course, the whole conference experience consisted of more than its strong technical program. It also included three thought-provoking keynote speeches: Grids Sandwiched by Web 2.0 and Multicore by Geoffrey Fox from Indiana University 1, Towards an International Computer Science Grid by Franck Cappello from INRIA, and Scale-up and Scale-out: Evolution and Trends in Parallel Processing by José E. Moreira from IBM, as well as a very well-attended industry track with presentations on Google Infrastructure for Massively Parallel Processing by Walfredo Cirne from Google; Promoting Cooperation in BitTorrent Communities by Miranda Mowbray from HP, and Opportunity and Challenges in e-Science by Fabrizio Gagliardi and Carlos Hulot from Microsoft. Furthermore, in parallel with the main conference track, there were seven workshops on emerging topics: the IEEE Technical Committee on Scalable Computing Doctoral Symposium, Agent-based Grid Computing (AGC'07), Biomedical Computations on the Grid (BioGrid'07), Global and Peer-2-Peer Computing (GP2P'07), Context-Awareness and Mobility in Grid Computing (WCAMG'07), Programming Models for Grid Computing (PMGC'07), and Latin American Grid Workshop (LAGrid'07). In the end, this massive wealth of high-quality papers, presentations, and discussions is the reason for the success of the conference. They turn the task of editing this special issue into a great challenge, albeit a delightful one. Selecting the best work among such great contributions is an arduous endeavor in itself, which we pursued together the Program Committee Co-chairs, to whom we are very grateful. But, it may be even more difficult to capture the ‘extra-technical’ discussions and conversations that happen in panels, Q&A sections, coffee breaks, and even in the conference's activities. Although they are often more well-informed speculation than scientific results, they are an important aspect of the conference, as they create a sense of community among the researchers and practitioners in the area, and often influence what endeavor each of us decides to pursue next. We addressed this challenge by inviting extended and updated versions of some stronger technical contributions in the congress, together with an article laying the argument of the most discussed and controversial keynote presented. We also give our view on the current state of the area and why we say it is going through ‘adolescence’ as a way to complement the keynote article and try to capture the rich debates that took place during CCGrid'2007. High-performance computing (HPC) plays an interesting and unique role in the computer industry. HPC is a small niche, albeit a very important one, as it often debuts technologies that later become mainstream. Therefore, the computer industry tends to invest much more and to follow the developments in HPC much closely than its market importance would suggest. We believe that this ‘close monitoring’ of HPC was essentially what put Grid computing in the spot light, turning it into one of the hottest research areas in computer science of the early 2000s. In the early 2000s, cluster computing has been consolidated as the way to HPC. Grid computing then appeared to be the promise of combining clusters, equipment, and talent into virtual organizations (VOs), delivering unprecedented levels of parallelism to high-performance applications, and making possible science that is just not achievable in any single place, by any single organization. The computer industry then made the observation that such a technology could also enable on-demand access and composition of any computational service provided by multiple independent sources. This could truly revolutionize the way we use computers, turning the focus from hardware, software, and peopleware into services, which could be composed on demand to serve whatever need one may have. This was great news for the community working on HPC Grids. Not only would their work be relevant to many more people, but more funding would also be available. The last years saw great advances in Grid computing, as well as in cluster computing, as clusters remained the key ‘building block’ for Grids, HPC, and large Internet services. The technical papers we selected for this special issue provide an excellent example of how far we have reached. Two of them cover real and practical applications, which have been deployed and used (climate studies 9 and image analysis 4). The other works advance the state of the art in cluster and Grid computing in important ways but always evaluating their contributions in real systems, with at least prototype implementations. The days of ‘architecture papers’ and ‘simplified simulation results’ are over. Grid computing has concrete results to show. Although still young, one can no longer claim that the area is in its ‘infancy’. Alas, the application of Grid computing beyond the HPC domain has been much slower. Many people believe that it is just a matter of time for it to happen. Other people have maintained that Grid computing at large was a difficult proposition because the standards being developed, based on Web Services (the so-called WS* protocols), were just too complex for wide acceptance. It used to be that believers would outnumber skepticals by at least one order of magnitude. In CCGrid'2007, we saw these groups having about the same size. We even had a keynote speech toying with the idea that ‘the WS* protocols are just too complex and heavyweight’ 1. Comparisons between the WS* standardization efforts and those of ISO/OSI network protocols were often heard in the discussions. We argue that this leaves the area in an ‘adolescent’ state, with its own identity at stake. Some people keep believing that Grid computing at large is going to happen based on WS* protocols, which is the major focus of the area so far. Others argue that we should just forget about computing at large and focus on a scientific high-performance application, for which the results are concrete and relevant. Yet a third group maintains that the fundamental computer science problems posed by dynamic composition of services remain hard and relevant, and that the scientific contribution we have been making in tackling them is the real ‘deliverable’ of the area. According to this third group, we just need to make a technological change, moving from the WS* technology to something simpler, likely Web 2.0 (an argument well articulated in 1). In reality, we do not know what the future holds for us. But it is clear that if the Grid computing community wants to have a real impact on computing as a whole, we need more people explicitly addressing problems and promoting use outside HPC. The buzzword appeal of the early days of Grid computing is over. It is time for concrete results. There are plenty of results in HPC, but can we do the same beyond it? The selected papers broadly cover key research issues in cluster computing and Grid computing as summarized below: Fox and Pierce 1 observe that Grids (as envisioned around 2001) are being pressured by both emerging new computing resources (multicore, cell processors, GPUs, reconfigurable computing, etc.) and alternative approaches to service architectures (collectively, Web 2.0). They suggest that it is time to reappraise Grids—both the nature of the resources that they aggregate and the middleware that glues these resources together. They discuss the application of Web 2.0 to support scientific research (e-Science) and related ‘e-moreorlessanything’ applications. Web 2.0 offers interesting technical approaches (protocols, message formats, and programming tools) to build core e-infrastructure (cyberinfrastructure) as well as many interesting services (Facebook, YouTube, Amazon S3/EC2, and Google maps) that can add value to e-infrastructure projects. They discuss why some of the original Grid goals of linking the world's computer systems may not be so relevant today as one has in many ways ‘too much computing’. Rather than cobbling together pre-existing heterogeneous systems, computing is so cheap that one can link together large scalable clouds with designed heterogeneity. Support for data-intensive systems is critical but these require interoperability interfaces at the data level and not always at the infrastructure level where Grids have focused. Web 2.0 can also support Parallel Programming 2.0—a better parallel computing software environment motivated by the need to run commodity applications on multicore chips. A ‘Grid on the chip’ will be a common use of future chips with tens or hundreds of cores. In spite of the unclear technology directions, they note that e-Science and more generally e-moreorlessanything are thriving with the advantages of distributed electronic enablement of many fields being very clear. According to M. Klemm et al. 2, most computational scientists have access to various worldwide computing resources in computational Grids. Alas, actually using these resources is far more difficult than desktop computing. Users have to deal with different architectures, network interconnects, etc. and also with the job scheduler that assigns CPUs to jobs. The user has to estimate the walltime it takes to run a job. As walltime is influenced by the degree of parallelism, by the input data, algorithms, and, more importantly, by unpredictable environmental issues (e.g. the amount of traffic and the communication network), it is difficult to correctly predict their walltime. With too short estimates, the scheduler terminates the job prematurely so that the user loses the computed results. Overestimation causes long waiting times in the queues of the cluster scheduler. Their paper proposes a solution to this problem that makes Grid computing more transparent. If an OpenMP application is about to exceed its time slice, it is automatically checkpointed and transparently migrated to either a new local reservation or a reservation on a different (possibly remote) system of the computational Grid. The key technical contribution is that the runtime system adapts the degree of parallelism such that the application can even run if the target platform has a different number of CPUs. Threads may be added or removed as necessary while a parallel region is active, making the application's degree of parallelism malleable. The approach only slows down the application by about 4% (reparallelization) plus 2% (checkpointing), which seems acceptable, given the overall gain in application throughput. The contribution by A. Vishnu et al. 3 explores InfiniBand becoming a very popular interconnect, due to its advanced features and being open standard. Large-scale InfiniBand clusters are becoming very popular, as reflected by the TOP 500 supercomputer rankings. However, even with popular topologies such as constant bi-section bandwidth Fat Tree, hot-spots may occur with InfiniBand, due to inappropriate configuration of network paths, presence of other jobs in the network and unavailability of adaptive routing. Their paper presents a hot-spot avoidance layer (HSAL) for InfiniBand, which provides hot-spot avoidance using path bandwidth estimation and multi-pathing using LMC mechanism, without taking the network topology into account. They propose an adaptive striping policy with batch-based striping and sorting approach, for efficient utilization of disjoint network paths. Integration of HSAL with MPI, the de facto programming model of clusters, shows promising results with collective communication primitives and MPI applications. S. Caton et al.'s 4 research demonstrates the use of a MATLAB-based image processing system to support applications deployed using Condor over a campus-wide grid. These resources are donated on a voluntary basis, and consequently the availability of workstations changes with time. At Cardiff University, they have access to a Condor system using approximately 2500 Windows XP workstations. Condor, as well as other distributed queuing systems and resource managers, does not consider particular requirements of image processing applications—thereby affecting the performance of such applications, especially when using a campus-wide grid. The main challenges are latency penalties for large data sets, granularity, and considerably shorter execution times than regular distributed computing applications. The use of volunteer resources adds further challenges to this list: unpredictable resource availability and longevity, multiple machine owners and administrators. They have augmented Condor to support deployment of such image processing applications using MATLAB. Their results demonstrate that opportunistic resources can facilitate disparate and short-running image processing applications without significant loss in performance and increase in cost. L. Nassif et al. 5 approach the resource selection problem challenges for achieving the best solution in the decision-making process, especially when considering many factors. A set of related works shows that a broker can select a resource for job execution considering aspects such as resource availability, access restrictions, job requirements, job execution time prediction, execution cost, and job execution success probability. This paper presents a taxonomy of resource selection systems to help in analyzing the related works and for classifying them concerning the decisor, the decision-making process, the number of resources selected, and the selection objectives. The authors then designed a solution, named MASK, based on user objectives. MASK is a broker middleware that uses the multi-agent system technology consisting of prediction, policy, and decision modules. The prediction model for job execution time was developed using the case-based reasoning paradigm. The prediction of a new job execution time is based on similar past job execution times. A policy model was developed based on fine-grain policies for access restriction verification purposes. The local policy verification anticipates the job submission, avoiding unsuccessful submissions. A decision model based on decision theory was developed. In this model, utility functions represent user preferences and the maximization of a multi-attribute utility model selects the machine with the best conditions to run a job. Results show that the prediction model is accurate and efficient in the prediction process and the distributed system runs faster than centralized approaches and considers access restrictions heterogeneities. In their paper, A. Elghirani et al. 6 mention that in large-scale data-intensive applications, data play a pivotal role in the execution of these applications, and data transfer is a primary cause of job execution delay. In environments such as the data Grids where execution of jobs that require large amounts of data is undertaken, a smart collaborative environment between the scheduling and data management services to achieve a synergistic effect on the performance of the Grid becomes essential. Their paper presents an intelligent data Grid framework where job scheduling and data and replica management are coupled to provide an integrated environment for efficient access to data and job scheduling. The data management service predicts and estimates the appropriate locations of replica and proactively replicates the data sets in these locations, while the intelligent Tabu Search-based scheduler incorporates information about the data sets, dispatching the jobs to the sites expected to promote minimum job execution time and better overall system utilization. This approach attempts to achieve a double optimization effect from both the replica management and the scheduling phases, while integrating scheduling and data replication to improve the performance of the Grid system. They have developed the cost model used by the framework and a heuristic to show the feasibility of the approach. Several experiments show that the approach improves the job makespan by 8–35%, depending on the scheduling and replication strategies used. The text by S. Ayyub et al. 7 presents the formation of a VO across otherwise autonomous organizations, providing an aggregation of many resources across a computational Grid. As the inclusion of more and more resources expands the Grid, the computational capabilities also increase. With this growth comes the ability to support increasingly larger applications. large applications require the of many computational for long of and make use of or large amounts of Several issues are for such applications. applications are to given the unpredictable nature of Grid In such a it becomes to applications, if so that a will only the from the last than from However, this both task and data management by the number of as well as The on how scheduling is while the may with the across the resources of multiple computational it is that the and data are across They have with three issues through various the long may be checkpointed into that the application this This should improve and scheduling in the of they a that between if they have become they a that third across with as long as the applications are with that are by both These three in in this paper, are important and of the to support large in The paper by et al. on for MPI applications. a parallel application's execution system to or the This is important because process that is used to applications to dynamic execution environments is by the of the application's process by the application's to or the availability of They have as an to the process and a for MPI applications. is integrated with the Internet a framework for dynamic application The approach and This paper presents the of the work and the approach to in MPI applications. it the for and its runtime and demonstrate the of results from a data parallel that has a regular communication are presented. We would to the authors for papers on their research in cluster computing and the Grid to this special and the for providing and in to this special we would to Geoffrey Fox for providing us an to this special

Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.