August 6, 2008 5:25pm Wolfgang Gentzch, RENCI, Duke & OGF Board of Directors

During the past 10 years, hundreds of grid projects have come and gone, passing away after funding ran dry. Most didn’t have a realistic strategy for sustainability, let alone a viable business model for their infrastructures, tools, applications or services. Often, the only asset left after the project’s end was the hands-on expertise gained by those involved, which is certainly valuable in the long run, but doesn’t justify the effort and funding. So far, in my opinion, grids didn’t keep up to their full promise.Full Article...

What clouds and grids can learn from each other

During the past 10 years, hundreds of grid projects have come and gone, passing away after funding ran dry. Most didn’t have a realistic strategy for sustainability, let alone a viable business model for their infrastructures, tools, applications or services. Often, the only asset left after the project’s end was the hands-on expertise gained by those involved, which is certainly valuable in the long run, but doesn’t justify the effort and funding. So far, in my opinion, grids didn’t keep up to their full promise.

What went wrong?

Sure, grids, by their very nature, are complex to design, build and maintain; and applications are cumbersome to run. It might take another 10 years of trial and error (and re-writing grid middleware?) to navigate the labyrinth of new technologies and paradigms, such as Utility Computing, Autonomic Computing, ASP, SOA, SOI, SaaS, PaaS, HaaS, Outsourcing, Hosting, Virtualization, Web 2.0 and Mashups...

And obviously, grids followed a natural curve: technology trigger, inflated expectations, trough of disillusionment, slope of enlightenment, and– finally–plateau of productivity. Some technologies disappear completely. For me, however, we are now facing a potential renaissance, triggered by the appearance of... clouds.

Cloud computing, as offered by companies like Sun, IBM, Amazon, Google, CloudCamp, and others, will soon become an important component of R&D, adding a new, ‘external’ dimension of flexibility by enhancing one’s ‘ home’ resource capacity when needed Existing businesses will use them during peak demand, service providers will host their applications on them and provide Software as a Service (SaaS), and start-ups will integrate them without needing to buy resources upfront.

Plumbing

While grids provide the ‘plumbing’ to enable access to distributed resources, clouds denote  service on a pay-per-use basis. Grids stand out because of their flexible, dynamic, feature-rich resources; they are thus innately complex. This complexity, however, must be hidden from the end-user (e.g. in the form of a cloud) if we want acceptance of this infrastructure.

Cloud applications will likely follow similar stages as grid-enabling ones. Just as challenging, though, are the cultural, mental, legal, and political aspects. It is difficult to imagine users easily entrusting their corporate assets and sensitive data to cloud service providers. Today, the status of clouds seems to be similar to that of grids in the early 2000s: a few simple and well-suited application run on clouds, but more complex and demanding applications will face many barriers to overcome.

Sure, clouds are easier to deploy than grids: Clouds are user-friendly, service oriented, on-demand. But if clouds aim to ‘replace’ grids, they face similar challenges. Even a single grid service running in a cloud image (or using a service in the cloud) will quickly face the roadblocks we know from grids today.  

The role of DEISA

One promising solution might be the Distributed European Infrastructure for Supercomputing Applications, or DEISA, which is moving from a grid to a cloud. DEISA Extreme Computing Initiative, or DECI, is successfully offering millions of supercomputing hours to the European e-Science community and helping scientists gain new scientific insights.

Why is DECI successful? Several reasons, in my opinion: it has a targeted focus on specific, long-running, supercomputing applications; many applications run on one single system;  it has user-friendly access to resources through DESHL and UNICORE; it gives consortium partners full autonomy; and there is an application task force (ATASKF) that helps users plug in to the supercomputing infrastructure. If all this were to stay, DEISA would have some real staying power. And then, we might have a DEISA Cloud which will become an external HPC node within your grid application workflow.

Lessons learned

So what can clouds learn from grids?

• We should lower our expectations in the first place.
• No claim for universality, please!
• We should keep clouds simple, focused, with services easily embedded as standard components.
• When building larger, more general infrastructures, we might think of a “grid of clouds” –a hierarchy which leaves as much autonomy as possible to smaller service components and contributing and collaborating partners.
• Leave coordinating functions to the existing overarching grid, thus bypassing mental, social, political and legal barriers which usually arise through more direct integration.

The good news is: clouds will help grids mature, and grids will help clouds avoid some  teething troubles. Clouds will teach grids that to be widely accepted and  sustainable, grids must be simple, user-friendly, service oriented, scalable, on-demand, with simple applications. And grids teach clouds that by raising the bar of expectations too high, they could easily plunge into the trough of disillusionment.

With this sea-change ahead of us, there will be a continuing need for support of the work of the Open Grid Forum (OGF). Certainly, existing processes can be agreed upon and developed that are streamlined, shortened, and more efficient outside it. But only an agreed-upon set of standards will enable e-infrastructures and grid applications to be easily built from different components, and move towards an agile, federated service platform. Only standards, such as those developed in OGF, will guarantee interoperability of technology components best suited for your applications–and thus reduce dependency on proprietary building blocks, keep costs under control, and increase research and business flexibility.

Wolfgang Gentzsch, DEISA Project, Duke University, and Board of Directors, Open Grid Forum

Show Article Preview...

February 11, 2008 3:20pm Domenico Talia

Grids are moving from a computation and data management platform to a pervasive, worldwide information and knowledge management infrastructure. This trend needs new models and technologies for enabling Grid computing to support fine grained Grid nodes and resources that can be connected in a simple, dynamic, and self-organizing way. Self-organizing knowledge grids can help to build up a pervasive infrastructure for managing and diffuse information and knowledge.Full Article...

Domenico Talia
University of Calabria, talia@deis.unical.it

Grids are moving from a computation and data management platform to a pervasive, worldwide information and knowledge management infrastructure¹. This trend needs new models and technologies for enabling Grid computing to support fine grained Grid nodes and resources that can be connected in a simple, dynamic, and self-organizing way. Self-organizing knowledge grids can help to build up a pervasive infrastructure for managing and diffuse information and knowledge.

Principles
A self-organizing knowledge grid is a system of irregularly placed, asynchronous, locally or remotely interacting knowledge processing elements. We can model this organism as a collection of "computational particles" sprinkled irregularly on a surface or mixed throughout a volume. The particles are possibly faulty, sensitive to the environment, and may take autonomous actions. In general, individual grid particles may be mobile. Each particle has large or modest computing power and generally a limited amount of memory although in some cases they manage large storage. The particles are not necessarily synchronized, although they compute locally or in cooperation with other particles and interact each other according different patterns.

Self-organizing knowledge grids do not need to have definite form or distinct shape; like self-organizing clouds of insects they can act as an aggregate of formless particles that constitute a shapeless mass of "computing protoplasm". The particles are all programmed to perform some tasks or provide some service and to interact with other particles or the external environment. Each one has means for storing local state. Each grid particle can communicate with nearby neighbors or remote elements. In self-organizing knowledge grids of small particles, they may communicate via wireless media and wired interconnections. For our purposes here, we can assume there is some interaction radius R that is large compared to the size of single particles and small compared to the size of an entire area or volume, and that two particles are considered to be neighbors if they are within distance R. The radius represents the particle's horizon that defines the maximum distance for discovering and interacting with other grid particles.

In self-organizing knowledge grids the number of particles can be very large. Thus, the entire self-organizing knowledge medium can be regarded as a large-scale knowledge Grid system that may change its form/configuration depending from particle behavior and from the external environment. Previous researches in Grid computing, peer-to-peer networks, global computing, Web services, massively parallel computing, knowledge discovery, pervasive computing², artificial intelligence, and self-organizing systems are sources of ideas for dealing with self-organizing knowledge grids. A self-organizing knowledge grid is a greater challenge than single systems mentioned above, however, because its mechanisms embody features and functionalities of all those systems. For example, it is must exploit self-organization, it must be independent of configuration and reliability of the individual particles, it exploits "smart dust" technologies³, it must open to new particles or new clusters or aggregates of particles, and adaptive to the problems to be solved or the user requirements.

Towards a Programming Model
Today Grids differ significantly from self-organizing knowledge grids in terms of application and machines granularity, connectivity, security, mobility, etc. These differences imply the necessity to devise new tools and new models for deployment and application development on such class of systems^4,5. In other words, mechanisms for setting up and programming self-organizing knowledge grids are needed. Programming those systems must be done either by programming a single particle or an aggregate of grid particles. This new paradigm could stimulate advancements also in the area of service computing towards the definition of more flexible and light service models.

Each particle can query and potentially modify its state and the state of another particle depending on the particle-to-particle interaction and on the state of the neighbor particles. Self-organization is the basic paradigm for particle programming and evolution. Since a limited or a massive amount of particles can compose a self-organizing knowledge grid, a programming model for such a kind of system must be able to manage small sized and very large sized self-organizing knowledge grids. An appropriate programming model for self-organizing knowledge grids comes from self-organizing programming models and from cellular programming languages that include constructs to define cell or particle behavior and particle-to-particle interaction patterns and evolution. A programming model must include the main following features

• Mechanisms for grid particle discovery,
• Mechanisms for particle aggregation,
• Constructs for single and aggregate particle evolution,
• Protocols and operations for particle communication, and
• Techniques for data and knowledge storing and discovery.

Those features must be devised departing from current Grid programming models even if they must guarantee the basic Grid computing principles.

A programming model for self-organizing knowledge grids should integrate cellular automata and nature inspired paradigms with peer-to-peer programming mechanisms and ontology-based Grid programming techniques. Those research fields may contribute in constructing a self-organizing knowledge grid programming model that will drive the development of self-organizing knowledge grids as future generation pervasive Grids.

Self-organizing knowledge grids can be effectively used for implementing knowledge-intensive pervasive applications. For example, they can provide the basic infrastructure for ambient intelligence applications. In this scenario each resource (i.e., a sensor, a computer, or a generic device) can be seen as a grid particle that exports its specific features and services and, at the same time, implements all the mechanisms defined by the self-organizing knowledge grid computing model.

Other significant application areas of self-organizing knowledge grids are decentralized and mobile information systems handing with large-scale information sources and collaborative information systems where each grid particle will represent a single node of the system acting as information provider or requestor or as a mediator between nodes that cooperate in achieving a common goal.

References
1. M. Cannataro and D. Talia, "Semantics and Knowledge Grids: Building the Next-Generation Grid", IEEE Intelligent Systems, vol. 19, no 1. pp. 56-63, 2004.
2. R. Want, T. Pering, "System Challenges for Ubiquitous & Pervasive Computing" Proc. ICSE, pp. 9-14, 2005.
3. J. M. Kahn, R. H. Katz and K. S. J. Pister, "Next Century Challenges: Mobile Networking for Smart Dust", ACM/IEEE Intl. Conf. on Mobile Computing and Networking (MobiCom 99), Seattle, WA, August 17-19, 1999.
4. Le. W. McKnight, J. Howison, S. Bradner, "Wireless Grids", IEEE Internet Computing, July/August 2004.
5. J. Blythe et al., "Transparent Grid Computing: A Knowledge-Based Approach," Proc. 15th Innovative Applications of Artificial Intelligence Conference (IAAI 03), AAAI Press, 2003.

Domenico Talia is a professor of Computer Engineering at the Faculty of Engineering, University of Calabria, Italy, http:// si.deis.unical.it/~talia Show Article Preview...

February 1, 2008 4:15pm Ignacio M. Llorente

Reprinted with permission by the author
This article describes the relationships between:

• Research projects
• Technology products
• Standardization bodies
• Open-source communitties

Research projects produce original concepts and technology, which are then subject to standardization if they are found to be relevant enough to be used in real world applications. Therefore, is obvious that research projects and standardization processes are tightly related. In general, research projects should lead to products that may be initial reference implementations; hence, contribution to standards should be an outcome in ICT projects. This outcome (standard contribution) is a way to transfer intellectual contribution and the technological innovations if the project to a wider audience.

Depending on the aim and scope of the project, there exist different ways to participate in standardization bodies, which are part of an IT research process.

Full Article...

Reprinted with permission by the author
I would like to start the blog of the dsa-research group by describing our view of the relationships between:

• Research projects
• Technology products
• Standardization bodies
• Open-source communitties

Research projects produce original concepts and technology, which are then subject to standardization if they are found to be relevant enough to be used in real world applications. Therefore, is obvious that research projects and standardization processes are tightly related. In general, research projects should lead to products that may be initial reference implementations; hence, contribution to standards should be an outcome in ICT projects. This outcome (standard contribution) is a way to transfer intellectual contribution and the technological innovations if the project to a wider audience.

Depending on the aim and scope of the project, there exist different ways to participate in standardization bodies, which are part of an IT research process.

• Exchange ideas and experiences between the research process staff and a standardization body working group relevant to the area of the research.
• The research process participants can provide requirements and use cases for the standardization body to take into account.
• The research process can benefit from previous standards and also be guided by best practices specified by the standardization bodies.
• The outcome of the research project may help to evolve and enhance the specifications being considered by the standardization bodies.
• The research process can lead to the creation of new working and research group dedicated to the particular field open by such research process.

The different feedbacks between a research project and standardization bodies (as well as the other participants and stakeholders) can be seen in below figure. A research project is a research and development effort that seeks to advance the knowledge and technology in a given domain. There should be two major outcomes from a research project:

• One is rendered in a software product incorporating the technological innovations of the project. This, in turn, conveys intellectual property to standardization bodies (by providing a reference implementation), Industry and Academia (through the transferring of the technology found in the product) and OSS Communities (it may start an incubator project or even a new community devoted to the new field open by the research).
• A second outcome of a research project more directly transfers the intellectual contribution to the Industry and Academia through research papers. This intellectual conveyance influences also the standardization bodies through exchange of ideas and experiences that allows the evolving and enhancing of specifications and provides requirements and use cases.

Ignacio Martín Llorente

Show Article Preview...

January 8, 2008 10:27am Ian Foster, Argonne National Laboratory and The University of Chicago

Reprinted with permission by the author

You’ve probably seen the recent flurry of news concerning “Cloud computing.” Business Week had a long article on it (with an amusing and pointed critique here). Nick Carr has even written a book about it. So what is it about, what is new, and what does it mean for information technology?

The basic idea seems to be that in the future, we won’t compute on local computers, we will compute in centralized facilities operated by third-party compute and storage utilities. To which I say, Hallelujah, assuming that it means no more shrink-wrapped software to unwrap and install.

Needless to say, this is not a new idea. In fact, back in 1960, computing pioneer John McCarthy predicted that “computation may someday be organized as a public utility”—and went on to speculate how this might occur.Full Article...

Reprinted with permission by the author

There's Grid in them thar Clouds*

You’ve probably seen the recent flurry of news concerning “Cloud computing.” Business Week had a long article on it (with an amusing and pointed critique here). Nick Carr has even written a book about it. So what is it about, what is new, and what does it mean for information technology?

The basic idea seems to be that in the future, we won’t compute on local computers, we will compute in centralized facilities operated by third-party compute and storage utilities. To which I say, Hallelujah, assuming that it means no more shrink-wrapped software to unwrap and install.

Needless to say, this is not a new idea. In fact, back in 1960, computing pioneer John McCarthy predicted that “computation may someday be organized as a public utility”—and went on to speculate how this might occur.

In the mid 1990s, the term grid was coined to describe technologies that would allow consumers to obtain computing power on demand. I and others posited that by standardizing the protocols used to request computing power, we could spur the creation of a computing grid, analogous in form and utility to the electric power grid. Researchers subsequently developed these ideas in many exciting ways, producing for example large-scale federated systems (TeraGrid, Open Science Grid, caBIG, EGEE, Earth System Grid, …) that provide not just computing power, but also data and software, on demand. Standards organizations (e.g., OGF, OASIS) defined relevant standards. More prosaically, the term was also co-opted by industry as a marketing term for clusters. But no viable commercial grid computing providers emerged, at least not until recently.

So is “cloud computing” just a new name for grid? In information technology, where technology scales by an order of magnitude, and in the process reinvents itself, every five years, there is no straightforward answer to such questions.

Yes: the vision is the same—to reduce the cost of computing, increase reliability, and increase flexibility by transforming computers from something that we buy and operate ourselves to something that is operated by a third party.

But no: things are different now than they were 10 years ago. We have a new need to analyze massive data, thus motivating greatly increased demand for computing. Having realized the benefits of moving from mainframes to commodity clusters, we find that those clusters are darn expensive to operate. We have low-cost virtualization. And, above all, we have multiple billions of dollars being spent by the likes of Amazon, Google, and Microsoft to create real commercial grids containing hundreds of thousands of computers. The prospect of needing only a credit card to get on-demand access to 100,000+ computers in tens of data centers distributed throughout the world—resources that be applied to problems with massive, potentially distributed data, is exciting! So we’re operating at a different scale, and operating at these new, more massive scales can demand fundamentally different approaches to tackling problems. It also enables—indeed is often only applicable to—entirely new problems.

Nevertheless, yes: the problems are mostly the same in cloud and grid. There is a common need to be able to manage large facilities; to define methods by which consumers discover, request, and use resources provided by the central facilities; and to implement the often highly parallel computations that execute on those resources. Details differ, but the two communities are struggling with many of the same issues.

Unfortunately, at least to date, the methods used to achieve these goals in today’s commercial clouds have not been open and general purpose, but instead been mostly proprietary and specialized for the specific internal uses (e.g., large-scale data analysis) of the companies that developed them. The idea that we might want to enable interoperability between providers (as in the electric power grid) has not yet surfaced. Grid technologies and protocols speak precisely to these issues, and should be considered.

A final point of commonality: we seem to be seeing the same marketing. The first “cloud computing clusters”—remarkably similar to the “grid clusters” of a few years ago—are appearing. Perhaps Oracle 11c is on the horizon?

What does the future hold? I will hazard a few predictions, based on my belief that the economics of computing will look more and more like those of energy. Neither the energy nor the computing grids of tomorrow will look like yesterday’s electric power grid. Both will move towards a mix of microproduction and large utilities, with increasing numbers of small-scale producers (wind, solar, biomass, etc., for energy; for computing, local clusters and embedded processors—in shoes and walls?) co-existing with large-scale regional producers, and load being distributed among them dynamically. Yes, I know that computing isn’t really like electricity, but I do believe that we will nevertheless see parallel evolution, driven by similar forces.

In building this distributed “cloud” or “grid” (“groud”?), we will need to support on-demand provisioning and configuration of integrated “virtual systems” providing the precise capabilities needed by an end-user. We will need to define protocols that allow users and service providers to discover and hand off demands to other providers, to monitor and manage their reservations, and arrange payment. We will need tools for managing both the underlying resources and the resulting distributed computations. We will need the centralized scale of today’s cloud utilities, and the distribution and interoperability of today’s grid facilities.

Some of the required protocols and tools will come from the smart people at Amazon and Google. Others will come from the smart people working on grid. Others will come from those creating whatever we call this stuff after grid and cloud. It will be interesting to see to what extent these different communities manage to find common cause, or instead proceed along parallel paths.

*An obscure cultural reference: the phrase “There’s gold in them thar hills” was first uttered, according to some, by an old prospector in the 1948 movie “Treasure of the Sierra Madre”, starring Humphrey Bogart.

Show Article Preview...

December 27, 2007 8:40pm Geoffrey Fox and Marlon Pierce, Indiana University

Grids are taking too long to solve the wrong problem at the wrong point in stack with a complexity that makes friendly usability difficult. We furthermore observe that Grids (as envisioned c. 2001) are being pressured by both emerging new computing resources (multicore, cell processors, GPUs, reconfigurable computing, etc) and alternative approaches to service architectures (collectively, Web 2.0). We thus believe it is time to reappraise Grids—both the nature of the resources that they aggregate and the middleware that glues these resources together.

In spite of the unclear technology directions, e-Science and more generally e-moreorlessanything are thriving with the advantages of distributed enablement of many fields very clear.
Abstracted from http://grids.ucs.indiana.edu/ptliupages/publications/GridSandwich.pdfFull Article...

Grids meet Too much Computing, Too much Data and never Too much Simplicity

Geoffrey Fox^1,2,3 and Marlon Pierce^1
1Community Grids Laboratory
²Department of Computer Science
³School of Informatics
Indiana University
{gcf, marpierc}@indiana.edu

Challenge
Let us discuss:
Grids are taking too long to solve the wrong problem at the wrong point in stack with a complexity that makes friendly usability difficult. We furthermore observe that Grids (as envisioned c. 2001) are being pressured by both emerging new computing resources (multicore, cell processors, GPUs, reconfigurable computing, etc) and alternative approaches to service architectures (collectively, Web 2.0). We thus believe it is time to reappraise Grids—both the nature of the resources that they aggregate and the middleware that glues these resources together.

In spite of the unclear technology directions, e-Science and more generally e-moreorlessanything are thriving with the advantages of distributed enablement of many fields very clear.
Abstracted from http://grids.ucs.indiana.edu/ptliupages/publications/GridSandwich.pdf

1. Too much Computing
Over the last 20 years we have seen an increase in the power of computers by roughly a factor of a million and this has enabled computational science to become a major force in an ever increasing number of academic fields. This has been nurtured by substantial institutional investment in computational infrastructure spanning campus, laboratory and national scale. This investment involves the use of both grids and parallel computing to increase computing capabilities by aggregating computers together in a distributed or local fashion respectively. Different problem classes are suited to different aggregation architectures; closely coupled problems are suited to closely coupled (i.e. low communication latency) parallel computer systems, loosely coupled problems are suited to loosely coupled (i.e. high communication latency) distributed computer systems. This hybrid strategy ensures an e-infrastructure (Cyberinfrastructure) of broad applicability. This capability must support simulations, the fusion analysis and analysis of data from repositories, instruments and sensors, the distributed users as well as their integration. Such a scenario supports emoreorlessanything including e-Science, e-Research, e-Business and even edigitalmedia and e-OpenSocial for the Net-Gens (or more appropriately for this article the e-Gens).

The USA Federal HPCC (High Performance Computing and Communication) initiative started in 1991 was a major driver for both technologies and applications for computational science with parallel computing and simulation as the focus. Following HPCC, Grids and Cyberinfrastructure/e-Infrastructure blossomed and there was a growing recognition of the importance of data which was not stressed with HPCC. In spite of these changes, the community was obsessed with performance shown in many ways. The well known Top500 list measures the stature of your institution by the performance of its supercomputer on matrix algebra. Much of Grid technology aims at linking computers across multiple administrative domains leading to increased total performance but at the cost of significant security challenges that translates into complex Grid virtual organization technology. Further there has been great interest recently in harnessing the potential of graphics boards to deliver very high performance on cluster nodes. When we started this (say around 1980 with the Cray-1 and XMP), we definitely had too little computing as only unrealistic 2D simulations were possible and this motivated much of the early parallel computing research. However the situation is changed and perhaps now most applications have the opposite problem – there is in fact Too much Computing.

Some fields can still use essentially unlimited (petascale) parallelized computing; a few examples are stewardship of the nuclear stockpile for national security and Quantum Chromodynamics or globular cluster simulations in fundamental science. However even in e-Research these are probably the exception; the common (average) case of computational science does not need petaflop computers. In particular the “new sciences” such as life science and social science definitely need e-Science but typically modest peak performance. Similarly, the digital humanities have a growing need for distributed computing infrastructure (mass storage, networking, middleware, and some processing) generally but no need for very high end computation. A field needs to be quite mature to develop algorithms and accurate phenomenology needed for useful petaflop computing. We have already enough computing in many parts of e-Research; we need to develop other aspects of Cyberinfrastructure. The situation is even clearer for mass market systems whose CPU chips will be 32-128way parallel in 5 years time, but we currently have little idea how to use them on commodity systems – especially on clients. There are perhaps at most 2 releases of standard software (such as Windows or Microsoft Office) in this time span and we need to find value in these new chips for the broad market so that the multicore instantiation of Moore’s law (roughly constant clock speed, increasing core density) will lead to improved performance. We need to address Too much Computing here with approaches that can be implemented in next 3-5 years.

2. Too much Data

Figure 1: Information architecture combining Web 2.0 and Grid Concepts. Wisdom is obtained by fusing and transforming data that comes from sensors, instruments, services, Grids and Clouds. Data is transformed by filters that perform data analysis, transformation, assimilation or production from simulations. Traditional Grids expose constituent services as illustrated by Filter Service surrounded by other (fluff services fs) services in dashed rectangles. Compute, Storage and Filter clouds hide this detail and expose data interfaces. Discovery is needed for both Clouds and Grid services and all components are linked by messages

The deluge of increasing data is a clearly recognized feature of e-moreorlessanything including 30 billion web pages with an increasing number of two-way interactive pages from sites like Flickr and YouTube. In e-Science, there are many observatories, satellites, sensors, high throughput screens and instruments whose data products also track Moore’s law with the LHC soon to generate 10 petabytes of data per year. These data sources of course feed repositories that support scientific discovery which also itself produces further auxiliary data adding to the Too much Data scenario. However there are interesting differences between the Data and Compute Deluges. Data is intrinsically distributed as both sources and the managing organizations are distributed. Further multidisciplinary research and commodity mashups link information of different kinds at different places. The data challenges involve its management, federation, access and analysis. This involves computing but architecturally the data and computing are naturally co-located to avoid large communication costs. Intel proposed the RMS (Recognition Mining Synthesis) class of applications corresponding to data-mining and gaming as enabled by future multicore chips. It is clear how this works for servers and especially servers near data sources/repositories but Internet Clouds, Supercomputer centers and Clients are not clearly co-located near relevant data. Clients could host data from local (say video and environmental) sensors plus data fetched from the network (Intranet and Internet). The latter might be mined automatically by the client to provide an "intelligent environment" for user sessions. Most data mining algorithms can be efficiently parallelized as long as the datasets are large enough. Thus we imagine that datamining of this "Too much data" will use up the "Too much computing"on client-side PC’s.

Internet storage and computing clouds as well classic supercomputer centers can be used as shown in figure 1 where the data deluge flows through the computing deluge leading to wisdom and effective decisions. We term the Clouds through which data, information and knowledge flows as Filter Clouds to indicate that they input and output streams of data. Clouds have an attractive interface defining the input and output data but how they and traditional computer centers cope with Too much nonlocal data is not clear. Perhaps this will drive the deployment of networks of even higher performance; this could be practical as it is bandwidth not latency that is relevant in costs scenarios represented by figure 1.

3. Never Too much Simplicity
There is no precise definition of Web 2.0, but it is operationally defined by a set of technologies (JSON, AJAX, etc) and a wide range of Web sites supporting user interaction among themselves (social networking) and with resources such as images and video. Media sharing and bookmarking are structured to allow communities (i.e. virtual organizations) to grow up around resources. Similarly peer production sites allow users (people in communities) to select and rate presented information. There are also very popular capabilities like Blogs and Wikis supporting communication either broadly or within an organization. These capabilities provide attractive tools enhancing collaboration in all areas of e-moreorlessanything.

Google maps illustrate the power of Web 2.0 deployment of important capabilities – in this case geographic information systems. Google Maps and related systems have transformed GIS technologies not just through their interactivity (enabled through innovative uses of AJAX, JSON, and image tile caching), but also through the simplicity they bring to development. As we will discuss later, Google Maps is by far the most popular API for mash-up building. Its sophistication and power are accessible through simple APIs and data models (KML) that can be manipulated by anyone with basic programming skills.

Web 2.0 has also encroached on more traditional territory of cyberinfrastructure. Web 2.0, with consequences that may be as transformational as Google Maps has been to GIS. Start Pages challenge portals for access to services and Web 2.0 mashups challenge workflows for integration of services. Further Web 2.0 cloud systems such as Amazon web services directly support the distributed storage, data management, and computing that were up to now the distinctive feature of Grids.

These clouds address "commodity usage" and do not currently provide massively parallel systems for the high end users. However, they do make the important distinction (missing in, for example, the OGSA use cases) between deployment and development. Deployment has very sophisticated challenges, and the Global/Open Grid Forum has attempted to define a standard set of use cases for this. Unfortunately the deployment scenarios often overlap with the Grid development process—that is, building a Grid and developing for a Grid are not adequately distinguished. In contrast, cloud systems, while at least as sophisticated as conventional Grid deployments, do a much better job at defining simple developer interfaces. Resource allocation, data replication, and security are all sophisticated capabilities in computing clouds, but they are hidden from the typical application developer.

Web 2.0 capabilities and technologies are simpler than their Grid counterparts and are user friendly for developers and users. For example as shown in figure 1, Cloud systems are "Grid islands" which only expose the key interface to input and output data and control information. Grids adopt the Web service philosophy where all services are exposed allowing greater power but at greater complexity. Originally we expected that Web Services would dominate a new generation of Enterprise software and that distributed systems would leverage commercial investment in this field and be built in terms of Web Services. However this is not what happened. There is general agreement on service architectures but Web Services were characterized by very specific specification of operating environment through WSDL Interfaces and SOAP header elements. This leads as shown in figure 1 to Grids where individual services are visible and potentially interoperable; as discussed above Clouds have a more limited but friendlier interface. Grid services tend to expose even more detail than Web services. Web 2.0 systems may consider such information important but leave such details to the application allowing a much simpler hosting environment. Rather than Web services dominating practical distributed systems, they will have some use but their complexity and Grid refinements are likely to be ignored as they are accessed by a sea of Web 2.0 services and mashups; Web 2.0 achieves interoperability by simplicity rather than the detailed WS-* specifications. Web 2.0 teaches us that we only need interoperability at a few sweet spots; user data formats at cloud interfaces and the lowest level protocols typified by HTTP transport. Web 2.0 teaches to re-use wherever possible; scripting rather BPEL for workflow for example. Web 2.0 teaches us the power of distributed simple specification; developer uploads of API’s to http://programmableweb.com rather than the stultified UDDI; use of microformats rather than large ontologies; rapid deployment of modules rather than lengthy development of universal systems.

There is still substantial uncertainty but history teaches us that simplicity is often a winning principle. Grids and e-infrastructure should examine its implications for both low level protocol and the high level user and user data interfaces. Too much computing and Too much data have nontrivial implications for computer and networking hardware as well associated software and standards. Perhaps five years ago, we settled on software and decided on needed standards too quickly; we should expect change and mix research with robust deployment this time around.

4. Reappraising Grids We will use the term "Grid" here to mean both deployed infrastructure (such as TeraGrid and OSG) and the binding middleware (Globus, Condor, SRB, etc).

Arguably, Grid middleware (the TeraGrid CTSS and OSG VDT) represent the successful culmination of more than a decade’s worth of investment in Grid technologies. Although there is room for improvement (both GRAM and WS-GRAM have scaling limitations and information services vary widely between TeraGrid, OSG, and EGEE, for example), Grid software is stable, well-packaged, relatively easy to install, and widely deployed on production systems. The infrastructure problems of data and computing time allocations, account management, and wide-area single sign-on have all been addressed. Interesting new services (such as the batch queue wait-time prediction service, QBETS) add value to Grids. Workflow tools (Kepler, Taverna, XBaya) all provide visual interfaces for integrating Grid and Web Services into composite tasks. There is still work to do to integrate all of these capabilities into tools that are useful for the end user.

From a different perspective, however, one can also argue that the major trends of Grid research and development have reached dead-ends, or at least rapidly diminishing returns.

Converting Users into Developers: A key problem with Grid middleware and Web services is the complication. It is very difficult for computational scientists to use, extend, and customize Grid software because the available tools and programming libraries are too complicated. Consequently, Grid development is largely in the hands of computer scientists and systems experts rather than domain scientists. The poor scientist is too often treated as a "customer" of the Grid development team. He or she may be the source of requirements and use cases, and may be drafted as a beta tester for the wonderful Grid infrastructure but will not be involved in the day-to-day development. This is an unnecessary waste of human resources.

This state of affairs is not by accident: most Grid developers (including the authors) have in the past advocated the adoption of “Enterprise” technologies (particularly Web Services and server-side portal technologies) for building Grids. As we have argued above, it is not clear at all that Enterprise technologies are developing along the most promising paths.

One of the many attractive aspects of Web 2.0 is the simplicity of its programming interfaces: anyone with general programming expertise can build a mashup. There really needs to be an equivalent mind set in the Grid development world.

To do this, organizations like the Open Grid Forum should change their focus. The problem (in our view) is that the OGF has concentrated too much on infrastructure deployment use cases (see OGSA Use Case documents for example) and not on developer use cases.

Virtual Organizations or Social Networks: Grids have been primarily focused on PKIderived virtual organizations that are used to connect real organizations with existing bureaucracy, managements, etc. From the user’s point of view, this mainly is useful for unifying accounts and providing single sign on. While useful and perhaps time-saving, this is not extremely compelling.

Going forward, Grids should instead take a lesson from Web 2.0 social networks. Unlike Grid VO’s, these are user driven and focus on users sharing information, data, and services.

Multiscale Grids: Within 5 years, it will be possible to buy servers and workstations with dozens of cores. Parallel computing, displaced by Grid (wide area distributed) computing for the last decade, is enjoying a renaissance. On the other hand, scaling algorithms for most scientific domains are much further away. Thus, we can argue that the “sweet spot” for many parallel computing jobs in the next decade will be about the same size as a single multicore server. Only very mature parallel computing fields will be able to utilize petascale computing.

Thus we foresee the need for Grid software itself to be much more scalable, integrating desktops and servers and individual cores. Most of today’s Grid services do not really add value until one attempts to integrate entire computing centers.

Application Services, not Infrastructure Service: Grid services provide job management, treating the underlying science code as an abstract job (typically in a queuing system) needing to be managed. This should be inverted. The goal should be making computational science services with simple, self-contained service APIs that can be programmed to. These application services can then be bound to underlying Grid services as necessary.

Cyberinfrastructure for Data Centric eScience and the Social Sciences and Humanities: The current cyberinfrastructure deployments are largely aimed at traditional computational users (i.e. scientists with codes to run on parallel machines and data to store on UNIX-like data archives) for obvious reasons. Unfortunately, this does not meet the needs of social sciences, humanities, and other academic research fields. These groups have some computing requirements (image processing, data mining), but they are more data-centric. They furthermore are probably in greater need of a simple to use, national scale IT infrastructure.Show Article Preview...

December 19, 2007 1:31pm Wolfgang Gentzch, RENCI, Duke & D-Grid

After publishing a report on major grid initiatives on the RENCI webpage earlier this year, I am getting emails from colleagues who are currently setting up a grid infrastructure (or are planning to do so), be it in research or in industry enterprises, asking questions especially about a good strategy for making grids sustainable. Here, I am trying to answer by presenting "my" 10 important rules on how to build a sustainable grid (and I know there are more than 10). These rules are derived from mainly four sources: my research on major grid projects published on the RENCI website, the e-IRG Workshop on "A Sustainable Grid Infrastructure for Europe" in Heidelberg in spring, the 2nd International Workshop on Campus and Community Grids at OGF20 in Manchester, and my personal experience with coordinating the German D-Grid Initiative. The rules presented here are mainly non-technical, because I believe most of the challenges in building and operating a grid are in the form of cultural, legal and regulatory barriers. Full Article...

Top 10 Rules for Building a Sustainable Grid
Wolfgang Gentzsch
RENCI, Duke & D-Grid, October 2007

After publishing a report on major grid initiatives on the RENCI webpage earlier this year, I am getting emails from colleagues who are currently setting up a grid infrastructure (or are planning to do so), be it in research or in industry enterprises, asking questions especially about a good strategy for making grids sustainable. Here, I am trying to answer by presenting "my" 10 important rules on how to build a sustainable grid (and I know there are more than 10). These rules are derived from mainly four sources: my research on major grid projects published on the RENCI website, the e-IRG Workshop on "A Sustainable Grid Infrastructure for Europe" in Heidelberg in spring, the 2nd International Workshop on Campus and Community Grids at OGF20 in Manchester, and my personal experience with coordinating the German D-Grid Initiative. The rules presented here are mainly non-technical, because I believe most of the challenges in building and operating a grid are in the form of cultural, legal and regulatory barriers.

Rule 1: Identify your specific benefit. Your first thought should be about your users and your organization. What’s in it for them? Identify the benefits which fit best: transparent access to and better utilization of resources; almost infinite compute and storage capacity; flexibility, adaptability and automation through dynamic and concerted interoperation of networked resources; cost reduction through utility model; shorter time-to-market because of more simulations at the same time on the grid. Grid technology helps to adjust an enterprise’s IT architecture to real business requirements (and not vice versa). For example, global companies will be able to decompose their highly complex processes into modular components of a workflow which can be distributed around the globe such that on-demand availability and access to suitable workforce and resources are assured, productivity increased, and cost reduced. Application of grid technology in these processes, guarantees seamless integration of and communication among all distributed components and provides transparent and secure access to sensitive company information and other proprietary assets, world-wide. Grid computing is especially of great benefit for those research and business groups which cannot afford expensive IT resources . It enables engineers to remotely access any IT resource as a utility, to simulate any process and any product (and product life cycle) before it is built, resulting in higher quality, increased functionality, and cost and risk reduction.

Rule 2: Evangelize your decision makers first. They give you the money and authority for your grid project. The more they know about the project and the more they believe in it (and in you) the more money and time you will get, and the easier becomes your task to lead and motivate your team and to get things done. Present a business case (current deficiencies, specific benefits of the grid (see Rule #1), how much will it cost and how much will it return, etc. They might also have to modify existing policies, top down, to make it easier for users (and providers) to cope with the challenges of and to accept and use the new services. For example, why would a researcher (or a department in an enterprise) stop buying computers when money continues to be allocated for buying it? This policy should be changed to support a utility model instead of an ownership model. If you are building a national grid, for example, convincing your government to modify its research funding model is a tough task.

Rule 3: Don’t re-invent wheels. In the early grid days, many grid projects tried to develop the whole software stack themselves: from the middleware layer, to the software tools, to grid-enabling the applications, to the portal and Web layer…and got troubled by the next technology change. Today, so many grid technologies, products and projects exist that you want to start looking for similar projects, select your favorite (successful) ones which fit best your users’ needs, and ‘copy’ what they have built, and that will be your prototype. Then, you might still have some time and money left to optimize it so it fully matches the requirements of your users. Consider, however, that all grids are different. For example, research grids are mainly about sharing (e.g. sharing resources, knowledge, data), commercial enterprise grids are about cost and revenue (e.g. TCO, ROI, productivity). Therefore, if your community is academic, look for academic use cases, if it’s commercial, look for commercial use cases in your respective business field.

Rule 4: KISS (Keep It Simple and Stupid). It took your users years to get acquainted with their current working environment and tools. Ideally, you won’t change that. Try hard to stick with what they have and how they do things. Plan for an incremental approach and lots of time listening and talking. Social effects dominate in grids. Join forces with the system people to change/modify mainly the lower layers of the architecture. Your users are your customers, they are king. Differentiate between two groups of users: the end users who are designing and developing the products (or the research results) which account for all the earnings of your company (or reputation and therefore funding for your research institute), and the system experts who are eager to support the end users with the best possible services. Therefore, you can only succeed if you demonstrate a handful of clear benefits to these two user groups.

Rule 5: Evolution, not revolution. As the saying goes: “never change a running system”.. We all hate changes in our daily lives, except when we are sure that things will drastically improving. Your users and their applications deeply depend on a reliable infrastructure. So, whenever you have to change especially the user layer, only change it in small steps and in large time cycles. And, start with enhancing existing service models moderately, and test suitable utility models first as pilots. And, very important, part of your business plan has to be an excellent training and communications strategy

Rule 6: Establish a governance structure. Define clear responsibilities and dependencies for specific tasks, duties and people during and after the project. An advisory board should include your representatives of your end-users as well as application and system experts. In case of more complex projects, e.g. consisting of an integration project and several application or community projects, an efficient management board should lead and steer coordination and collaboration among the projects and the working groups. The management board (Steering Committee) should consist of leaders of the sub-projects. Regular face-to-face meetings are very important.

Rule 7: Money, money, money. Don't have unrealistic expectations that grid computing will save you money initially.. In their early stage, grid projects need enough funding to get over the early-adopter phase into a mature state with a rock-solid grid infrastructure such that other user communities can join easily. In research grids, for example, we estimate this funding phase currently to be in the order of 3-5 years, with more funding in the beginning for the grid infrastructure, and later more funding for the application communities. In larger (e.g. global) research grids, funding must cover Teams or Centers of Excellence, for building, managing and operating the grid infrastructure, and for middleware tools, application support, and training. Also, today's funding models in research and education are often project based and thus not ready for a utilitarian approach where resource usage is based on a pay-as-you-go approach. Old funding models first have to be adjusted accordingly before a utility model can be introduced successfully. For example, today’s existing government funding models are often counter-productive when establishing new and efficient forms of utility services (see Rule #2). In the long run, grid computing will save you money through a much more efficient, flexible and productive infrastructure.

Rule 8: Secure some funding for after the end of the project. Continuity especially for maintenance and support are extremely important for the sustainability of your grid infrastructure. Make sure at the beginning of your project that additional funding will be available after the end of the project, to guarantee service and support and continuous improvement and adjustment of the infrastructure.

Rule 9: Try not to grid-enable your applications in the first place. Adjusting your application to changing technologies costs a lot of money and takes a lot of your precious time. Did you macro-assemble, vectorize, multitask, parallelize, or multithread your application yourself in the past? This is not what you or the user should do. Better to use the money to buy (lease, rent, subscribe to) software as a service.or to hire a few consultants who grid-enable your application and/or (even better) help you enable your grid architecture to dynamically cope with the applications and user requirements (instead vice versa). Today, in grids, we are looking more at chunks of independent jobs, (or chunks of transactions). And we let our schedulers and brokers decide how to distribute these chunks onto the best-suited and least-loaded servers in the grid, or let the servers decide themselves to share the chunks with their neighbors automatically whenever they become overloaded.

Role 10: Adopt a 'human' business model. Don't invent new business models. This usually increases the risk for failure. Learn from the business models we have with other service infrastructures: water, gas, telephony, electricity, mass transportation, the Internet, and the World Wide Web. Despite this wide variety of areas, there is only a handful of successful business models: on one end of the spectrum, you pay the total price, and the whole thing is yours. Or you pay only a share of it, but pay the other share on a per usage basis. Or you rent everything, and pay chunks back on a regular basis, like a subscription fee or leasing. Or you pay just for what you use. Sometimes, however, there are ‘hidden’ or secondary applications. For example, electrical power alone doesn’t help. It’s only useful if it generates something, e.g. light, or heat, or cold, etc. And this infrastructure is what creates a whole new industry of new appliances: light bulbs, heaters, refrigerators, etc. Back to grids: providing the right (transparent) infrastructure (services) and the right (simple) business model will most certainly create a new set of services which most probably will improve our quality of life and make a few companies and some creative individuals very rich.

References:

• 2^nd International Workshop on Campus and Community Grids (Draft), Eds: Laura McGinnis, David Wallom, Wolfgang Gentzsch, http://forge.gridforum.org/sf/go/doc14617?nav=1
  
• Grid Initiatives: Lessons Learned and Recommendations, Report, RENCI Website: http://www.renci.org/publications/reports.php
  
• A Sustainable Grid Infrastructure for Europe, Executive Summary of the e-IRG Open Workshop on e-Infrastructures, Heidelberg, Germany. e-IRG Website: http://www.e-irg.org/meetings/2007-DE/workshop.html
  
• D-Grid Website: http://www.d-grid.de/index.php?id=1&L=1
  

Show Article Preview...