November 2006 B
High Productivity Computing Systems and the Path Towards Usable Petascale Computing
D. E. Post, DoD High Performance Computing Modernization Program
R. P. Kendall, Carnegie Mellon University Software Engineering Institute

Development and Production Workflows

The development and production workflow for a typical CSE project is highly iterative and exploratory. Each stage of software development involves many steps that are closely linked. If the steps can be completed successfully, the work proceeds to the next step (Figure 5). For most realistic cases, multiple issues arise at each step and resolution of the issues often requires iteration with prior steps. The detailed architecture of the code evolves as the code is developed and issues are discovered and resolved.

The degree to which each step becomes a formal process depends on the scale of the project. A small project involving only one or two people need not devote a lot of time to each process. Nonetheless, even small projects will go through almost all of the steps defined below. It is thus worthwhile for almost all projects to go through the checklist to ensure that they don’t miss a step, which would be simple to address early in the project, but difficult much later in the project.

Throughout this paper we define stakeholders as everyone who has a stake in the project including the sponsors, the users and customers, the project team, the project and institutional management, the groups who provide the computer and software infrastructure, and sub-contractors. Sub-contractors include everyone who develops and supplies crucial modules and software components for the project, but who are not part of the project team and not under the direct control of the project management.

I. Formulate Questions, Issues and General Approach

The time scale for this phase is generally three months to a year. The first step involves assessing the state of the science and engineering, its potential for solving the problem of interest, and the development of a roadmap and high-level plan for the project. A key element is the assessment of prior and existing methods for solving this problem with an analysis of their strengths and weaknesses. Prior and existing computational tools provide highly useful prototypes for the proposed project; they embody the methods and algorithms that have been successful in the past and demonstrate the strengths and weaknesses of those methods. These help potential sponsors, users, stakeholders and domain experts achieve a common view of the problem. For the science community, this phase would result in a proposal for submission to a funding agency (e.g., NSF, DOE SC, etc.). This phase also would provide a document that will be essential for developing a customer base, getting additional support, and communicating the project goals, purpose, and plan to the stakeholders, including prospective project team members.

Figure 5

Figure 5. Comprehensive workflow for a computational science and engineering project, including code
development, production run, and analysis and assessment.

This stage involves knowledge of all of the development tasks cited above, but emphasizes detailed knowledge of software engineering and software project management, and computational algorithms and libraries. However, a note of caution is appropriate. Extensive use of software tools for project management is premature and can be a serious distraction. Similarly, extensive assessment of algorithms and methods is also premature. A high-level plan and general code architecture is needed before detailed work begins.

Pages: 1 2 3 4 5 6 7 8

Reference this article
"Large-Scale Computational Scientific and Engineering Project Development and Production Workflows," CTWatch Quarterly, Volume 2, Number 4B, November 2006 B. http://www.ctwatch.org/quarterly/articles/2006/11/large-scale-computational-scientific-and-engineering-project-development-and-production-workflows/

Any opinions expressed on this site belong to their respective authors and are not necessarily shared by the sponsoring institutions or the National Science Foundation (NSF).

Any trademarks or trade names, registered or otherwise, that appear on this site are the property of their respective owners and, unless noted, do not represent endorsement by the editors, publishers, sponsoring institutions, the National Science Foundation, or any other member of the CTWatch team.

No guarantee is granted by CTWatch that information appearing in articles published by the Quarterly or appearing in the Blog is complete or accurate. Information on this site is not intended for commercial purposes.