November 2006 B
High Productivity Computing Systems and the Path Towards Usable Petascale Computing
D. E. Post, DoD High Performance Computing Modernization Program
R. P. Kendall, Carnegie Mellon University Software Engineering Institute

Software Development Tool Categories

The code development and production phases of software projects involve many different types of activities. In general each requires different tools and methods. We were able to define four broad categories of tools (and methods) that are typically required in different phases of the scientific software lifecycle. The potential suppliers for these tools and methods include platform vendors, commercial third party vendors, academic institutions, and open source developers.

1. Code Development computing environment:

This includes the computer operating system (e.g., Linux, AIX, True64, etc.), text editors, interactive development environments (e.g., Eclipse), languages and compilers (Fortran, C, C++, JAVA, etc.) including language enhancements parallel computers (Co-array Fortran, UPC, HPF, OpenMP, Pthreads, etc.), parallel communication libraries (e.g., MPI), symbolic mathematics and engineering packages with a high level of abstraction (Mathematica, Maple, Matlab, etc.), interpretative and compiled scripting languages (PERL, Python, etc.), debuggers (e.g., Totalview), syntax checkers, static and dynamic analysis tools, parallel file systems, linkers and build tools (e.g., MAKE), job schedulers (e.g., LFS), job monitoring tools, performance analysis tools (e.g. Vampir, Tau, Open Speedshop, etc.). This software can either be supplied by the platform vendor or by third parties. For instance, AIX is supplied by IBM. Etnus markets the debugger Totalview.

2. Production Run computing environment:

This includes running the code and collecting and analyzing the results. Many of the tools for the code development environment are required (operating system, job scheduler, etc.). In addition there are specific tasks that involve problem setup (e.g., mesh generation, decomposing the problem domain for parallel runs, etc.), checkpoint restart capability, recovery from faults and component failures (fault tolerance), monitoring the progress of a run, storing the results of the run, and analyzing the results (visualization, data analysis, etc.). Some of this software is supplied by the platform vendor and some by third parties. CEI, for instance, markets Ensight, a massively parallel 3D Visualization tool. Research Systems markets IDL, a data analysis tool. A key task is verification and validation which requires tools for comparing code results with test problem results, experimental data and results from other codes.

3. Software engineering and software project management tools:

These tasks involve organizing, managing and monitoring the code development process. Tools that would help with this task include configuration management tools (e.g., CVS, Perforce, Razor, etc.), code design and code architecture (e.g., UML) although there are few examples of code design tools being used for HPC applications, documentation tools (word processors, web page design and development tools, etc.), software quality assurance tools, project design tools, and project management tools (Microsoft Project, Primavera, etc.). Most of these are supplied by commercial third party vendors. Development of code development collaboration tools for multi-institutional code development teams will also be important in the future (probably a third party task).

4. Computational algorithms and libraries:

These tasks involve development and support of computational algorithms and libraries that are incorporated into working code. These include computational mathematics libraries (e.g., PETSc, NAG, HYPRE, and Trilinos.), physical data libraries, low-level memory management libraries (e.g., MPI), etc. These are supplied by computer platform vendors, commercial vendors, academic and national laboratory institutions, and the open source community.

For tasks that call for selection of an approach or method, the expectation is that the vendor will provide options and some guidance (documentation and consultation) on which approach or method is most appropriate for a set of specific requirements. In general a formal tool for making the selection is not required.

In the discussion that follows, the categories of software tools defined above will be listed under each major workflow stage.

Pages: 1 2 3 4 5 6 7 8

Reference this article
"Large-Scale Computational Scientific and Engineering Project Development and Production Workflows," CTWatch Quarterly, Volume 2, Number 4B, November 2006 B. http://www.ctwatch.org/quarterly/articles/2006/11/large-scale-computational-scientific-and-engineering-project-development-and-production-workflows/

Any opinions expressed on this site belong to their respective authors and are not necessarily shared by the sponsoring institutions or the National Science Foundation (NSF).

Any trademarks or trade names, registered or otherwise, that appear on this site are the property of their respective owners and, unless noted, do not represent endorsement by the editors, publishers, sponsoring institutions, the National Science Foundation, or any other member of the CTWatch team.

No guarantee is granted by CTWatch that information appearing in articles published by the Quarterly or appearing in the Blog is complete or accurate. Information on this site is not intended for commercial purposes.