Cyberinfrastructure for Multidisciplinary Science in Brazil
Marco A Raupp and Bruno Schulze, National Laboratory for Scientific Computing, LNCC
Michael Stanton and Nelson Simões, National Research and Education Network, RNP
CTWatch Quarterly
February 2006


Long-distance, high-speed and low-cost networking has encouraged the development of applications taking advantage of geographically distributed resources, opening up new research directions that were previously limited or unexplored for economic and practical reasons. The establishment of Cyberinfrastructure allows mature, scalable computing for different application communities.

Grid initiatives in Brazil were initially driven by international collaborations in several application areas and the pursuit of higher network bandwidth and larger computational facilities. In response to this demand, the National Laboratory for Scientific Computing – LNCC headed a proposal formulated together with representatives of application groups, computer and computational science, computer networking, high-performance computing, and federal government funding agencies belonging to the Ministry of Science and Technology (MCT). This early proposal was based on a number of existing international initiatives and focused on improving connectivity and communication performance for the coordinated use of existing regional HPC centers, as well as a number of academic and research institutions as potential users. At this time there were seven regional HPC centers and nine academic and research institutions involved, with connectivity provided by the Brazilian National Research and Education Network – RNP, and funding by the MCT funding agencies. The application areas included high-energy physics, bioinformatics, climate and weather forecasting and oil industry needs, among others.

In 2003, the legal framework was established for a National System for HPC (SINAPAD) with LNCC being designated the national coordinator (on behalf of MCT), and was recognized as part of the Federal Education, Science & Technology infrastructure. SINAPAD consisted of a network of regionally distributed, operational HPC centers aimed at providing computation on demand for education and scientific purposes, with a proposed operational structure based on mid-sized computer systems and clusters organized into a grid for the sharing of resources and reduction of idle time. Clusters of a few hundred processors were planned for each center, combining distributed and shared memory machines, facilities for data storage and handling, user friendly access through a web portal, security, accounting, and the option of several alternative architectures.

HPC and Networking History

Brazil has a territorial extent of about 8.5 million km2, making it the fourth largest country in the world, with correspondingly large demands in order to reach all parts of the country. The federal constitutional order is similar to the USA, with 26 states and a Federal District around the national capital, Brasília. The history of academic HPC in Brazil began in 1990, based on the concept of government-funded regional centers open to universities and research institutes, as in Table 1, with seven such centers being established between 1992 and 1997. INPE formally joined the system only in 2002, contributing with its former production machine formerly dedicated to environmental sciences.

Internet technology had already been brought to Brazil by concerted initiatives by state governments in a number of the more populous states, such as São Paulo, Rio de Janeiro and Rio Grande do Sul, coordinated nationally by the federal government through RNP. The first national IP backbone network in Brazil was launched by RNP in 1992 to serve the national academic and research community, and connected 11 cities 1. Since 1992, the national network run by RNP has evolved through four significant increases in capacity and usually also technology, where the fourth such change was initiated in November, 2005. Table 2 summarizes these changes. It is relevant to discuss here the Project GIGA testbed network (2004-) and the most recent national network upgrade initiated in 2005.

Center Comment
LNCC National Laboratory for Scientific Computing initially IBM SP2, later upgraded with SUN and SGI machines
UFRGS Federal University of Rio Grande do Sul initially Cray XMP and later upgraded to a Cray T90
UNICAMP State University of Campinas initially IBM SP2, later upgraded with SUN machines
UFMG Federal University of Minas Gerais initially IBM SP2, later upgraded with SUN machines
UFRJ Federal University of Rio de Janeiro initially CRAY, recently upgraded with SGI Altix machines
UFC Federal University of Ceara
INPE/CPTEC National Space Research Institute/
Climate and Weather Forecasting Center
initially NEC SX 4, later upgraded to a SX 6

Table 1. SINAPAD centers
Year Technology Link capacities Comment
2005 IP/WDM 2.5 and 10 Gbps IPÊ network core to 10 capitals; metro networks in 27 capitals
2003 IP/SDH 34, 155, 622 Mbps also: IP/WDM interstate gigabit testbed network (GIGA)
1999 IP/ATM, IP/FR VC ‹ 45 Mbps,
access ‹ 155 Mbps
RNP2 national network; testbed metro networks in 14 cities (ATM)
1995 ‹ 2 Mbps also: commercial IP deployed in Brazil
1992 Internet 9.6 and 64 kbps first IP network (RNP)
Table 2. Evolution of academic networking in Brazil

Cyberinfrastructure Networking

With the objective of developing and deploying new optical networking technologies, CPqD 2 and RNP formed a partnership to build an optical testbed network, connecting seven cities in the adjoining states of Rio de Janeiro and São Paulo. This project, named GIGA 3, has been supported financially since December 2002 by the federal government's Fund for the Development of Telecommunications Technology (FUNTTEL) and has resulted in building a 700 km network using dark fiber provided without cost by four telcos and lit up using WDM optical equipment produced by PADTEC, a spin-off of CPqD. This network provides IP service over Gigabit Ethernet to end users in 17 research institutions and four telcos in seven cities (see Figure 1). Most users are R&D groups developing subprojects of Project GIGA, under contract to CPqD or RNP. For RNP, the establishment of this optical networking testbed was the first step in implementing its long-term strategy, known as the National Optical Initiative (ION), which emphasizes the provision of future networking services to end-users through a "facilities-based" infrastructure, such as dark fiber or WDM waves, rather than one based on renting telco-provided services, such as ATM or SDH/Sonet.

The testbed network built by Project GIGA has provided extensive hands-on experience in the design, deployment and operation of both metropolitan and long-distance optical networks, and thus has served as a basis for future development of networks to serve the wider research and higher education community. Within RNP, forward network planning has been heavily influenced by the experience gained in Project GIGA, particularly related to identifying and serving particular user groups, and also as regards the technologies employed. Attention has also been given to new directions in networking adopted in other countries during the last five years, highlighted by such initiatives as National Lambda Rail 4, CAnet4 5 and SURFnet6 6, among others, and widely reviewed in the excellent series of reports produced by the European SERENATE study 7. It was decided that RNP should accompany the global tendency to increase the link capacities of its national network to multiple Gbps. This was in fact undertaken in November, 2005, but was preceded by a migration to SDH/Sonet links starting in 2003, with the abandonment of ATM and Frame Relay (FR) as link technologies. The fact that this first migration could be achieved with an increase in aggregate link bandwidth of around six times (from 350 Mbps in 2003 to over 2 Gbps in 2005) with a 30% reduction in cost is a reflection of a combination of newer technologies and the introduction of competition in the telecommunications marketplace.

In 2005, RNP moved onto the next stage of its ION strategy, replacing the links between the 10 principal cities of its national network by a solution in the form of unprotected, transparent lambdas (waves) of 2.5 and 10 Gbps. The core of what is now known as the IPÊ 8 network (see Figure 2) was commissioned in November. The increase in aggregate link bandwidth for this IPÊ network core is from around 1.6 Gbps (SDH) to 60 Gbps (waves), or almost 40 times, at only three times the cost! In fact, the overall cost of the whole national network is now just 30% more than in 2003, for an increase in aggregate bandwidth of almost 180 times. As an essential complement to the deployment of the IPÊ network, RNP is also engaged in a nationwide project, known as Community Networks for Education and Research (Redecomep), to deploy optical metropolitan area networks in all 27 capital cities by December 2006. These networks are designed to provide gigabit access to the RNP point of presence (PoP), as well as interconnecting all the campi of RNP clients in these metro areas 1. International connectivity for RNP's IPÊ network is based on links to commodity networks and also research and education connections (RE) directly connecting to other RE networks (REN), such as CLARA (the regional Latin American REN), Abilene (US NREN), CalREN (California REN), Géant (European REN), among others. A summary of these is shown in Table 3.

Figure 1

Figure 1. Location of the Project GIGA testbed network (2004)

Figure 2

Figure 2. IPË network in 4Q2005
Type Comment
Commodity 310 Mbps from Rio de Janeiro and 45 Mbps from S. Paulo via Global Crossing
RE 155 Mbps (with possible temporary extensions to up to 622 Mbps) from São Paulo via CLARA network, which connects to most other national RENs in Latin America at up to 155 Mbps, to Géant at 622 Mbps through a São Paulo-Madrid link, and to CalREN at 1 Gbps through a Tijuana (México)-San Diego link. The CLARA network was built through the European Union's ALICE project 9.
RE To the USA through a S. Paulo-Miami link with 1.2 Gbps bandwidth, shared with the São Paulo state network (ANSP), as from January 2006 (This link, like the Tijuana-San Diego link, are provided through the WHREN/LILA project 10 of NSF's International Research Networks Connections (IRNC) program).
Table 3. International connections available to RNP's IPÊ network

Activities in Cyberenvironments

At LNCC, early grid-related research and development activities started within Project ComCiDis (Distributed Scientific Computing) in 2002, with the purpose of providing guidelines and directions to the SINAPAD initiative, as well as to establish and strengthen national and international collaborations.

The research program of Project GIGA coordinated by RNP includes the thematic area of Large-Scale Distributed Applications, initially with twelve supported subprojects, allowing for the development and maturing of such technology between high-speed interconnected partners, targeting grid middleware and also some applications. The testbed traffic is restricted to supported subprojects. We focus on two of these subprojects, InteGridade 11 and Sinergia, where NCSA 12 participates as an international collaborator on cyberinfrastructures and cyberenvironments. These two subprojects are actually conducted as a single project, involving five with testbed access, and two more connected via Internet only. The institutions involved are: LNCC - ComCiDis (Distributed Scientific Computing Group), UFF - Institute of Computing, CBPF 13, UNICAMP - Institute of Computing, PUC-Rio 14 – Parallelism Lab, and UFRGS – Institute of Informatics, and UFES 15 - Dept. of Informatics. We also count on RNP and SINAPAD as natural partners for providing connectivity and computational resources, respectively. An additional international partner is Monash University (Melbourne, Australia) for access to Nimrod-G 16 based international testbeds and exploring applications based on Nimrod-O. As business partners, we have IBM participating through its academic software program and a local company (Taho) concentrating on wireless connectivity.

The objectives and goals of this joint project include: scalable computing infrastructure, tools and applications; tools and portals for monitoring, submission and scheduling of applications; development of some application-specific interfaces; development and integration of new services, integration of sensors and wireless resources; and application testbeds. The resulting activities distributed among the partners have been the following: implementation of a Grid with the inclusion of some of the partners' local clusters; implementation of a portal; establishment of policies; a monitoring service; scheduling policies and a scheduling service; a data integration service; aspects related to service orchestration; security comformance monitoring; network management services; automatic application transformation tools; accounting; and the porting of some applications.

The current grid infrastructure for the ComCiDis and InteGridade-Sinergia projects includes the use of Globus 2 17, for interconnections, and of SGE 18, OpenPBS 19, and Condor 20 as schedulers within clusters. The implemented portal is servlet-based and includes a monitoring service based on a LUA 21 script collecting information from Globus-MDS. Regarding some current activities we have been working on the development of a data integration service 22, automatic application transformation developments based on EasyGrid 23 24, scheduling 25, hierarchical submission 26, dynamic adaptation 27, and wireless grids 28, among others. We are migrating to Globus 4 and in terms of security, we have been working with Globus certificates and are now moving to MyProxy 29. OpenCA 30 is being used for certification in the SINAPAD grid.

Together with NCSA, activities include building cyberenvironments and global infrastructure. With LNCC activities include becoming an external international node of NCSA resources and a part of the NCSA Condor pool. This involvement also includes development of grid middleware services, portals for specific applications, scheduling algorithms and service, security issues, support for service-oriented applications, the definition of common services for a certain set of applications and fine-grain authorization.

Applications have been developed to use the cyberinfrastructure facilities (at LNCC and also at some of the partners) for use in bioinformatics, haemodynamics of the human cardiovascular system, oceanography, climate studies and geoprocessing, for which reference groups have been set up at LNCC. Other application areas include molecular dynamics (in physics), astronomy, engineering with developments for the oil industry, and environmental modelling of the Amazon region hydrologic basins, with some spin-offs such as e-knowledge and e-government.

In SINAPAD, the goals are to increase the number of regional centers from seven to ten, in order to cover the whole country, and to deploy a total computational and storage resources of 4 Tflops and 30 TB, respectively, in 2006, with respective increases to 5 Tflops and 50 TB in 2007, targeting applications demanding large data set storage and management. The better use of machines also depends on software, including new programming paradigms, user interfaces, and distributed databases - therefore, these are also areas for research and development and of investments.

In terms of networking, through the deployment of the IPÊ network in 2005 and the ongoing deployment of optical metro networks in capital cities expected to be completed by the end of 2006, RNP is bringing about a significant change in the quantity and quality of communications resources at the disposal of the Brazilian research and higher education community, permitting the widespread use of advanced applications. Future efforts will be directed towards extending these facilities more widely, bringing multiple gigabit connectivity to the remaining 17 state capitals and also to population centers outside the metropolitan districts of the national and state capitals.

Author Affiliation Links

National Laboratory for Scientific Computing, ­ LNCC -
National Research and Education Network, ­ RNP -

1Stanton, M.A., Ribeiro Filho, J.L., Simões da Silva, N. "Building Optical Networks for the Higher Education and Research Community in Brazil", COMNETS2005, Oct. 2005.
2 R&D center of the then former telecommunications monopoly -
3 Scarabucci, R.R., Stanton, M.A. et al., Project GIGA – High-speed Experimental Network. TRIDENTCOM'05, p. 242-251, Feb. 2005.
4 NLR, "About National LambdaRail" -
5 CANARIE, "About CA*net4" -
6 SURFnet, "Network" -
7 Study into the evolution of European Research and Education Networking -
8 Ipê is Brazil's national flower, and pronounced in Portuguese as the abbreviation IP (Internet Protocol).
9 ALICE (Latin America Interconnected with Europe -
11 InteGridade –
12 National Center for Supercomputing Applications – University of Illinois Urbana-Champaign
13 Brazilian Center for Physics Research –
14 Pontifical Catholic University of Rio de Janeiro –
15 Federal University of Espirito Santo –
16 The Nimrod Project –
17 Globus Project –
18 Sun Grid Engine Project –
19 OpenPBS Portable Batch System -
20 Condor Project –
21 The Programming Language Lua –
22 Porto, F., Schulze, B. et al., An adaptive distributed query processing grid service, VLDB-DMG LNCS 3836, Springer Verlag 2005.
23 EasyGrid Project –
24 Lima, B.A. et al., Efficient hierarchical self-scheduling for MPI applications executing in computational Grids, MGC 2004, ACM Press, p. 41-46, Oct. 2004
25 Boeres, C. et al., Efficient hierarchical self-scheduling for MPI applications executing in computational Grids, MGC 2005, ACM Press, Nov. 2005, Nov. 2005.
26 Vargas, P. et al., Hierarchical submission in a grid environment, MGC 2005, ACM Press, Nov. 2005.
27 Milanes, A., Rodriguez, N., Schulze, B., Managing jobs with an interpreted language for dynamic adaptation, MGC 2005, ACM Press, Nov. 2005.
28 Lima, L. et al., Peer-to-Peer Resource Discovery in Mobile Grids, MGC 2005, ACM Press, Nov. 2005.
29 MyProxy Credential Management Service –
30 OpenCA Labs –

URL to article: