Esperienze di sistemi meteorologici numerici in configurazione di servizio basati su calcolo ad alte prestazioni in APAT Franco Valentinotti Quadrics,

Presentazione sul tema: "Esperienze di sistemi meteorologici numerici in configurazione di servizio basati su calcolo ad alte prestazioni in APAT Franco Valentinotti Quadrics,"— Transcript della presentazione:

1 Esperienze di sistemi meteorologici numerici in configurazione di servizio basati su calcolo ad alte prestazioni in APAT Franco Valentinotti Quadrics, Ltd. Attilio Colagrossi APAT CAPI04 Milano, 24-25 Novembre 2004

2 Indice q APAT e servizi meteo: storia e stato attuale q Il dualismo servizio-ricerca: criteri ed architetture q I sistemi di calcolo - La catena operativa e la complessità dei modelli - Il modello meteorologico BOLAM - QBolam e il sistema basato su APE100 - PBolam e il sistema basato su ALTIX350 - Esperienze su altri sistemi di calcolo q Conclusioni

3 APAT e servizi meteo: storia e stato attuale APAT Agenzia per la Protezione dellAmbiente e i Servizi Tecnici Istituita nel 2002 Svolge attività tecnico-scientifiche di interesse nazionale per la protezione dellambiente, dellacqua e del suolo. Incorpora le competenze precedentemente attribuite allANPA ed al Dipartimento dei Servizi Tecnici Nazionali – Servizio Idrografico e Mareografico Nazionale, Servizio Geologico Nazionale, Biblioteca

4 APAT e servizi meteo: storia e stato attuale 1998: il Dipartimento dei Servizi Tecnici Nazionali avvia il Progetto Idro-Meteo-Mare in collaborazione con ISAC-CNR e ENEA OBIETTIVI Analisi e previsione della situazione meteorologica sul territorio e dello stato del mare Mediterraneo Monitoraggio in tempo reale, produzione di analisi e previsione dei campi di interesse, valutazione dei fenomeni idrometeorologici e dei rischi associati

5 APAT e servizi meteo: storia e stato attuale MODELLI UTILIZZATI BOLAM (inizializzato sulle analisi ECMWF) WAM POM FEM

6 APAT e servizi meteo: storia e stato attuale Requisito fondamentale: Esecuzione dei modelli in CONFIGURAZIONE DI SERVIZIO ECMWFBOLAM WAM POM FEM …… dal 2001

7 APAT e servizi meteo: storia e stato attuale Ambiente di calcolo basato su computer ad alte prestazioni: inizialmente….. APE 100 ora…………. ALTIX 350

8 Il dualismo servizio-ricerca: criteri ed architetture router alpha server 4100 APE100 ECMWF Sun spark station Unità di storage alpha Internet LAN di palazzo ADSL Venezia Servizio Laguna Veneta

9 Il dualismo servizio-ricerca: criteri ed architetture ora…. script di basso livello, file system, elaborazioni…ad hoc tra poco….. Open technologies: Linux, Apache, MySQL, PHP, Java

10 The operational chain: the models meteorological The 3D meteorological model BOLAM running at two different resolutions: High Resolution:30 km grid spacing Very High Resolution:10 km grid spacing ocean 3 ocean models: WAM: a 2D model for the prediction of amplitude, frequency and direction of the sea waves ; POM: a shallow-water circulation model for the prediction of surface elevation and horizontal velocities ; VL-FEM: a 2D high res. circulation model using finite elements to better describe the Venice Lagoon morphology.

11 The computational domain POM POM: grid covering the whole Adriatic Sea with about 4000 pts. and a variable resolution, with grid size decreasing when approaching Venice (from 10 to 1 km). VL-FEM VL-FEM: mesh covering the whole Venice Lagoon with more than 7500 elements and a spatial resolution varying from 1 km to 40 m. qH.R. BOLAM qH.R. BOLAM: coarse grid with 160×98×40 pts. and 30 km of resolution. qV.H.R. BOLAM qV.H.R. BOLAM: fine grid with 386×210×40 pts. and 10 km of resolution. qWAM qWAM: grid covering the whole Mediterranean Sea with about 3000 pts. and 30 km of resolution.

12 The BOLAM computational cost 2 days of forecast in ~1 hour The operational requirement V.H.R. BOLAM 10 3 flop / grid pt. / t. step 3·10 6 grid points Time step of 80 s ~ 7 TFlop / 2-days ~ 2GFlops sustained

13 The Meteorological Model BOLAM GENERAL FEATURE A 3D primitive equations (momentum, mass continuity, energy conservation) model (in the hydrostatic limit) Prognostic variables: U, V, T, Q, Ps NUMERICAL SCHEME Finite difference technique in time and space Advection: Forward-Backward Advection Scheme (FBAS), explicit, 2 time-levels, centered in space Diffusion: - horizontal:4th order hyperdiffusion on U, V, T, Q 2nd order divergence damping on U, V - vertical:implicit scheme on U, V, T, Q PHYSICS ROUTINES They only involve computations along the vertical direction.

14 Quadrics QH1 q128 processors q6.4 GFlops q512 MByte Server DEC 4100 Year 1997: the QBolam and APE100 choice General features qSIMDSingle Instruction Multiple Data qTopology3D cubic mesh qModule2 × 2 × 2 processors qScalabilityfrom 8 to 2048 processors qConnections3D first neighbours, periodic at the boundariesProcessor qMADMultiplier & Adder Device Pipeline 50 MFlops of peak qMemory4 MByte per processor (distributed) Master Controller qZ-CPUInteger operation Memory addressing

15 The parallel code QBolam Data Distribution Strategy qStatic Domain Decomposition N. of subdomains = N. of PEs Subdomains of same shape and dimensions qConnection between subdomains using Frame Method Boundary data of the neighbouring sub-domains copied into the frame of local domain qColumn Data Type Structure Ad hoc libraries for communications and arithmetical operations between columns The BOLAM code has been redesigned for the SIMD architecture and rewritten in TAO language

16 QBolam Performance on Quadrics/APE100 machine typeQH1 QH4* N. of processors128 512 QBolam modelHRVHR resolution30 km10 km N. of ops./time step0.57 GFlop2.90 GFlop Time step240 s80 s Execution time/time step0.297 s1.333 s0.392 s Performances1.92 GFlops2.12 GFlops7.21 GFlops % of peak performance30 %33 %28 % days of simulation2,5 days2 days elapsed time8' 16''1h 53' 35''48' 25'' * Misure effettuate su Quadrics/APE100 QH4 del centro di calcolo ENEA Casaccia - Roma

17 goal The goal of the project is to substitute the existing previsional operational system with a new one in the next future: qSimplify the operational chain: all models and interfaces will be executed on one machine only qParallel architecture upgrade: Cluster Linux, Open Source qSimulation model upgrade: the first result is the PBolam code development, a parallel meteorological model. q4 dual CPU node q1.4 GHz ItaniumII q44.8 GFlops of peak q8 GByte of memory (physically distributed) qSMP thanks to the NUMAFlex technology (6.4 GBytes/s) single system image qOpenMP, MPI Year 2004: PBolam and Cluster Linux SGI Altix 350

18 The parallel code PBolam PBolam is a parallel version code for distributed memory architecture of the meteorological model BOLAM. qPortable Fortran90 MPI standard Posix qVersatile any number of processors any number of grid points qEasy to maintain same data type structure as BOLAM same variables/subroutine name as BOLAM General Features qStatic Domain Decomposition Number of subdomain equal to number of processes, but not fixed as QBolam Parallelepiped subdomains, but they may have differents shape and dimension qData Distribution Strategy All vertical levels on the same process Subdivision on the horizontal: N Lon / P Lon N Lat / P Lat N Lev where P = P Lon P Lat (number of processes) are choose to minimize communication time qFrame Method Boundary data of the neighbouring sub- domain copied into the frame of local domain: exchange in North-South / East-West Parallelization strategy

19 VHR PBolam performance on Altix Execution time vs. number of processes/data distribution Execution time for all possible P Lon P Lat = P [2,8] combination was measured Execution time of one step decreases when P increases Communication time is quite constant when P increases Execution time of the physics phase is quite constant, for a fixed P Execution time of one step, for a fixed P, is minimum when also communication time is minimum

20 VHR PBolam performance on Altix Communication time increases slowly when P increases, because total data involved in the exchage increase slowly too Since total execution time is 1/P, communication time increases from 1% to 10% This behaviour is evident also in the speed up curve: S = (time NP =1 ) / (time NP =P ) Execution time vs. P for the best data distribution

21 MPI Latency 1.8 s MPI Bandwidth 900 MB/s Elite4 Switch Elan4 NIC AMD Cluster with QsNet II q 8 dual CPU node q 2.2 GHz Opteron q70.4 GFlops of peak performance q 8 GByte of distributed memory qQsNet II interconnect

22 Altix vs. AMD Cluster Itanium is faster than Opteron (Preliminary results show a 1.5 factor) QsNet II shows better performance Less increase of percentage of communication on the total time means best speed up curve, especially when the number of processes grows

23 Conclusion The VHR execution time has been reduced with Altix 350: PBolam performance (6.3 GFlops, 14% of peak) is 3 time QBolam perf. from 100 min. to 20 min. of elapsed time, including I/O Now APAT has a meteorological parallel code PBolam, portable on several cluster Linux In the next future, all the previsional chain will be simplified because all models and interfaces will be executed on one machine only SW architecture more suitable to perform both research and service activities