R. Smareglia Workshop della Commissione Calcolo e Reti dell’INFN 27 - 30 Maggio 2014.

Presentazione sul tema: "R. Smareglia Workshop della Commissione Calcolo e Reti dell’INFN 27 - 30 Maggio 2014."— Transcript della presentazione:

1 R. Smareglia Workshop della Commissione Calcolo e Reti dell’INFN 27 - 30 Maggio 2014

2 12 Citta’ 18 Strutture Telescopi Nazionali TNG – LBT - SRT Partecipazione ESO Partecipazioni missioni ESA Progetti ESFRI: SKA – E-ELT - CTA

3 L’astronomia… scienza osservativa L’Astronomia e’ una scienza scienza osservativa non sperimentale …. Ma per “fortuna” c’e’ molto da osservare

4 Esempio: Scoprire i Transienti Una volta scoperti, tutti i transienti sembrano simili, ma possono rappresentare fenomeni fisici notevolmente differenti.

5 L’astronomia lavora su tutte le lunghezze d’onda. 5

6 Infrastructure HW (ex.) GAIA EUCLID CTA SW (ex.) Teoretrical / Numerical Simulation Virtual Observatory Open Access (CRIS)

7 Infrastructure HW (ex.) GAIA EUCLID CTA SW (ex.) Teoretrical / Numerical Simulation Virtual Observatory Open Access (CRIS)

8 GAIA Lancio:19 dicembre 2013 Fine Missione:2018 Estensione:2019 Orbita:Lissajous in L2 Distanza:1.5 X 10 6 Km Coutesy of : R. Morbidelli, M lattanzi

9 A bordo del satellite Strumentazione Telescopi:2 (6 specchi) Piano Focale:106 CCD Totale Pixels: ~ 1 X 10 9 Altri strumenti ausiliari per il monitoraggio del satellite e lo stato della strumentazione di bordo Cosa osserva Stelle:~ 2 X 10 9 Galassie:~ 1 X 10 6 Quasar:~ 5 X 10 5 Esopianeti:~ 1 X 10 4 Migliaia di corpi minori del sistema Solare Mole dei dati trasferiti40 GB/die Rate0.5 MB/sec Raw data~ 100 TB Archivi dati finale > 1 Petabyte Coutesy of : R. Morbidelli, M lattanzi

10 Piattaforma di missione Processing capacity: 14 servers HP DL580 G7 per un totale di 416 cores e 3.5TB RAM. 3 servers sono dedicati esclusivamente al database cluster (Oracle RAC technology). Internet link : 1 Gbps (300 Mbps guarateed) by GARR network Storage attuale: ~432 TB (2 X HP P7400 storage units). Ulteriore H/W costituisce la piattaforma di test e validazione: HP EVA 4400 Storage Array with a 40 TB disk space 11 HP Proliant Server Processing/ Test Database 5 HP DL580 G7 4 HP DL580 G5 1 HP DL380 for service management of test platform SGI Altix 450: 8 CPU Dual Core, 256 GB RAM Coutesy of : R. Morbidelli, M lattanzi

11 Euclid Mission M2 mission in the framework of the ESA Cosmic Vision Programme Euclid mission objective is to map the geometry and understand the nature of the dark Universe (dark energy and dark matter) Actors in the mission: ESA and the Euclid Consortium (institutes from 12 European countries and USA, funded by their own national Space Agencies) Launch : 2020 For more information see : http://sci.esa.int/science-e/www/area/index.cfm?fareaid=102 http://www.euclid-ec.org Coutesy of : F. Pasian

12 Euclid at a Glance Coutesy of : F. Pasian

13 Each OU produces algorithms which are integrated and executed in the national SDCs (Science Data Centres) OUs are transnational An organization based on the decomposition in Organization Units (OU), corresponding to a subset of overall EUCLID Data Processing. OU-VISOU-NIROU-SIROU-EXTOU-MEROU-PHZOU-SPEOU-LE3OU-SHE SOC ESAC MOC ESOC scientific community EA DDS Ground Station Euclid External data (PanStarrs, DES,...) SDC OU-SIM VIS ImagNir ImagNir SpectroExt DataEuclidisationSpectro MeasLevel 3Morpho & ShearPhot Red Sh. Simulation SDC OU coordinator OU Deputy Coordinator VObs EA is built jointly by EC and SOC, and is managed by SOC. «Internal» and «public» EA functions – the latter allows access to a subset of EA data The Ground Segment at a glance SDC The SGS System Team provides support and tools for the whole of the SGS (SOC + EC-SGS) EC-SGS Project Office SDC ESA/SOC and the EC SGS have developed, and are committed to maintain, a tight collaboration in order to design and develop a single, truly integrated SGS. Coutesy of : F. Pasian

14 Key Architecture Concepts  Constraints:  > 25 PB of data (Euclid + external surveys needed for multi-color photometry), public archive ~ 13 PB  unlimited volume of simulations  catalogue: 10 10 objects (~ 1500 parameters each)  subtle effects to be captured  reprocessing cycles  «data reduction» = data expansion  No single dedicated SDC – distributed data and processing  Each SDC is both a processing and a storage «node»  Run the pipeline where the main input data is stored  Each SDC runs the same code through virtualisation  Separation of metadata from data  Kind of Map/Reduce  full lower-level processing on sky patches, up to the preparation of catalogues of objects  Higher level of processing based on data cross-matching  Robust, Reliable, Scalable, Maintainable Coutesy of : F. Pasian

15 The CTA project is an initiative to build the next generation ground-based very high energy gamma-ray instrument. It will serve as an open observatory to a wide astrophysics community and will provide a deep insight into the non-thermal high-energy universe Coutesy of : G. LaManna, C.Vuerli

16 Computing Requirements We need to process all events caught from the telescope arrays (showers) being able to discriminate, identify and further process only those triggered by GAMMA cosmic rays The CTA observatory is expected to produce a main data stream for permanent storage of the order of 1 to 10 GB/s for about 1000 hours of observation per year, thus producing a total data volume in the range 1 to 10 PB per year Large CPU time will be needed to process these data, with about 700 CPU days to calibrate and reconstruct 1 hour of raw data. The high data rate of CTA together with the large computing power requirements for Monte Carlo simulations impose dedicated computer resources Coutesy of : G. LaManna, C.Vuerli

17 17 Data Management CTA will be operated as a conventional astronomical observatory A complex archive system has to be built and put in place It shall be able to store all raw data caught from the CTA arrays as well as data produced by subsequent reconstruction/analysis processes Synthetic data coming from MC simulations has to be preserved as well Coutesy of : G. LaManna, C.Vuerli

18 Infrastructure HW (ex.) GAIA EUCLID CTA SW (ex.) Teoretrical / Numerical Simulation Virtual Observatory Open Access (CRIS)

19 Computer Simulation - Cosmology - Galaxy Formation - Galaxy Cluster - IGM ( Intergalactic Medium) - Etc, etc.. - N-body - Smoothed Particle Hydrodynamics ( GADGET code)GADGET - Galaxy Formation analytic model - Etc, etc. - HPC: 159 Million hours of CPU/core BGQ CINECA @ 2013 - Storage required  2,660 TB /y

20 Evolution Homogeneous Observations (~ MB – GB ) Few “simple” objects (~10 1 -10 3 ) Few Parameters Big Homogeneous Survey (~10-100 TB) Huge objects (~10 6 -10 9 ) Dozens of Param. Archivi e Survey of “Federated sky” (~ PB) All Sky Hundreds of Param. Past Next FuturePresent Virtual Observatory NB.: remember  difference between Experiment vs Obvervatory

21 Why VO (Vobs) Change of paradigm done: astronomers use remote distributed data in their everyday work Many more papers from data retrieved from archives than from original observations (IUE - 1995, HST, …) Why and how? – A common data format since the 70s (FITS) (now used by Biblioteca Apostolica Vaticana) – Strong tradition of international collaboration – Open data (in general after a proprietary period) – Driven by community needs (on-line observation archives, on-line services)

22 VO Networking of on-line resources from 1993-4 (added-value services, journals, archives) Seamless access to on-line data (~2000) The astronomical Virtual Observatory The VO framework : standards and data access tools – discover, access, use data Standards defined by the International Virtual Observatory Alliance (IVOA) Procedure inspired from W3C When possible generic elements (OAI-PMH, SKOS/RDF)

23

24 Virtual Observatory The Virtual Observatory (VO) is the vision that astronomical datasets and other resources should work as a seamless whole. The International Virtual Observatory Alliance (IVOA) is an organization that debates and agrees the technical standards that are needed to make the VO possible. It also acts as a focus for VO aspirations, a framework for discussing and sharing VO ideas and technology, and body for promoting and publicizing the VO.

25

26 No central point, a multi-polar world, a global endeavour “Open” and inclusive model A thin interoperability layer on top of the data holdings Anyone can register a data service or build a tool (more than 100 “authorities” with a registered service) The VO is invisible but used because people use the services and the tools! Key for success: seamless access to data AND interoperable tools

27

28 Data is one of the infrastructures of the discipline 2008 2010

29 CRIS - INAF Current Research Information System L’ archivio istituzionale di tutta la produzione scientifica e tecnologica dell’Istituto Nazionale di Astrofisica Nato nel 2009 per fornire informazioni e statistiche sull’attività globale dell’ente, documentare l'attività di ricerca svolta e fornire le referenze bibliografiche delle pubblicazioni prodotte Il sistema è il risultato della cooperazione fra 4 principali attori: Information Technologists (sviluppo software),Bibliotecari/Documentalisti (biblioteconomia e bibliometrica), Amministrativi (gestione dei fondi e del personale), Ricercatori (essenzialmente fornitura di dati e validazione) Creazione, mantenimento e implementazione a cura di un DBManager/Webmaster (Fulvio Naldi fino al 2013) con l’aiuto di un Advisory Committee (AC) composto da 1 o 2 membri per ciascuna struttura dell’INAF) I membri dell’AC sono anche l’interfaccia con le comunità scientifiche locali Coutesy of : A. Gasperini

30 Il sito è organizzato in sezioni : "struttura", "progetti & attività", “incarichi & formazione”, "risultati" e "dati finanziari” E' stato disegnato in modo da minimizzare le duplicazioni e garantire l'integrità dei dati (cautela indispensabile dato il numero elevato di utenti che possono inserire e modificare i propri dati pur esistendo diversi livelli di utenza con funzioni diverse) Aggiornamento periodico dei nuovi dati bibliografici che appaiono su ADS (SAO/NASA Astrophysics Data System) e in cui almeno un autore corrisponde a uno dei nomi dell'elenco del personale INAF. Dati relativi al personale provenienti in gran parte da Astro-Dip, l’Anagrafica dei dipendenti INAF CRIS - INAF Current Research Information System

31 Dati relativi a 2193 persone (al 22 maggio 2014):  68383 pubblicazioni scientifiche  29 brevetti  154 pubblicazioni "edited by"  995 oggetti finalizzati alla divulgazione  2331 altri oggetti ( soprattutto cataloghi di dati astronomici) E’ stato utilizzato dall’INAF come strumento ufficiale per la preparazione del Piano Triennale 2011-2013 Oltre all’implementazione e al potenziamento delle funzionalità del CRIS l’INAF deve andare nella direzione della creazione di un repository istituzionale ad accesso aperto. Sottoscrizione del “Position statement sull’accesso aperto ai risultati della ricerca scientifica in Italia” (4 ottobre 2013) da parte del Presidente INAF

32