Il progetto EGI-Engage e la grid di produzione italiana Riunione CCR, 16 settembre 2015 Luciano Gaido.

Presentazione sul tema: "Il progetto EGI-Engage e la grid di produzione italiana Riunione CCR, 16 settembre 2015 Luciano Gaido."— Transcript della presentazione:

1 Il progetto EGI-Engage e la grid di produzione italiana Riunione CCR, 16 settembre 2015 Luciano Gaido

2 EGI-Engage: stato del progetto EGI-Engage è la prosecuzione di EGI-InSPIRE Coordinatore progetto: EGI.eu INFN partecipa come lead beneficiary di una JRU con INAF e INGV Il Grant Agreement e il Consortium Agreement sono stati firmati Richiesto un amendment perchè un partner tailandese (unfunded) è uscito dal progetto per problemi sulla procedura di arbitrato Attività iniziate il 1 marzo NO kick-off meeting  EGI Conference a Lisbona (18-22 maggio) 2EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

3 EGI-Engage: Informazioni generali Durata: 30 mesi Partner coinvolti: 47 (di cui 43 funded) 3 TotaleEGI.euINFN Effort (PM)1.161276,523,8 %948,0% Budget (€)8.000.0002.827.13635,3 %599.5007,5 % Budget totale JRU italiana (INFN+INAF+INGV): 721.5 K€ (9%) EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

4 Distribuzione effort nel progetto 4 Work Packageattivitàpartner coinvolti% effort (PM) NA1Project management48.0% NA2 Strategy, policy and communication 2017.0% JRA1-JRA2-SA1 E-infrastructure Commons, Platform for the Data Commons, Operations 1235.0% SA2 Knowledge Commons (Distributed Competence Centres) 4739.0% EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

5 EGI-Engage: Attività INFN Due ruoli di coordinamento generale (EGI), assegnati all’INFN dopo una call pubblica (persone senior scelte in base al CV): Profilo “Operations Officer”: (1 FTE) per Vincenzo Spinoso (30 PM in SA1.1) [Bari] Profilo “Technical Outreach Expert”: (1 FTE) per Diego Scardaci In particolare: 7.5 PM in NA1.2, 1 in NA1.3, 6.5 in JRA1.2 e 15 in JRA2.3 [Catania] Altre attività: DCC Mobrain (SA2.5): 2 PM [Padova] DCC DARIAH (SA2.6): 13.5 PM [Catania] DCC Lifewatch (SA2.7): 3 PM [Bari] JRA2.4 (GPU): 6 PM [Padova*] NA2.2 (Marketplace & Pay4Use): 5 PM [Bari] NA2.3 (SME engagement): 2.5 PM [CNAF] SA1.3 (Platform for long tail): 2 PM [Catania] * Contributo da Milano per modifiche a BLAH 5EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

6 Risultati del primo semestre: NA2.1 6 NA2.1 “Communication and Dissemination”: worked on the organisation of the EGI flagship events, the EGI Conference in Lisbon (May 2015) and started preparation for the EGI Community Forum in Bari (9-13 Nov 2015). supported dissemination and outreach with its communication channels such as: the EGI Inspired newsletter case studies and external articles launched a revised EGI Champions programme. EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

7 Risultati del primo semestre: NA2.2 7 NA2.2 “Strategy, Business Development and Exploitation”: exploring cross-­border procurement opportunities, understanding how to move the pay-­for‐use prototype into production creating the conceptual model of the EGI marketplace improving the management of EGI services through better defined processes and procedures according to the FitSM IT service management standard. The Open Science Commons, discussed in Lisbon among policy makers, national research ministry representatives and e-­Infrastructure representatives, has been adopted at a European level in the conclusions of the European Competitiveness Council (May 2015) The implementation plans advanced in the area of the Data Commons: a white paper on how the novel initiative “Open Science Cloud” could be an instrument to implement the data commons in all scientific disciplines has been produced and it will be further advanced with the contribution of user communities. the EGI strategy has been finalized through a broad consultation process started in 2014 An activity to revise and update the EGI service portfolio and solutions has started at PM05. EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

8 Risultati del primo semestre: NA2.3 8 NA2.3 “SME/Industry Engagement and Big Data Value Chain”: spent the first period setting up the business engagement programme for outreach to industry started discussions with organizations to formalize potential collaborations including the Big Data Value Association. an initial market analysis was conducted in selected sections (marine, agriculture) EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

9 Risultati del primo semestre: JRA1.1 9 JRA1.1 “Authentication and Authorization Infrastructure”: has been working on establishing contacts with other projects and infrastructures (AARC, GN4, EUDAT2020 and PRACE) in order to work together towards an interoperable AAI A list of new AAI services including identity and service provider proxy services, have been added to the AAI service roadmap of EGI supported user communities – most importantly the EPOS Research Infrastructure – in defining its AAI infrastructure the analysis of the interoperability of the EGI and EUDAT AAI services has started pilot activities to test a non‐X.509 based AAI model for the EGI Cloud federation continued (https://wiki.egi.eu/wiki/VT_AAI) an operational plan to roll out such services into production will be produced at the beginning of the second part of the first reporting period EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

10 Risultati del primo semestre: JRA1.2, JRA1.3, JRA1.4 e JRA1.5 10 JRA1.2 “Service Registry and Marketplace”: focused on requirements gathering and analysis to prepare a concept for the marketplace and identify interfaces with other EGI tools many surveys and interviews involving different stakeholders have been organized to collect input from service providers and user communities JRA1.3 “Accounting”: some developments around Data, Cloud and Storage accounting collection of requirements for the new Accounting Portal JRA1.4 “Operations tools”: a number of developments to improve existing EGI tools according to EGI needs JRA1.5 “Resource Allocation – e-­GRANT”: continued implementation of improvements needed for EGI Resource Allocation and EGI Pay-­for-­ ‐Use process EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

11 Risultati del primo semestre: JRA2.1 11 JRA2.1 “Federated Open Data”: gathered open data use cases and requirements from different data providers and research disciplines during the specific session of the EGI conference in Lisbon started collecting communities’ requirements to capture plans and need for data management a milestone report (M4.1 "Open Data Platform: requirements and implementation plans“) is being prepared an initial architecture for the open data platform has been designed. a Virtual Team involving EGI technology providers and resource centres was kicked off aiming at the implementation of a distributed testbed for testing of data and storage federation solutions. The current two primary use cases are related to the Human Brain Project and ELIXIR EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

12 Risultati del primo semestre: JRA2.2 12 JRA2.2 “Federated Cloud”: efforts focused in setting up process for tracking and prioritization of the needed developments maintenance of the existing integration tools regarding the support of OCCI support – the VM management open standard of OGF developed the Keystone-­VOMS integration module and new version of the accounting collection tools for OpenStack (https://keystone- voms.readthedocs.org/en/stable-juno) initial contact with some of Product Teams for UMD integration has been established and the integration is now an ongoing process the EGI Cloud federation model was revised and discussed with the aim of accommodating the needs of community clouds only supporting cloud middleware framework native interfaces. EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

13 Risultati del primo semestre: JRA2.3 13 JRA2.3 “e-­Infrastructures Integration”: started integration of D4Science by porting of two selected use cases to the EGI Federated Cloud defined integration plan of the EGI Federated Cloud and the Canadian distributed cloud infrastructure (CANFAR) to realize the astronomy “Data Commons” integration of CANFAR will be a pathfinder to support the LOFAR and SKA research infrastructures defined integration plans between the Australian research cloud (NeCTAR) and EGI. collaboration activities have been identified to support life science, astronomy and astrophysics and brain research including the possibility to share virtual laboratories and community‐specific tools on the cloud EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

14 Risultati del primo semestre: JRA2.4 14 JRA2.4 “Accelerated Computing”: working on enabling GPGPU and co-­processor support for the EGI HTC and Cloud platform an HTC testbed has been set up and a new version of BLAH has been prototyped for including GPU directives to be passed to the underlying LRMS. the main cloud technologies have been reviewed and GPGPU virtualization in KVM/QEMU has been analyzed and performance tests have been carried out a grid site with GPGPU support has been set up in Florence (cegpu.cerm.unifi.it:8443/cream-pbs-batch) a cloud site with GPGPU support has been set up in Slovacchia (keystone3.ui.savba.sk) EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

15 Risultati del primo semestre: SA1.1 15 SA1.1 “Operations Coordination”: focused on provisioning of reliable production infrastructure by improving and modifying several procedures and manuals supporting operations activities work towards sustainable delivery of EGI Core activities through oversight of the performance  preparing new bidding for period of 20 Months starting from May 2016 significant improvement has been also done in terms of UMD Software provisioning by optimization of verification and releasing the middleware distribution process. Task a cui collabora V. Spinoso, verifica a luglio positiva  confermata attribuzione attività e budget a INFN EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

16 Risultati del primo semestre: SA1.2 16 SA1.2 “Development of Security Operations”: has been working on security requirements in particular security requirements on endorsed Virtual Machine images new versions of the “EGI-­CSIRT Critical Vulnerability Handling” and “"EGI Software Vulnerability Group – Strategy and Vulnerability Issue Handling" procedures have been produced and work started on four new or revised policies work towards planning and developing a security challenge of the EGI Federated Cloud services coordination of activities with security teams from other e-­Infrastructures has also started. EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

17 Risultati del primo semestre: SA1.3 17 SA1.3 “Integration, Deployment of Grid and Cloud Platforms”: spent effort on prototyping the long tail of science platform integrated with the Catania Science Gateway Framework work towards defining how to integrate the ESA authentication system with the Federated cloud has been performed by EGI and ESA the activities for the D4Science integration focused on the selection of the use cases and services that will run in the EGI Federated cloud EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

18 Risultati del primo semestre: SA2.1 18 SA2.1 “Training”: worked on the creation of an experts’ network that would collaborate in provisioning and designing training activity provisioning of training events, developing online content in the form of webinars about the EGI HTC, data management and cloud services the task focused on designing and preparing a framework to support trainers and training provisioning EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

19 Risultati del primo semestre: SA2.2 19 SA2.2 “Technical User Support”: focused efforts in setting up process for gathering and tracking user requirements the engagement plan has been periodically revised and updated contacts have been established with two initiatives: the Human Brain Project and the International Cancer Genome Consortium all Competence Centres, with the exception of the BBMRI one – who is starting at PM07 –, delivered significant work towards their objectives. Most of the work has been done around requirements gathering which will be the base for further developments. EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

20 EGI-Engage support to the DCC EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015 EGI- Engage BBMRIDARIAHEISCAT-3DELIXIREPOSLifeWatch MoBRAIN/ INSTRUCT Environment (disaster mitigation)

21 Competence Centres in EGI-Engage ELIXIR SA2.3 BBMRI SA2.4 MoBrain SA2.5 DARIAH SA2.6 LifeWatch SA2.7 EISCAT_3D SA2.8 EPOS SA2.9 Disaster Mitigation SA2.10 M6M24M18M1M30 Training Technical User Support SA2.1 SA2.2 Life sciencesHumanities Environmental sciences Generic Mar’15Aug’15Sep’16Mar’17Sep’17 Prep. phase M12 Mar’16

22 Competence Centers: ELIXIR e BBMRI ELIXIR (Pan-European RI to build a sustainable infrastructure for biological information supporting life science research) [CSC] CC will select representative use case workflows to develop demonstrators targeting FedCloud develop best practices materials creation of VMs for demonstrators Datasets replication BBMRI (Biobanking and BioMolecular resources RI) [SURFsara] Cloud related objectives: Secure data storage infrastructure biobank interoperability and data discovery allowing data sharing respecting protection level required by owners of data and samples Biobank Workflow support: Big-data processing with BioBankCloud (based on Hadoop) EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.201522

23 Competence Centers: MoBrain e DARIAH MOBRAIN (Research from Molecule to Brain)[SURFSara] Builds on the experience of WeNMR and NeuGrid4You Cloud related objectives: Bring computing to the data managing data, running jobs with Scipion Offer custom virtual machines to research for specific scenarios Enable GPU processing in the EGI infrastructure  GPUs in Cloud DARIAH (research from the areas of the Arts and Humanities)[SRCE/RBI] Cloud related objectives: DARIAH eScience Gateway on EGI (based on WS-PGRADE/gUSE) Storing and Accessing DARIAH contents on EGI (based on gLibrary) Multi-Source Distributed Real-Time Search and Information Retrieval big data stream and batch processing techniques EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.201523

24 Competence Centers: Lifewatch e EISCAT_3D LIFEWATCH ( capture and address the requirements of Biodiversity and Ecosystems research communities) [CSIC] Cloud related objectives: Big data & Ecological Observatories  tools for managing and processing the data from observatories & simulations (R support required) Workflows & Virtual Labs  web gateways, COMPSs Citizen Science: receive data by citizens, analyze these data open data access? EISCAT_3D (Next generation incoherent scatter radar system)[EISCAT] Cloud related objectives: Develop the EISCAT portal to access the data archives Basic reanalysis in the portal processing of (big?) data EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.201524

25 Competence Centers: EPOS e Disaster Mitigation EPOS (Design of the use of grid and cloud for the integrated solid Earth Sciences research as part of the European Plate Observing System) [INFN/INGV] Cloud related objectives: Identify and validate authentication and authorisation services for EPOS Test cloud resources and usage models compute/data resources for applications hosting services Science gateway (no specific platform yet) DISASTER MITIGATION (Customised IT services to support the climate and disaster mitigation researchers)[ASGC] Cloud related objectives: web portals for tsunami wave propagation simulations and for WRF-based weather simulation Application of the simulation portals for scientific scenario in disaster mitigation EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.201525

26 La grid di produzione italiana 26 Con la conclusione di EGI-Inspire, e soprattutto la fine di IGI, la struttura di gestione, controllo e supporto della Grid di produzione italiana ha subito un lungo e complesso processo di riorganizzazione. Credo sia doveroso ringraziare tutte le persone che, a vario titolo, hanno contributo alle attività di operations, mantenimento del middleware, user support e formazione. EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015 Vorrei menzionare e ringraziare in modo particolare le persone che hanno contribuito al coordinamento delle attività: -Valerio Venturi -Paolo Veronesi -Daniele Cesini -Emidio Giorgio Un ringraziamento speciale va ad Alessandro Paolini, che ha svolto le attività di operations con notevole cura e dedizione.

27 La grid di produzione italiana: alcune novità 27 Con l’avvio di EGI-Engage e l’impegno INFN in alcuni altri progetti importanti, la struttura di coordinamento delle operations è cambiata in modo significativo. Tuttavia questi cambiamenti sono prevalentemente ‘dietro le quinte’, ma non cambiano sostanzialmente procedure e canali di comunicazione (vedi le prossime slide). La nuova struttura di coordinamento e gestione delle operations a livello italiano, che è in fase di attivazione, sarà più snella rispetto alla precedente e prevede il passaggio del coordinamento a Vania Boccia (sezione di Napoli). Vania sarà anche la persona di riferimento per le operations e per la security in EGI. In queste attività sarà coadiuvata da Sergio Traldi (PD) e Diego Michelotto (CNAF). La nuova organizzazione sarà operativa (sperabilmente) a inizio ottobre e Cristina Aiftimiei sta dando un grande contributo per garantire la continuità e facilitare la transizione (grazie!!!) EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

28 Su più di 90 siti registrati nel GOCDB: 34 siti sono in stato ‘closed’ dei 55 siti di produzione con middleware EMI/UMD 3, 2 sono ‘suspended’ e 2 in fase di ri- certificazione e di 4 si sta valutando la chiusura Attività recenti svolte in ambito operations Censimento dei siti «attivi» Dismissione dei siti con problematiche serie, in accordo con i siteadmin Aggiornamento dei contatti dei SiteOperationManager e dei SecurityOfficer sul GOCDB Ottimizzazione delle mailing lists Attività utili: Revisione/aggiornamento delle documentazioni utili ai siteadmin L’infrastruttura Grid italiana EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.201528

29 Nulla è cambiato, si fa riferimento alla PROC09 EGI raggiungibile dal link: https://wiki.italiangrid.it/twiki/bin/view/GridOversight/Info4RODs#Certificazione_dei_siti https://wiki.italiangrid.it/twiki/bin/view/GridOversight/Info4RODs#Certificazione_dei_siti Prevede i seguenti passi: Registrazione sul GOCDB del nuovo sito (e relative informazioni di contatto) (ad opera dei siteadmin) Controllo delle informazioni pubblicate dal sito Test di funzionalità dei servizi GRID Procedure di certificazione/ri-certificazione EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.201529

30 Operations portal (https://operations-portal.egi.eu): ROD dashboard (Regional Operators on duty) COD dashboard (Central Operators on duty)  il team COD è stato sostituito dal team EGI Operations Support Security dashboard Availability/Reliability monitoring Accounting portal EGI: http://accounting.egi.eu/egi.php?Path=1.41 Accounting portal INFN: https://faust01.to.infn.it/#/dashboard/script/pledge_sum.js Ticketing systems: Italian GRID helpdesk: https://xgus.ggus.eu/ngi_it/index.php?mode=index GGUS: https://gus.eu/index.php L’organizzazione delle operations in EGI è descritta in questa wiki: https://wiki.egi.eu/wiki/EGI_Infrastructure_operations_oversight Tool utili/disponibili EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.201530

31 Mailing list attive: igi-noc@lists.italiangrid.it: è la lista dei National Operation Officers helpdesk@lists.italiangrid.it: è la email dei site supporters grid-sec-contacts@lists.italiangrid.it: è la lista dei security officer dei siti della NGI_IT Link utili: Procedure CSIRT: https://www.italiangrid.it/grid_operations/grid-security/operational-procedures Procedure OPERATIONS: https://wiki.egi.eu/wiki/Operations_Procedures Mailing list attive e link utili EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.201531

32 Oltre le operations… 32 Middleware: l’INFN ha sviluppato e ha mantenuto un certo numero di componenti Grid che sono usate sulla grid europea (wLCG/EGI): VOMS, CREAM, STORM, WMS,... L’INFN si è impegnato a garantire a wLCG il supporto e l’aggiornamento (minimale) di queste componenti finchè sarà necessario  non c’è personale dedicato per questo, il supporto viene garantito dal personale INFN (a t.i. o t.d.) User Support (aka attività verso altre comunità di utenti): È svolto in alcune sezioni sulla base di attività/interessi locali o nell’ambito di alcuni progetti vedi prossima slide EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015

33 I documenti CCR sulla Cloud: dovrebbero tenere conto delle attività cloud in EGI-Engage e degli altri progetti per sfruttare al meglio le attività e i risultati nei progetti per capitalizzare le attività CCR al di fuori dell’Ente Il prossimo Community Forum di EGI si terrà a Bari dal 10 al 13 novembre. E’ una occasione da non perdere per: rafforzare i nostri legami con altre comunità di utenti presentare i risultati che abbiamo ottenuto e la nostra visione sulla cloud consolidare la nostra posizione a livello nazionale (giornata nazionale il 9 novembre) E’ possibile organizzare altri eventi rilevanti (‘colocated events’), ad es. per INDIGO I prossimi passi? 33EGI-Engage e la grid italiana, Riunione CCR, Roma 16.9.2015 Servono però una strategia e una regia per tutte le attività per e con le altre comunità (non solo scientifiche) Dalla mia presentazione al WS CCR di maggio Da fare! In corso, ma la giornata nazionale è ancora da definire Siamo ancora lontani dalla meta