Il progetto EGI-Engage: un percorso verso la cloud Workshop CCR, 26 maggio 2015 Luciano Gaido

Credits Molte slide in questa presentazione sono state prese (e in parte rielaborate) dalle presentazioni della conferenza EGI che si è tenuta la scorsa settimana a Lisbona. E’ quindi doveroso ringraziare: Tiziana Ferrari, EGI.eu Enol Fernandez, EGI.eu/CSIC Gergely Sipos, EGI.eu/SZTAKI Diego Scardaci, EGI.eu/INFN Sergio Andreozzi, EGI.eu 2EGI-Engage e la Cloud, Workshop CCR, Frascati

EGI-Engage: stato del progetto EGI-Engage è la prosecuzione di EGI-InSPIRE Coordinatore progetto: EGI.eu INFN partecipa come lead beneficiary di una JRU con INAF e INGV Responsabile nazionale: L. Gaido Project Legal Signatory: A. Staiano (direttore sezione di Torino) Project Office italiano: INFN-Torino Il Grant Agreement è stato firmato da tutti i partecipanti e il Consortium Agreement è in dirittura d’arrivo. Data inizio: 1 marzo NO kick-off meeting  EGI Conference a Lisbona (18-22 maggio) 3EGI-Engage e la Cloud, Workshop CCR, Frascati

Visione d’insieme 1/2 Offer and expand an e-Infrastructure Commons solution Security operations and AAI infrastructure, accounting, monitoring… Discoverability of services through a European federated marketplace Cross e-Infrastructure access policies and business models Cross-border service procurement with e-Infrastructures and Research Infrastructures 4 Ensure the continued coordination of the EGI Community – Strategy, policy and business development, to continue and expand activities including the evolution of the EGI governance Evolve the EGI Solutions, related business models and access policies for the long tail of science, Research Infrastructures and industry (including SMEs) – Bring new solutions to full maturity and tailor the solutions to the need of specific user groups – Engagement with commercial service providers and consumers for the development and adoption of a common service marketplace EGI-Engage e la Cloud, Workshop CCR, Frascati

Visione d’insieme 2/2 Implement the Knowledge Commons Open access to training, education material, coordinated user support effort Open access to scientific codes, and virtual appliances for repeatability of science Evolve the EGI Cloud towards a Open Data Cloud combining IaaS, PaaS, SaaS and open data publication, access and re-use Fostering re-use of open data User-driven innovation through a network of eight Competence Centres (BBMRI, DARIAH, EISCAT-3D, ELIXIR, EPOS, INSTRUCT, LifeWatch and environmental science) The EGI technical platforms will be evolved according to the needs of the user community and will be integrated with the services of other e-Infrastructures in Europe and world wide 5 EGI-Engage e la Cloud, Workshop CCR, Frascati

EGI-Engage: Informazioni generali Durata: 30 mesi Partner coinvolti: 47 (di cui 43 funded) 6 TotaleEGI.euINFN Effort (PM) ,523,8 %948,0% Budget (€) ,3 % ,5 % Budget totale JRU italiana (INFN+INAF+INGV): K€ (9%) EGI-Engage e la Cloud, Workshop CCR, Frascati

Distribuzione effort nel progetto 7 Work Packageattivitàpartner coinvolti% effort (PM) NA1Project management48.0% NA2 Strategy, policy and communication % JRA1-JRA2-SA1 E-infrastructure Commons, Platform for the Data Commons, Operations % SA2 Knowledge Commons (Distributed Competence Centres) % EGI-Engage e la Cloud, Workshop CCR, Frascati

EGI-Engage: Attività INFN Due ruoli di coordinamento generale (EGI), assegnati all’INFN dopo una call pubblica (persone senior scelte in base al CV): Profilo “Operations Officer”: (1 FTE) per Vincenzo Spinoso (30 PM in SA1.1) [Bari] Profilo “Technical Outreach Expert”: (1 FTE) per Diego Scardaci In particolare: 7.5 PM in NA1.2, 1 in NA1.3, 6.5 in JRA1.2 e 15 in JRA2.3 [Catania] Altre attività: DCC Mobrain (SA2.5): 2 PM [Padova] DCC DARIAH (SA2.6): 13.5 PM [Catania] DCC Lifewatch (SA2.7): 3 PM [Bari] JRA2.4 (GPU): 6 PM [Padova*] NA2.2 (Marketplace & Pay4Use): 5 PM [Bari] NA2.3 (SME engagement): 2.5 PM [CNAF] SA1.3 (Platform for long tail): 2 PM [Catania] * Contributo da Milano per modifiche a BLAH 8EGI-Engage e la Cloud, Workshop CCR, Frascati

Open Science Commons 9 E’ uno degli obiettivi di EGI, come del resto di molti altri attori in Europa e nel mondo, tra cui: la EC che sta cercando di spingere verso una armonizzaizone/integrazione dei servizi di base per le e-Infrastructures  vedi progetto AARC* l’e-Infrastructure Reflection Group (e-IRG) ha prodotto vari documenti con raccomandazioni per un approccio ‘commons’ sono in atto precessi di fusione tra attori diversi in Europa (vedi Geant Association = Dante + Terena) possibili (anche se difficili) altre sinergie, ma le roadmap degli ESFRI (e la loro configurazione come ERIC) in genere conducono nella direzione opposta * AARC: Design of the next generation AAIs to support global research and education as result of integrating and enhancing existing services or components as much as possible, of piloting missing components enhancing and combining existing components, to develop new ones to allow for their inter-operability and to address of harmonising key policy issues that are needed to offer a secure and scalable infrastructure EGI-Engage e la Cloud, Workshop CCR, Frascati

Open Science Commons: When implemented… Researchers from all disciplines have easy, integrated and open access to the advanced digital services, scientific instruments, data, knowledge and expertise they need to collaborate and achieve excellence in science, research and innovation. They feel engaged in governing, managing and preserving these resources for everyone’s benefit, with the support of all stakeholders. EGI-Engage e la Cloud, Workshop CCR, Frascati Condivisibile in generale, ma passare all’implementazione può non essere banale

Commoning in EGI Types of shared resources Large-scale computing/storage/cloud IaaS- PaaS-SaaS/data services Applications, tools, science gateways Knowledge, expertise, training Rules Various types of access modes e.g., policy-based, excellence- driven, membership-based Not yet fully harmonized across Member States EGI-Engage e la Cloud, Workshop CCR, Frascati Governance – Mainly service providers at the moment – Evolving to include research infrastructures – Advisory board for user communities Funding – National funding agencies, EC, service providers, user communities 11

Attività Cloud in EGI 12 Iniziate in EGI-InSPIRE: realizzazione di un testbed cloud ibrido (public + community) per verificare l’interoperabilità tra stack cloud diversi (OpenStack, OpenNebula, Synnefo/Okeanos, …) adozione di standard de jure (OCCI, CDMI), standard de facto in futuro? strumenti comuni: Monitoring (Nagios) Accounting (Apel) VM repository (AppDB) Obiettivo in EGI-Engage: cloud computing integrated into the existing production infrastructure EGI-Engage e la Cloud, Workshop CCR, Frascati

FedCloud Infrastructure One year of production Resources 21 providers from 14 NGIs 17 interested in joining from 7 new NGIs Average A/R 82%/83% Usage ~700K VMs ~9M CPU hours wall time EGI-Engage e la Cloud, Workshop CCR, Frascati

Federated Cloud user engagement and support Growing support network 11 out of 13 countries that operate cloud sites 76 use cases from 40 communities 75 users in catch-all VO EGI-Engage e la Cloud, Workshop CCR, Frascati

Joining Federated Cloud Resource Providers Follow the EGI Certification Procedure common for HTC and Cloud Resourceshttps://wiki.egi.eu/wiki/PROC09 Use the installation manuals for your CMFhttps://wiki.egi.eu/wiki/MAN10 Technology Providers Contact the TF mailing list Users Contact

Integration Status CapabilityOpenStackOpenNebulaSynnefo OCCI VM Management Yes Integration with VM image management Yes Contextualization Yes OCCI Block Storage Yes In Progress CDMI Object Storage In ProgressN/AYes EGI AAI Integration Yes Monitoring Yes Accounting records Yes Information System Yes

High Level Tools (PaaS & SaaS) Extend the IaaS capabilities of the EGI cloud ‘Alternatives’ of the OCCI client and API More than OCCI External contributions (  support many other clouds too) New developments expected (e.g. INDIGO-Datacloud) SaaS Identity Federation Catania Science Gateway PaaS for automating deployments Helix Nebula Slipstream Programming framework for auto-parallelisation COMPSs Abstraction on top of various HPC/HTC/cloud VMDIRAC Workflow development and enactment WS-PGRADE VM lifecycle manager Vcycle

Dai Virtual Team (VT) ai Competence Center (DCC) 18 Le Virtual Research Communities (VRC) sono sempre state un attore importante in EGI: in EGI-InSPIRE: la maggior parte delle attività veniva fatta, e finanziata, a livello nazionale (nelle NGI) EGI garantiva il coordinamento a livello europeo, attraverso i cosiddetti NGI International Liason (NIL) alcune attività pilota in Virtual Team specifici in EGI-Engage: continua ad esistere la figura dl NIL (ma non è finanziata dal progetto) durante la preparazione del progetto sono state selezionate 7 proposte (+ 1) presentate dalle comunità di utenti (Distributed Competence Center) fanno parte delle attività ufficiali di EGI-Engage e sono finanziate (2 M€, 39%) scopo: integrare alcuni servizi già esistenti, sviluppati dalle comunità, nell’infrastruttura EGI  tipicamente cloud EGI-Engage e la Cloud, Workshop CCR, Frascati

Distributed Competence Centre (DCC) Promote reuse of solutions of common interest across research communities Evolve the EGI technical services with community requirements and provide a test environment with NGIs/EIROs  co-development Promote the integration of community services Scientific applications Joint training programme Technical user support Implementation in WP6 (Knowledge Commons) 7 Competence Centres for 7 Research Infrastructures (RI) from the ESFRI roadmap 1 Asian Competence Centre for Disaster Mitigation CC = group of Scientific institutes + Software providers + Resource centres 5-7 core partners + Unfunded contributors from any NGI Focused workplan with tasks, effort, owners, deliverables, deadlines

Ambition European Open Knowledge Hub: a coordinated network of competence centres Sustained by multiple stakeholders – e-Infrastructures, RIs, Virtual Research Communities, Data Providers … Offering federated scientific software, applications, tools, knowledge and expertise – Scientific software is open, documented, discoverable, supported – Open source publications + datasets + scientific software (repeatability of science) – Knowledge and expertise from a network of European training and education centres – Different capabilities (HTC, HPC, cloud, open research data, tools, applications, software…) Open Knowledge Hub E- Infrastructure support and training Training and education rom ESFRI RIs and VRE projects Distributed Competence Centres Training: SA2.1 (EGI.eu) Technical Support: SA2.2 (CSIC) DCC: SA2.3 – SA2.10

EGI-Engage support to the DCC EGI-Engage e la Cloud, Workshop CCR, Frascati EGI- Engage BBMRIDARIAHEISCAT-3DELIXIREPOSLifeWatch MoBRAIN/ INSTRUCT Environment (disaster mitigation)

Competence Centres in EGI-Engage ELIXIR SA2.3 BBMRI SA2.4 MoBrain SA2.5 DARIAH SA2.6 LifeWatch SA2.7 EISCAT_3D SA2.8 EPOS SA2.9 Disaster Mitigation SA2.10 M6M24M18M1M30 Training Technical User Support SA2.1 SA2.2 Life sciencesHumanities Environmental sciences Generic Mar’15Aug’15Sep’16Mar’17Sep’17 Prep. phase M12 Mar’16

Competence Centre outcomes EGI-Engage e la Cloud, Workshop CCR, Frascati Federated Cloud High Throughtput Computing Security, access control Data management and federation Gateway systems BBMRIDARIAH EISCAT- 3D ELIXIREPOS LifeWatch MoBRAIN / INSTRUCT Environme nt (disaster mitigation) EGI community European Research Area Science gateways Data portals Scalable applicationsSustainable platformsCommunity clouds Training resources User supportData products GPU systems 23

Competence Centers: ELIXIR e BBMRI ELIXIR (Pan-European RI to build a sustainable infrastructure for biological information supporting life science research) [CSC] CC will select representative use case workflows to develop demonstrators targeting FedCloud develop best practices materials creation of VMs for demonstrators Datasets replication BBMRI (Biobanking and BioMolecular resources RI) [SURFsara] Cloud related objectives: Secure data storage infrastructure biobank interoperability and data discovery allowing data sharing respecting protection level required by owners of data and samples Biobank Workflow support: Big-data processing with BioBankCloud (based on Hadoop) EGI-Engage e la Cloud, Workshop CCR, Frascati

Competence Centers: MoBrain e DARIAH MOBRAIN (Research from Molecule to Brain)[SURFSara] Builds on the experience of WeNMR and NeuGrid4You Cloud related objectives: Bring computing to the data managing data, running jobs with Scipion Offer custom virtual machines to research for specific scenarios Enable GPU processing in the EGI infrastructure  GPUs in Cloud DARIAH (research from the areas of the Arts and Humanities)[SRCE/RBI] Cloud related objectives: DARIAH eScience Gateway on EGI (based on WS-PGRADE/gUSE) Storing and Accessing DARIAH contents on EGI (based on gLibrary) Multi-Source Distributed Real-Time Search and Information Retrieval big data stream and batch processing techniques EGI-Engage e la Cloud, Workshop CCR, Frascati

Competence Centers: Lifewatch e EISCAT_3D LIFEWATCH ( capture and address the requirements of Biodiversity and Ecosystems research communities) [CSIC] Cloud related objectives: Big data & Ecological Observatories  tools for managing and processing the data from observatories & simulations (R support required) Workflows & Virtual Labs  web gateways, COMPSs Citizen Science: receive data by citizens, analyze these data open data access? EISCAT_3D (Next generation incoherent scatter radar system)[EISCAT] Cloud related objectives: Develop the EISCAT portal to access the data archives Basic reanalysis in the portal processing of (big?) data EGI-Engage e la Cloud, Workshop CCR, Frascati

Competence Centers: EPOS e Disaster Mitigation EPOS (Design of the use of grid and cloud for the integrated solid Earth Sciences research as part of the European Plate Observing System) [INFN/INGV] Cloud related objectives: Identify and validate authentication and authorisation services for EPOS Test cloud resources and usage models compute/data resources for applications hosting services Science gateway (no specific platform yet) DISASTER MITIGATION (Customised IT services to support the climate and disaster mitigation researchers)[ASGC] Cloud related objectives: web portals for tsunami wave propagation simulations and for WRF-based weather simulation Application of the simulation portals for scientific scenario in disaster mitigation EGI-Engage e la Cloud, Workshop CCR, Frascati

Il percorso Cloud di EGI 28 Fedcloud di EGI: la partecipazione INFN a questa attività: è iniziata in EGI-InSPIRE (Bari e Catania) la partecipazione dei centri INFN non è finanziata esplicitamente in EGI-Enagage ma è la base per le attività di supporto ai DCC dentro al progetto e fuori (ad es. nelle proposte per la call VRE a cui abbiamo partecipato  unico sopravvissuto: WestLife) altre sedi sono interessate (CNAF, Torino,....) è il luogo (uno dei luoghi?) dove si ‘sperimenta’ l’interoperabilità tra cloud diverse (anche basate su stack differenti) e i servizi comuni di base La sinergia con INDIGO-DataCloud è totalmente definita, non è stata esplicitata, ma è sufficientemente chiara, quella con PRISMA/OCP (tra le altre cose, per le attività di definizione di business model, modelli di provisioning e SME engagement) EGI-Engage e la Cloud, Workshop CCR, Frascati Da capire la relazione con la Corporate Cloud INFN e la Cloud nazionale multidisciplinare

L’INFN nell’universo cloud (1/2) 29 L’INFN è coinvolto in varie attività relative alle cloud, con partner, scopi e ambiti diversi (e anche diverso grado di maturità): Fedcloud di EGI Sperimentazione cloud multiregione Attività per infrastrutture cloud ‘locali’ (es. Cloud padovana, Cloud universitaria torinese, etc.) INDIGO-DataCloud Prisma Open City Platform ReCaS (DataCenter, accessibili via Cloud) European Open Science Cloud (proposal in fase di definizione) Altre proposte in fase di definizione EGI-Engage e la Cloud, Workshop CCR, Frascati

L’INFN nell’universo cloud (2/2) 30 L’INFN partecipa anche a progetti e iniziative scientifiche (ad es. progetti ESFRI) che, anche se non come scopo principale, hanno a che fare con servizi e infrastrutture cloud: stretta collaborazione con varie comunità di utenti il riconoscimento all’INFN di una attitudine multidisciplinare è un asset importante sia a livello nazionale che europeo in particolare per l’evoluzione dell’infrastruttura HTC nazionale (Cloud vs Grid) Opportunità nel Work Programme di H2020 (draft): Data and Distributed Computing e-infrastructures for Open Science (consolidamento)  nuovamente fondi per i core service Prototyping innovative e-infrastructure platforms and services for research and education communities, industry and the citizens at large (innovazione) EGI-Engage e la Cloud, Workshop CCR, Frascati

I prossimi passi? 31 I documenti CCR sulla Cloud: dovrebbero tenere conto delle attività cloud in EGI-Engage e degli altri progetti per sfruttare al meglio le attività e i risultati nei progetti per capitalizzare le attività CCR al di fuori dell’Ente Il prossimo Community Forum di EGI si terrà a Bari dal 10 al 13 novembre. E’ una occasione da non perdere per: rafforzare i nostri legami con altre comunità di utenti presentare i risultati che abbiamo ottenuto e la nostra visione sulla cloud consolidare la nostra posizione a livello nazionale (giornata nazionale il 9 novembre) E’ possibile organizzare altri eventi rilevanti (‘colocated events’), ad es. per INDIGO EGI-Engage e la Cloud, Workshop CCR, Frascati Servono però una strategia e una regia per tutte le attività per e con le altre comunità (non solo scientifiche)