Il progetto EGI-InSPIRE Luciano Gaido INFN-Torino II Workshop congiunto CCR-INFNGRID Acireale, maggio 2010

Ringraziamenti Alcune slide di questa presentazione sono prese dalle presentazioni fatte al quinto EGEE User Forum (Uppsala, aprile 2010) da: –Steven Newhouse (Project Director di EGI-InSPIRE e direttore di EGI.eu) –Tiziana Ferrari (Chief Operations Officer di EGI.eu) –Michael Gronager (direttore di NDGF) Grazie! 2

European e-Infrastructure European Data Grid (EDG) –Explore concepts in a testbed Enabling Grid for E-sciencE (EGEE) –Moving from prototype to production European Grid Infrastructure (EGI) –Routine usage of a sustainable e-infrastructure 3

The EGI-InSPIRE Project Integrated Sustainable Pan-European Infrastructure for Researchers in Europe A 4 year project with €25M EC contribution –Project cost €69M –Total Effort ~€330M –Staff ~ 170FTE Project Partners (48) EGI.eu, 37 NGIs, 2 EIROs, 8 AP Funded Un-Funded 4

Scalable Community Interactions USERS VOs Virtual Research Community User Community Board USERS VOs Virtual Research Commmuity USERS NGINGI NGI Helpdesk E G I. e u Training Events Trainers Apps. DB EGI Helpdesk VRC Helpdesk Other Helpdesk ESFRI Project VOs Virtual Research Community 5

6 Objectives from INFRA : Distributed computing infrastructure (DCI) Continued operation and expansion of todays’ production infrastructure “Set up an organisation that will enable the sustainable provision of grid services to the European scientific community”  –EGI global operations services –NGI/EIRO operations services “The EGI should ensure a seamless and progressive transition in service provision from the current arrangements to a new scheme that is more sustainable organisationally and financially”  –Extension of accounting to support different business models “The EGI should promote close collaboration and interoperability with similar infrastructures in other parts of the world.”  –Extension of operational procedures and policies –Enhancement of monitoring and accounting tools

7 Struttura del progetto NA3  30.5 FTE/anno SA1  105 FTE/anno (erano in EGEE-III) SA2  10.1 FTE/anno SA3  20.2 FTE/anno (ma solo per 3 anni) JRA1  6.8 FTE/anno

8 Da EGEE-III ad EGI-InSPIRE EGEE-III – Middleware engineering by JRA1 – Integration, Testing and certification by SA3 EGI-InSPIRE – Middleware engineering by external providers (EP) – Integration, Testing and certification by a collaboration between EP and EGI SA2 – Operational Tools by JRA1

9

10 MU = Middleware Unit DMSU = Deployed Middleware Support Unit EA = Early Adopter OU = Operations Unit

Supporto di I livello: responsabilità di SA1 attività: assistenza al deployment e alla configurazione Supporto di II livello: responsabilità di SA2 (TSA2.5: Deployed Middleware Support Unit - DMSU) attività: fornire supporto per il tuning dei componenti o patch per problemi minori Supporto di III livello: responsabilità degli External Provider attività: effettuare il debug e risolvere i bug noti, tenendo traccia dei problemi mediante GGUS Il supporto al middleware

12 Workflow del supporto

13 Interfacce verso i Software Provider (SP) Technology Coordination Board – Forum for strategic technology guidance with members representing: EGI.eu technical team Prime External SPs Users aka supported Virtual Research Communities Resource Providers Service Level Agreements Negotiated and Monitored in the Middleware Coordination Board (MCB) 2nd to 3rd level support interface Day to day interface for support issues Second level support, the DMSU, provides a flexible interface between independent projects

14 Le Networking Activities (NA)

15 Le NA in Italia Effort totale per la JRU Italiana  NA2: 20 PM Attività: dissemination, user forum representation, events  NA3: 40 PM Attività: technical coordination for user communities (application DB,site validation, application porting), coordination of support for user communities Nota: il modello di confinanziamento è piuttosto complesso perché la percentuale di finanziamento a carico della EC dipende, oltre che dal tipo di attività (ad es. NA, SA, JRA) come era in EGEE-III, anche dalla categoria dei task (ad es. task globali, task generali, task internazionali)

16 Operations Scopo principale: continuare la gestione, mantenimento e supporto dell’infrastruttura Grid, sostendendo l’espansione dell’infrastruttura Si articola in due attività distinte: SA1: Operations Attori principali: le NGI (National Grid Infrastructures) e una struttura di coordinamento centrale: EGI.eu JRA1: Operational tools

17 Activities SA1: Operations NGI International Tasks EGI Global Tasks JRA1: Operational Tools Maintenance of the currently deployed operational tools. Development of the operational toolset for further automation and national deployment scenarios. Regionalization of tools Support of new service charging models once defined

18 SA1: Operations Objectives Operation of a secure, reliable European-wide production grid infrastructure federated from national grid initiatives and EIROs (European International Research Organizations), that is integrated and interoperates with other grids worldwide. It will: Provide users with a secure infrastructure through the establishment of the operational security teams Validate new releases of the middleware and operational tools through a coordinated staged roll-out to sites Establish the monitoring services needed to manage the production grid infrastructure Provide a central accounting infrastructure Operate the EGI Helpdesk, integrated with national instances, to provide support to users and Grid operators

SA1: Operations EGEE-III: le operations erano gestite da 10 Regional Operations Centres (ROC) in un contesto “federativo” a parte qualche eccezione (tra cui l’Italia) un ROC raggruppava più nazioni (infrastrutture) EGI: Una infrastruttura per nazione (NGI) Un Operations Centre per NGI (salvo eccezioni) SA1 per la JRU italiana: 424 PM Attività: le stesse fatte dal ROC italiano in EGEE-III + supporto di rete

20 JRA1: Operational tools –ongoing maintenance and further development of the deployed operational tools –development of the operational tools to support a national deployment model (tool regionalisation) –Accounting for the use of different resources within the production infrastructure –Providing an integrated operations portal for the staff running the production infrastructure Objectives: evolution of the operational tools used by the production infrastructure, including:

21 JRA1 Tasks Partners –CERN, CNRS (France), CSIC (Spain), GRNET (Greece), INFN (Italy), KIT (Germany), SRCE (Croatia), STFC (UK) Tools –Operations portal, helpdesk (GGUS), configuration repository (GOCDB), accounting repository and portal, Nagios-based monitoring infrastructure Effort JRA1 per la JRU italiana: 50 PM Attività: Activity leader e sviluppo accounting

22 Relationship with EGI management boards Operations Management Board (OMB) for the policies needed to provide a reliable transparent infrastructure composed of multiple national infrastructure providers.

23 Internal groups and boards Security Policy Group (SPG) development and maintenance of security policies that define the expected behaviour of sites and users to ensure a secure distributed computing infrastructure (NGIs, sites, application communities) Software Security Group (SSG) has representatives from the software providers contributing software to EGI in order to ensure a common coherent approach to the security frameworks. Security Coordination Group (SCG) coordination between the NGI operational security groups, the software security groups, the security policy activities both inside EGI and in other infrastructures and projects User Services Advisory Group (USAG) has representatives from the user communities to feedback to the EGI.eu on the user facing operations tools and support processes. Operational Tools Advisory Group (OTAG) has representatives from the NGI/EIRO Operations Centres and provides feedback on the operational tools and how they need to be adapted in response to EGI‘s requirements.

24 Attività SA2 SA2: Provisioning the Software Infrastructure To ensure a high quality middleware distribution composed from the best middleware providers To collect and coordinate the users requirements and feed it back to the software providers To support users and sites in the deployment of services Effort SA2 per la JRU italiana: 44 PM Attività: Supporto specialistico per alcune componenti middleware all’interno della Deployed Middleware Support Unit (DMSU)

25 Attività SA3 SA3: Services for Heavy User Communities (HUC) –High Energy Physics (HEP) –Life Sciences (LS) –Astronomy and Astrophysics (A&A) –Computational Chemistry and Materials Sciences and Technologies (CCMST) –Earth Sciences (ES) –Fusion (F) Effort SA3 per la JRU italiana: 125 PM Attività: Supporto per vari servizi/comunità: GreiC (26 PM), CCMST (9 PM), HEP (60 PM), VISIVO (30)

26 Attività SA3 Nel progetto originario l’attività SA3 doveva essere prevalentemente di coordinamento/raccordo delle attività svolte in altri progetti, i cosiddetti Specialized Support Centres, SSC, (cfr ROSCOE, SAFE) Poiché gli SSC non sono stati approvati, è necessaria una ridefinizione di obiettivi ed attività di SA3, per individuare attività che siano sostenibili con l’effort interno (ma anche esterno) disponibile. Alcune occasioni importanti: –Technical Workshop di EGI a settembre (Amsterdam) –workshop wLCG a luglio –Workshop tematici: a giugno sullo storage e altri più avanti

27 Conclusioni (1/2) Middleware –Software prodotto da provider esterni può essere un passo importante verso la sostenibilità –Necessaria sinergia (e il rispetto di precisi SLA) –Necessario un buon bilanciamento tra competenze interne (DMSU) ed esterne Operations –Non si parte da zero ma dalle solide strutture di EGEE-III (ROC, tools, procedure) –La transizione da EGEE-III ad EGI-InSPIRE non sarà banale, soprattutto in alcuni paesi Servizi per gli utenti –Riorganizzazione in corso

28 Conclusioni (2/2) Le sfide: –Estensione ulteriore del ‘servizio’ con conseguenti nuove sfide per l’integrazione di nuove NGI e nuove Organizzazioni di Ricerca a livello europeo (EIRO): Nuovi requirement Revisione e snellimento delle procedure esistenti per assicurare la scalabilità Definizione di Operational Level Agreement flessibili –Integrazione di NGIs che usano middleware diversi Necessaria interoperabilità (aiuto dal progetto EMI) –Integrazione di nuove infrastrutture di calcolo distribuite (DCI) Estensione delle procedure e tool esistenti e dei framework di monitoring e accounting