Il MW di Grid da EGEE-II a EGI Evoluzione del Processo del Middleware dallo Sviluppo alla Integrazione, Certificazione e Deployment A.Ghiselli Riunione con i referee di infn-grid 21 aprile 2008

2 Mware - INFN EGEE –Security VOMS/VOMS Admin/VOMS-SAML (OMII-Europe) AuthZ service (like GPbox) –CREAM/CREAM-BES (OMII-Europe) –BLAH –WMS/EIS –DGAS/DGAS-RUS-UR(OMII-Europe) INFN-Grid –StoRM –GridICE –OGF GLUE-schema (OMII-Europe) –GLUEMAN (OMII-Europe) ETICS –ETICS-tools per building, testing

3 MW in EGEE-II processo di pre-certificazione, integrazione e packaging Nuovo TAG –JRA1 consegna il tag CVS per il componente middleware da certificare ad SA3: – in particolare crea una patch in Savannah contenente la lista degli.rpm e relative versioni per il componente da certificare, insieme alle istruzioni per installare e configurare correttamente la componente. Integrazione e packaging –SA3 prepara un repository di certificazione con la lista completa degli.rpm che servono per installare la componente e con le configurazioni YAIM Pre-certificazione –Una volta pronto il repository e yaim, un partner esterno inizia a certificare la componente

4 MW in EGEE-II processo di pre-certificazione, integrazione e packaging PROBLEMI (esempio del WMS) per quanto riguarda il passo 2), sia la preparazione del repository di certificazione che la creazione degli script yaim per configurare correttamente il wms, hanno subito ritardi, spesso dovuti a tempi di attesa di comunicazione tra SA3-CERN e sviluppatori - per quanto riguarda il passo 3), forse la scelta del partner esterno (Imperial College) non e' stata delle migliori, si sono riscontrati tempi molto lunghi anche solo per installare e configurare un WMS su cui eseguire i tests di certificazione. Poi talvolta, dopo i primi tests si scopriva che la macchina utilizzata per la certificazione non aveva i requisiti hardware adatti.  Manca un responsabile del processo della componente MW

5 Dal DoW di EGEE-III attivita’ di SA3 TSA3.1. Integration and packaging The integration of the gLite distribution will be performed by a core team located in one place. This team will be led by a release manager who drives the component release process and ensures that the associated documentation is of acceptable quality and uniformity TSA3.2. Testing and certification Testing and certification are the most important tasks ensuring the released gLite distribution provides the required functionality, performance, scalability, and dependability. We distinguish between testing and certification whereby testing leads to production readiness of a component and will be carried out by SA3 teams that are closely linked with JRA1 teams and certification ensures that after modifications the components still work inside the stack and fulfill the functional and performance requirements, thus it includes regression tests, and will be carried out by the SA3 certification team.

6 TSA3.1: Integration and packaging The effort required for this task is 186 PM, provided by: –CERN 138 PMs: Coordination of the activity, development and tracking of the process for integration and release management, maintenance, evolution and operation of integration tools, configuration tools, repositories, process tracking tools. Integration and packaging of the overall distribution. Interaction with SA1. Collection and maintenance of documentation – INFN 24 PM: Integration and packaging work related to gLite WMS, CE components, VOMS/VOMS –Admin, DGAS, authorisation and prioritisation frameworks. –TCD 4 PM: Integration and packaging of security infrastructure middleware, tools for interoperation –STFC 6 PM: Integration and packaging of service discovery and information system APIs –CESNET 6 PM: Integration and packaging for logging and book keeping components and Job Provenance services

7 TSA3.2: Testing and certification This task will test and certify the gLite middleware stack, develop the necessary test suites, and operate the distributed test and certification testbeds. Apart from standard functional and performance tests, interoperation, security, and vulnerability testing will be included. Pilot services will be set up for large scale tests on the production infrastructure if necessary. The effort required for this task is 319 PM, provided by: –CERN 146 PM: Coordination of the activity, development and tracking of the process for testing and certification, maintenance, evolution and operation of testing tools, virtualized testbeds, operation of a large scale testbed (120+ nodes), coordination and tracking of partners test activities, coordination of testes with SA1 for PPS and pilot services, coordination and participation in patch certification, driving the regression tests. –INFN 48 PM: Testing up to production readiness of the components integrated by INFN. This includes participation in patch verification and test case development.

8 Differenza fra egee-ii ed egee-iii Seguendo il nuovo concetto di "Cluster of Competence" (come descritto nel Description of Work), ogni componente middleware sara' seguita nel ciclo completo dallo sviluppo all'integrazione fino al deployment in certificazione da un team localizzato vicino agli sviluppatori JRA1 della componente. Questo significa che partner SA3 vicino e in stretta collaborazione con gli sviluppatori JRA1 eseguiranno il lavoro di integrazione, packaging e testing fino a che la componente abbia raggiunto un livello accettabile per poter essere usata in produzione. La differenza fondamentale con EGEEII e che dovrebbe ottimizzare il processo e' che tutto il lavoro di integrazione, packaging e testing di pre-certificazione verra' eseguito da un team composto da persone SA3 + JRA1 nello stesso posto in cui la componente viene sviluppata. Questo dovrebbe ridurre/eliminare i ritardi di comunicazione tra team distribuiti come in EGEEII e velocizzare i tempi di rilascio per ogni componente.

9 necessita' di modifica per ottimizzare il processo Perche’ Il man power e’ quasi tutto centralizzato al CERN, quindi il concetto di ‘cluster of competence’ non e’ sufficientemente supportato. RICHIESTA: Per avere una garanzia di ottimizzazione del processo e’ necessario che la certificazione di SA3 in EGEE III sia sotto la supervisione del responsabile di Jra1 del componente certificato che rimane l'unico responsabile della conclusione del processo con successo. Il componente certificato che passa poi a SA3 per la certificazione finale deve aver passato tutti i normali tests di accettanza a un livello di scala tale di poter essere usato cosi' com'e' in EGEE.

Il Middleware in EGI

11 “Market” based –EGI model is focused on the static operation of well consolidated services –Assume that some external provider will: Produce the set of coordinated services needed for the robust and resilient operation of the e-Infrastructure Introduce the required standards for interoperability Introduce the new functionalities required by new applications or EGI operational needs Incorporated development –The MWare development cycle is built in in the EGI model with the shared goal of arriving to the required consolidated interoperable services after a series of development cycles –Unified management of the development cycle has proven for gLite and the other EU stacks to be essential to achieve stable operations of evolving services Scenarios for EGI

12 Main Grid Middleware Stacks Available Today From Europe –gLite universe –UNICORE –ARC –OMII-UK From US: Globus, Condor/VDT From Asia: NAREGI, CrownGrid Europe plays a key role in middleware development

13 The 3 EU Mware stacks ARC (Advanced Resource Connector) –developed by the NorduGrid collaboration since 2001 and supported by EU funds –deployed in more than 60 sites, with over 20,000 CPUs. –adopted by the Nordic DataGrid Facility (NDGF) gLite –developed by EU supported projects (EDG, EGEEs) since 2001 –deployed in 250 sites comprising more than 50,000 CPUs and very large (25 PB (Petabytes)) storage systems. UNICORE (Uniform Interface to Computing Resources) –middleware for the European distributed HPC Grid infrastructure DEISA as well as in the starting PRACE initiative for European PetaFlop/s Supercomputers. EU support several projects… Services of the 3 stacks are often complementary, though not yet fully interoperable Essential commonalities, e.g. the security framework, very similar resource and task description, and some common underlying protocols

14 The Role of EGI for Grid Middleware Evolve current middleware stacks towards: –Achieving standard-based interoperability systems/organizations able to provide/accept services from other systems/organizations and to use the services exchanged to enable them to operate effectively together –Adding new functionalities based on requirements from: supported VOs potentially new VOs operations

15 Tasks for MW TF Specify the set of key service components (incl. interfaces) required for the European Grid infrastructure Describe transition period (as roadmap) from today’s stack-based approach to a candidate implementation of the EGI service components specification Provide cost estimations (contributions, service charges, project grants) for both tasks

16 Remarks from MB to TF Deadline: Monday, 5 May 2008 Relation to Globus and other international MW providers should be taken care of in terms of a collaboration agreement Cloud technology should not be excluded Interaction with Industry through EGEE Industry Forum (Action Bob: Identify person) Identify possibilities for enlargement of user base for general services

17 TF “Middleware” - Members Alistair Dunlop (UK) Achim Streit (DE) Farid Ould-Saada (NO) Mirco Mazzucato (IT) Ludek Matyska (CZ) Ian Bird (CERN) Christoph Witzig (CH) Sergio Andreozzi (IT) Ignacio Martin Llorente (Spain) Uwe Schwiegelshohn (Germany)

18 The gLite Consortium An opportunity for the EU Companies and Research Institutions

19 Why a gLite Consortium Need to preserve and evolve the EGEE middleware in the scenario of the European Grid Initiative (EGI) and National Grid Initiatives EGI will support more stacks and foster convergence: gLite, Unicore, ARC… gLite need to speak with one voice and continue to evolve and support users Globus, Unicore and ARC have already created their Consortia

20 The likely membership The core of the gLite Open Consortium will be constituted by the same institutions that are involved with the gLite developments, i.e.: Elsag- Datamat (IT), CESNET (CZ), INFN (IT), SWITCH (CH), FOM (NL), UH.HIP, CERN, STFC, UNIMAN. The second level will be constituted by the Institution who operate or use the gLite services. Additional industrial or business partners should be included in order to create the conditions for an expansion of the Consortium’s activities towards the commercial markets.

21 Funding model Open Source software development and offering of related maintenance and support services by middleware Consortia has a less straightforward economic sustainability model than service activities ( e.g. network bandwidth offering by the Dante/NRENs organization). The Consortium should aim at extending and generalizing the offer of services to non-skilled communities (new VOs, industries, government, extra-network research institutions, etc) to reach sustainability. This will take time and the continuation of the development efforts to generalize and standardize the offered services It is an absolute requirement for EGI that the current European co- funding will continue after the EGI transition and the funds necessary to maintain and evolve the current stack will be made available through “reserved” projects a’ la Géant subscribed by the Consortium members. Additional R&D activities aiming at service generalization and standardization, again co-funded by the European Commission, will create the base for a general uptake and for an increased sustainability

22. Relationship with EGEE-III and EGI (and how they could work together) The gLite Open Consortium in the initial period should coexist with the EGEE-III JRA1 activities, preparing to act directly as the dedicated entity for the gLite development and support activities at the end of the EGEE III project. The gLite middleware JRA1 activity remains during the initial “EGEE-III phase” a privileged middleware provider, while the gLite Open Consortium will be the legal entity which will progressively maintain coordinated all the gLite development activities ( ) taking them over at the end of EGEE III. In the “EGI phase” (from 2010), the gLite Open Consortium will coordinate all the former gLite JRA1 activities (and relative personnel) and will be the middleware provider in the European arena (where UNICORE and ARC are other providers ). EGI should strongly support a strategy that keeps the European leadership in grid middleware technology by continuing to guarantee leading edge services to the European infrastructure that is supposed to manage.