Stato del progetto LHC Computing Grid Alberto Masoni Commissione Scientifica Nazionale I 24-9-2003 http://www.cern.ch/lcg

STATO DEL PROGETTO LHC COMPUTING GRID RIORGANIZZAZIONE DELLA STRUTTURA MANAGERIALE STATO DI AVANZAMENTO NELLE DIVERSE AREE FABRIC GRID TECNOLOGY IL PROGETTO EGEE APPLICATION DEPLOYMENT DI LCG-1 Cfr discussione ultima Commissione I

RIORGANIZZAZIONE DELLA STRUTTURA MANAGERIALE PROBLEMI EVIDENZIATI NEL CORSO DELL’ LHCC MAN POWER REVIEW DI SETTEMBRE E DISCUSSI NELL’ SC2 12/9/03 REVISIONE DELLA RAPPRESENTATIVITA’ DEGLI ESPERIMENTI NEL PEB E NELL’ SC2 VERRA’ DISCUSSA NEL PROSSIMO SC2 (24/9)

The LHC Computing Grid Project Structure Common Computing RRB Reviews The LHC Computing Grid Project Reports Resource Matters Project Overview Board Other Computing Grid Projects Project Manager Project Execution Board Software and Computing Committee (SC2) EU DataGrid Project Requirements, Monitoring Other HEP Grid Projects RTAG GDB implementation teams Other Labs

FABRIC AREA PREPARAZIONE DEL COMPUTING CENTRE PROCEDE SECONDO LA SCHEDULA TRANSIZIONE A LCG-1 DI LXBATCH NON EFFETTUATA A CAUSA DEL RITARDO NELLA RELEASE DI LCG-1 UN RECUPERO E’ POSSIBILE GRAZIE ALLA POSSIBILITA’ DI UN RAPIDO ‘RAMP UP’ DEL NUMERO DI NODI 440 NODI E 60 DISK SERVERS (PARTE DEL BUDGET 2004) VERRANNO INSTALLATI A NOVEMBRE COSI’ DA ESSERE DISPONIBILI PER I DATA CHALLENGES 2004 GIA’ A GENNAIO 1 GB/S DATA CHALLENGE (04-03) TEST EFFETTUATO GRAZIE ALLA TEMPORANEA DISPONIBILITA’ DI UN NUMERO DI TAPE DRIVES AGGIUNTIVI 920 MB/S PER 3 GIORNI 600 MB/S SU CASTOR (BREVE TEST CAUSA LIMITATA DISPONIBILITA’ HARDWARE)

GRID TECHNOLOGY: Evolution of the Grid Middleware Middleware in LCG-1 ready now for use – initial tests show reasonable reliability scalability (performance) and stability still to be worked on still low functionality. Early experience with the Web Services version of the Globus middleware (Globus Toolkit 3) and experience with the Open Grid Services Architecture (OGSA) and Infrastructure (OGSI) have been promising Good experience this year with packages linking experiment applications to grids – e.g. AliEn, Dirac, Octopus, .. Second round of basic Grid requirements nearing completion (HEPCAL II) Working group on common functionality required for distributed analysis (ARDA) nearing completion The main point is that from AliEn (the most advanced project) and the experience of many other projects, a preliminary architecture could be drawn. ARDA is in the process to select concepts, technologies, approaches that would be the starting point for a possible project on distributed analysis Presentazione stato progetto LCG LHCC Open Session 24/9/03 Les Robertson: L. Bauerdick ARDA report SC2 12-09-03

GRID TECHNOLOGY IL PROGETTO EGEE IL PROGETTO EDG SI CONCLUDE EGEE: Enabling Grids for E-science in Europe e’ previsto partire ad aprile 2004 FINANZIAMENTO: ~32 MEUR

Highlights from first EU negotiation meeting NO SHOW STOPPERS, several points to clarify, basic programme of work agreed Clarify the resources and access policy for user communities to EGEE infrastructure EGEE will start the middleware re-engineering activity by providing an initial set of components based on the selection made by the LCG project at the end of 2003. The currently deployed LCG middleware includes modules from the US (Globus, Condor) and the EU DataGrid project (data management tools, resource broker, VO organisation tools, fabric management tools). Grid middleware is evolving rapidly in anticipation of the forthcoming OGSI standard and EGEE partners are tracking these developments frequently. The functionality and scope will be at least comparable to that provided by the final versions of the EU DataGrid project software. Fabrizio Gagliardi: Presentazione stato progetto EGEE GDB Meeting 9/9/03

Timeline 2 EGEE Collaboration meeting early October Final version of TA by October 12, discussion and last possible negotiation meeting with the EU on October 17th Contract signature early next year Official start of the project on April 1st 2004 Fabrizio Gagliardi: Presentazione stato progetto EGEE GDB Meeting 9/9/03

IL PROGETTO LCG E IL PROGETTO EGEE LCG e EGEE condividono la responsabilità e il management dei seguenti settori: Middleware: implementazione delle raccomandazioni stabilite in HEPCAL II (HEP basic Grid Requirements) e ARDA (Working group dell’ SC2 sulle funzionalita’ comuni per l’ analisi distribuita) Infrastructure operation: LCG costituirà il nucleo fondamentale per lo sviluppo della grid EGEE Deployment of HEP application: integrazione con gli esperimenti LHC (risorse limitate) Questo aspetto e’ particolarmente rilevante in quanto significa che EGEE fara’ propri i requirements stabiliti da LCG

APPLICATIONS AREA: PROGETTI Software Process and Infrastructure (SPI) (A.Aimar) Librarian, QA, testing, developer tools, documentation, training, … Persistency Framework (POOL) (D.Duellmann) Relational persistent data store Core Tools and Services (SEAL) (P.Mato) Foundation and utility libraries, basic framework services, object dictionary and whiteboard, math libraries Physicist Interface (PI) (V.Innocente) Interfaces and tools by which physicists directly use the software. Interactive analysis, visualization Simulation (T.Wenaus) Generic framework, Geant4, FLUKA integration, physics validation, generator services Close relationship with -- ROOT (R.Brun) ROOT I/O event store; analysis package Group currently working on distributed analysis requirements – which will complete the scope of the applications area

POOL Status First production release of the POOL object persistency system made on time in June Level 1 milestone of the LCG project The base functionality requested by experiments for the data challenges in 2004 First experiment integration milestones – met at end July - use of POOL in CMS pre-challenge simulation production Completion of first ATLAS integration milestone scheduled for this month POOL is now being deployed on the LCG-1 service Close collaboration organised between POOL team and experiment integrators Take-up by the experiments now beginning Les Robertson: Presentazione stato progetto LCG LHCC Open Session 24/9/03

DEPLOYMENT AREA: The LHC Grid Service Qui si parrà la tua nobilitate OBIETTIVIVO PRINCIPALE: FORNIRE UN SERVIZIO DI PRODUZIONE PER I DATA CHALLENGES NEL 2004 IL PLANNING ORIGINARIO PREVEDEVA: M-1.1 Luglio 2003 partenza del servizio pilota M-1.2 Fine 2003 disponibilità del sistema di produzione La prima milestone ha due mesi di ritardo, essenziallmente legati al ritardo di una componente di grid middleware (RGMA) la componente e’ stata sostituita con un’ altra, sia pure con minori funzionalita’ (MDS) grazie al fondamentale contributo INFN Cfr discussione ultima Commissione I La componente di grid middleware continua a mancare e non e’ chiaro se sara’ disponibile per la release da utilizzare nei Data Challenges degli esperimenti OBIETTIVO: cercare di mantenere la seconda milestone per garantire il servizio di produzione per i Data Challenges degli esperimenti

The LCG Service Main Elements of a Grid Service Middleware: Integration, testing and certification Packaging, configuration, distribution and site validation Operations: Grid infrastructure services Local Regional Centre operations Operations centre(s) – trouble and performance monitoring, problem resolution, global coverage Support: Integration of experiments’ and Regional Centres’ support structures Grid call centre(s); documentation; training Coordination and Management Area Manager – Ian Bird (CERN) Grid Deployment Board chair – Mirco Mazzucato (Padova) National membership Policies, resources, registration, usage reporting Security Group chair – David Kelsey (RAL) Security experts Close ties to site security officers Security model, process, rules Daily Operations Site operations contacts Grid operations centre Grid call centre Les Robertson: Presentazione stato progetto LCG LHCC Open Session 24/9/03

LCG Service Status Middleware package – components from – European DataGrid (EDG) US (Globus, Condor, PPDG, GriPhyN)  the Virtual Data Toolkit Agreement reached on principles for registration and security Certification and distribution process established and tested - June Rutherford Lab (UK) to provide the initial Grid Operations Centre FZK (Karlsruhe) to operate the Call Centre Pre-release middleware deployed to the initial 10 centres – July The “certified” release was made available to 13 centres on 1 September – Academia Sinica Taiwan, BNL, CERN, CNAF, FNAL, FZK, IN2P3 Lyon, KFKI Budapest, Moscow State Univ., Prague, PIC Barcelona, RAL, Univ. Tokyo Les Robertson: Presentazione stato progetto LCG LHCC Open Session 24/9/03 Status on 24 July 2003 – I will email an update on the 11th August There are two activities going on in parallel: -- Pre-production software is being deployed to 10 Regional Centres (Academia Sinica Taipei, BNL, CERN, CNAF, FNAL, FZK, IN2P3 Lyon, Moscow State Univ., RAL, Univ. Tokyo). This is a final validation of the distribution and installation process and tools, and establishes the network of service support people, but the Grid middleware is not the final version yet. About half of the sites have successfully installed this – a few (including BNL and Lyon) are very slow. -- Final testing is under way of the first production set of Grid middleware. This is being done by a group of grid experts from the LHC experiments (the “loose cannons”). There are a few “show stoppers” and about 15 very serious bugs. The target is to install the production software by the end of the month – one week from now!

LCG Service – Next Steps Experiments now starting their tests on LCG-1 Still a lot of work to be done - especially operations-related tasks This will require active participation of regional centre staff Preparing now for adding new functionality in November to be ready for 2004 Implies deployment of a second multi-site testbed Web-site being set up at the Grid Operations Centre (Rutherford) with online monitoring information – see http://www.grid-support.ac.uk/GOC/ Les Robertson: Presentazione stato progetto LCG LHCC Open Session 24/9/03

LCG Service Time-line Level 1 Milestone – Opening of LCG-1 service computing service physics agree spec. of initial service (LCG-1) 2003 2004 2005 2006 2007 first data open LCG-1 (schedule – 1 July) open LCG-1 (achieved) – 1 Sept used for simulated event productions Level 1 Milestone – Opening of LCG-1 service 2 month delay, lower functionality than planned use by experiments will not start before October decision on final set of middleware for the 1H03 data challenges will be taken without experience of production running reduced time for integrating and testing the service with experiments’ systems before data challenges start next spring additional functionality will have to be integrated later Les Robertson: Presentazione stato progetto LCG LHCC Open Session 24/9/03

LCG Service Time-line computing service physics agree spec. of initial service (LCG-1) 2003 2004 2005 2006 2007 open LCG-1 (achieved) – 1 Sept used for simulated event productions LCG-2 - upgraded middleware, mgt. and ops tools principal service for LHC data challenges Computing model TDRs* LCG-3 – second generation middleware validation of computing models TDR for the Phase 2 grid Phase 2 service acquisition, installation, commissioning experiment setup & preparation Phase 2 service in production Les Robertson: Presentazione stato progetto LCG LHCC Open Session 24/9/03 first data * TDR – technical design report

LCG-1 DEPLOYMENT STATUS Summary Middleware was 3 months late – lesser functionality Deployment process was slow at some sites – not just lack of expertise System is now deployed at scale foreseen Experiments planning their testing now Tight timescale for improvements ready for 2004 DC’s Getting sites to contribute real effort is difficult Ian Bird: Lcg-1 Deployment status LHCC referees 23/9/03

Human Resources Consumed Les Robertson: Presentazione stato progetto LCG LHCC Open Session 24/9/03 without Regional Centres

RUOLO DELL’ INFN: CONTRIBUTO AL PROGETTO Contributo al management Contributo al personale 4 associates, 8 fellows 744 KCHF hardware Tier-1 + 3 Tier-2 nel Testbed LCG-1 + altri siti che stanno svolgendo un ruolo attivo e importante in questa fase delicata di attuazione del servizio di produzione LCG-1

CONCLUSIONI Il progetto europeo EGEE (~ 32 MEUR) e’ entrato nella fase finale di negoziazione Continuita’ con EDG/EDT Sinergia con LCG nel management, nelle scelte sul middleware, nell’ infrastruttura POOL (Persistency Framework) sta rispettando la schedula, per l’ integrazione con gli esperimento cfr i rispettivi talks Il servizio LCG-1 e’ partito ma… ha un ritardo di circa 2 mesi Mancano ancora alcune componenti di grid middleware Il mantenimento della milestones sull servizio di produzione (necessario per i Data Challenges degli esperimenti) determinera’ quasi certamente una riduzione delle funzionalita’ (in quale misura sara’ chiaro nei prossimi due mesi) La riorganizzazione del management dovrebbe consentire di superare alcuni problemi nell’ interazione con gli esperimenti