ATLAS HLT/DAQ V. Vercesi for the ATLAS Italia HLT/DAQ Group CSN1 Trieste Settembre 2006

V. Vercesi - INFN Pavia 2  S. Falciano (Roma1) Coordinatore Commissioning HLT  A. Negri (Irvine, Pavia) Coordinatore Event Filter Dataflow  A. Nisati (Roma1) TDAQ Institute Board chair e Coordinatore PESA Muon Slice  F. Parodi (Genova) Coordinatore b-tagging PESA  V. Vercesi (Pavia) Deputy HLT leader e Coordinatore PESA (Physics and Event Selection Architecture)  Attività italiane  Trigger di Livello-1 muoni barrel (Napoli, Roma1, Roma2)  Trigger di Livello-2 muoni (Pisa, Roma1)  Trigger di Livello-2 pixel (Genova)  Event Filter Dataflow (LNF, Pavia)  Selection software steering (Genova)  Event Filter Muoni (Lecce, Napoli, Pavia, Roma1)  DAQ (LNF, Pavia, Roma1)  DCS (Napoli, Roma1, Roma2)  Monitoring (Cosenza, Napoli, Pavia, Pisa)  HLT/DAQ system commissioning and exploitation (Everybody)

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 3 ATLAS TDAQ Dual(quad)-CPU nodes SDX1 USA15 UX15 ATLAS detector Read- Out Drivers ( RODs ) First- level trigger Read-Out Subsystems ( ROSs ) UX15 USA15 Dedicated links Timing Trigger Control (TTC) 1600 Read- Out Links Gigabit Ethernet RoI Builder Regions Of Interest VME ~150 PCs Data of events accepted by first-level trigger Event data requests Delete commands Requested event data Event data ≤ 100 kHz, 1600 fragments of ~ 1 kByte each LVL2 Super- visor DataFlow Manager Event Filter (EF) pROS ~ 500~1600 stores LVL2 output ~100~30 Network switches Event data pulled: partial ≤ 100 kHz, full ~ 3 kHz Event rate ~ 200 Hz Data storage Local Storage SubFarm Outputs (SFOs) LVL2 farm Network switches Event Builder SubFarm Inputs (SFIs) Second- level trigger

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 4 Efficienza dell’algoritmo di selezione è > 99% Tutte le inefficienze sono dovute a zone dello spettrometro non coperte da RPC. In particolare: Settori speciali (Feet), ascensore, crack  =0 e supporti del magnete. Efficienza dell’algoritmo di selezione è > 99% Tutte le inefficienze sono dovute a zone dello spettrometro non coperte da RPC. In particolare: Settori speciali (Feet), ascensore, crack  =0 e supporti del magnete. No RPC hit No Trigger Low-Pt Trigger DC3 data DC2 data Muon sources 6 GeV 6 GeV threshold Lumi= GeV 20 GeV threshold Lumi=10 34  /K b c W327 tNegligible Low-pT 6 GeV Threshold Efficienza globale Level-1 nel Barrel: 83% Low-pT Trigger 79% High-pT Trigger Level-1 Muon Trigger System

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 5 One Switch rack - TDAQ rack port GEth for L2+EB One ROS rack - TC rack + horiz. Cooling - 12 ROS 48 ROBINs One Full L2 rack - TDAQ rack - 30 HLT PCs Partial Superv’r rack - TDAQ rack - 3 HE PCs Partial EFIO rack - TDAQ rack - 10 HE PC (6 SFI - 2 SFO - 2 DFM) Partial EF rack - TDAQ rack - 12 HLT PCs Partial ONLINE rack - TDAQ rack - 4 HLT PC (monitoring) 2 LE PC (control) 2 Central FileServers RoIB rack - TC rack + horiz. cooling - 50% of RoIB 5.5 surface: SDX1 underground : USA15 Pre-series system in ATLAS point-1 8 racks (10% of final dataflow, 2% of EF) ROS, L2, EFIO and EF racks: one Local File Server, one or more Local Switches Machine Park: Dual Opteron and Xeon nodes, uniprocessor ROS nodes Operating System: Net booted and diskless nodes, running SLC3

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 6 Event Building Parameterizing observed EB measurements, we better understand the conditions for stable & performant operations TS = number of requests from SFI nodes for event fragments WT = number of events processed in parallel by L2 farm  = L2 accept ratio

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 7 Pre-series studies

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 8 ROD Crate DAQ  Last major upgrades  To accommodate detectors’ requests  New API for operational monitoring  To get “coherent” operation information Simultaneous module readout in a crate  RCD creates a packet of info and sends it to Monitoring  A monitoring task analyses the packet  Further improvements in last TDAQ version  OKS schema for RCD and ROS simplified  Detector modules in a crate can be managed Serially or in multi-threaded mode Each state transition can be set as multi-threaded  Can dramatically improve initialization time  RCD plans  The RCD structure should be stable enough  All the main requests from detectors have been fulfilled  We continue supporting the implementations  Main task is now RCD commissioning  Goes on with ROD commissioning  Includes support to all detectors

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 9 Monitoring Framework

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 10Monitoring  Reliable and stable GNAM CORE  Configurable from on-line database, with histogram history  Logging system (to migrate to ERS)  Access Manager to ATLAS conditions DB  Detector libraries  MDT, RPC, Tile, CSC, (TGC): in use for commissioning  Interface to Event Display available (for some detectors)  Main issue: Smart Monitoring  Automatically catch most low-level problems  Needed for reaching and maintaining stable running conditions  Common tools needed  Histogram comparison toolkit  Error message routing and filtering  Detector experts  Identify most common problem sources  Implement and test alarm strategies

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 11 HLT Selection algorithms  Lots of ongoing activities on algorithm development side  LVL1 and LVL2 algorithms run in CSC production for all trigger slices  EF algorithms almost finished (target , next week)  Lots of ongoing work to improve/test/validate algorithms  PESA algorithms being reviewed (Project Milestone)  ID LVL2, ID EF, Calo LVL2 e/g reviews have already taken place  Muon LVL2+EF ~90% complete, Calo EF e/  in October  Calo jet/tau/etmiss simple extension of e/   Building up comprehensive information about performance  Ongoing work for testing in athenaMT/athenaPT  Measurements of system (timing) and physics performance  LVL1/HLT AODs fully available in Rel 12 for trigger-aware analyses (Project Milestone)  Most data in place for ESD & AOD in release , Apr 06  More hypothesis algorithms, slice configs and TriggerDecision in

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 12 HLT Steering  Il cuore delle sequenze di selezione e decisione  Sviluppi basati sull’attuale implementazione (12.0.2)  Introduzione di selezioni topologiche elementari  Primi test attualmente in sviluppo nel settore della B-fisica (B→ , J/Psi→ , B S →D S  con  D S →  )  Introduzione di meccanismi per la sincronizzazione delle sequenze di algoritmi  Selezioni combinate di diversi oggetti (ad es. singolo e, singolo jet, singolo  )  Sincronizzazione delle selezioni topologiche  Iterazione del design di implementazione  Ridisegno dell’implementazione dello Steering  Nuove interfacce, più generali e flessibili, per gli algoritmi di feature extraction e hypothesis testing  Ristrutturazione della navigazione, disaccoppiata dallo Steering, per essere facilmente utilizzata nelle analisi Offline  Interfacciamento con l’introduzione del concetto di Trigger Chain  Fase di disegno quasi ultimata: inizio dell’attività di sviluppo e test

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 13 Electrons  Eff. w.r.t truth for E T =25GeV e±  Time per track in top events (~1.3 tracks/RoI with pTmin=0.5GeV)  Brem recovery to be optimized: in progress…  Optimizations ongoing also for eff/rej Photons  Use only calorimeter information at L2 and EF  Use CSC data, single photons E T = 20, 60 GeV, filtered QCD di-jets with E T (hard) > 17 GeV 97% 98% AlgorithmKalmanFitterGSF Time/track9.5 ms1.24 s Eff %  20i Rate 2  20i Eff %  60 Rate  60 L Hz Hz L2 Calo92.76 Hz Hz EF Calo85.34 Hz Hz

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 14 Efficiencies for SM signals trigger 2e15i67.2%--- e25i92.9%79.6% e6020.4%6.9% all94.8%80.3% Z ee W e Efficiencies given after kinematical cuts: 2e in |  | 15 GeV for Z  ee 1e in |  | 25 GeV for W  e Use trigger optimisations giving 80% overall eff (outside crack) for 2e15i, e25i, e60. e25i efficiency stays flat after turn-on. efficiency stays flat after turn-on. Turn on curves similar for e60. # after L1+L2+EF # after kinematical cuts eff  Method for single-electron trigger efficiency determination from data in progress (, ) Z ee Z 

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 15 Trigger efficiency from Z → μ + μ -  Double Object (DO) method  Control sample: “Good” Z from 2 offline  +  - with loose selection cuts + 1  ± trigger signature satisfied  Trigger efficiency determined from counting in how many cases the second  ± satisfies the trigger requirements EF TrigMoore (MuonSpectrometer standalone) EF TrigMoore (MuonSpectrometer + InnerDetector) RecoMC ε EF (%)96.4 ± 0.2 ε L1+EF (%)79.9 ± ± 0.2 RecoMC ε EF (%)94.5 ± 0.2 ε L1+EF (%)78.3 ± ± 0.2 For  (µ20)>70% statistical uncertainty after 30min at L=10 33 cm -2 s -1: ~1-2% Ongoing study using complementary method of orthogonal signature from ID (DOS method), almost finished

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 16  Barrel:  Final update of the LUT available: optimized with respect to the momentum resolution and to the efficiency  Hypothesis exists and the P T cut is optimized for the 6 GeV threshold  Endcap:  Bugs found in the TGC reconstruction code and in the analysis programs  Situation improved: harsh zones still remains, but results are understood now  Only TGC data has been used: try to measure muon P T and to parameterize the track path  An offline study of the momentum resolution obtained fom MDT data is also under way: preliminary results are in agreement with those we got using only TGC data. Large dishomogenity Endcap stations outside magnetic field Innermost TGC wheels have reduced coverage in small sectors EndCap 6 GeV 20 GeV Endcap: 1.05 <l  l <2.4 CSC data prelim. p T (GeV) efficiency

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 17  Very active trigger slices, just one year ago only small part of the selections were available (new groups joining TDAQ)  Jets  Algorithms for LVL1/LVL2/EF reco and hypothesis in place  Lots of work carried out to allow trigger-aware analysis  Ongoing work to understand physics performance  Tau  Algorithms for LVL1/LVL2/EF reco and hypothesis in place  Use tauRec at EF, support for other tau reco packages foreseen  Ongoing work to find best sequence at LVL2: first calo reconstruction or first tracking reconstruction  L2 selection ran successfully in athenaMT in  ETmiss  Study performance of ETmiss at LVL1 and EF

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 18 Prescale Jets l First place where to study and implement pre-scales l Reconstructed Jet E T distribution for a combination of J1 to J8 samples, scaled according to their respective cross-section

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 19 Algorithms in Pre-Serie  Motivation  Get the trigger PESA algorithms working online  Make sure the latest offline developments get running online  Historical background  Tools allowing the emulation of online running were developed  athenaMT (LVL2) and athenaPT (EF)  A group of people was formed in order to provide full slice job options suitable for online running  Jets (Ignacio Aracena)  Taus (Pilar Casado, Richard Soluk)  Egamma (Xin Wu, Imma Riu)  Muons (Alessandro Di Mattia, Diana Scannicchio)  One single integrated job options built successfully able to run in the Pre-Serie L2 and EF farms from real data format (bytestream)  Try this integrated job options in an online partition with different input files  Single electrons, Top events, Muon events  Produce a unified input BS file to be tested online  Produce a BS output file from SFO and try extending the menu to other signatures (e.g. di-objects) using the same algorithms slices to see the trigger behaviour

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 20Running L2 partition running ONLINE single electrons with the integrated job options in the nightlies of

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 21 More running

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 22Databases Online OracleDB Offline master CondDB Tier-0 recon replica Diagnostics replica Tier-1 replica Tier-1 replica Online Detectors HLT farm Diagnostics Tier-0 farm Bypass (ATCN / CERN-IT) ATLAS Point1 CERN Computer centre Remote sites Isolation / cut Calibration Updates including special Muons calibration path ATCN Network OKS in-memory OODB COOL RelDB CORAL* DCS Athena Server Data: in the DB, plus in (large) files referenced from the DB RDB systems used: Oracle, MySQL, sqlite *layer to decouple clients

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 23 Trigger Configuration  TriggerTool  GUI for DB population  easy and consistent menu changes for experts (LVL1 and HLT)  TriggerDB  stores all information to configure the trigger: LVL1 menu, HLT menu, HLT algorithm parameters (JO), HLT release information  stores all versions used, with a key  Configuration and Conditions DB  DB available at Point 1 and replicated to external sites  Retrieval of information for running  get information by key via two paths  extraction of data in XML/JO files  direct read-out  for both online + offline running TriggerDB online running offline running shift crewoffline userexpert TriggerTool DB population scripts Data Flow: R/O interface Configuration System compilers

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 24 Example database for TrigConf LVL1 HLT jobOptions HLT menu HLT release keys: stored in Conditions DB => Trigger conditions precisely known

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 25 HLT Large Scale Tests  Test complete HLT on many nodes  Scalability of DAQ, DB, HLT sw  Emphasis on DB, HLT (no LVL1)  DAQ: verify scalability & stability  DataBase:  Explore caching tools (DBProxy)  Detector oriented tests  1000 to 1200 nodes  LST 2005: 600 (512 used for LVL2)  Will be provided by IT (agreed) with lxbatch etc. machines  Non-optimized network  Big enough for full size Level2, ¼ size Event Filter  Test Level-2 and Event Filter together (+EB and SFO)  As many trigger algorithms as possible  Large number of events, recycled  Collect monitoring info  Physics plots (produced by algorithms)  Detailed timing histograms, etc.  Need to have SHIFTS!

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 26 LVL2 ID in cosmics  Tracking algorithms  IdScan, modified to handle single tracks with large impact parameters  SiTrack, version with special LUT tuned for muons not coming from IP  Variables studied  For T2Id, Offline and MC:  Ntracks, d0, phi0, z0, eta0  Efficiency calculations  Data: LVL2 wrt Offline  Simulation: LVL2 and Offline wrt truth Cosmics at SR1: instrumented region, two SCT sectors

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 27 Trigger-aware analysis  Analyses using trigger information as a “pre- processor” to correctly evaluate efficiencies, physics reach, etc.  The reconstructed objects, used by the trigger are saved in the ESD/AOD file  They can be used for comparison with truth/reconstructed information  It is possible to re-play the trigger decision, by running the hypothesis algorithms on these objects  Only the settings of the hypothesis algorithms can be changed in the analysis  The effect of different threshold settings can be measured ProductionAnalysisData taking

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 28 Trigger & Physics Weeks  Follow up on TDAQ and Computing operation  e.g. where are the bottlenecks on (instantaneous and average) rates?  Updates on trigger slices and initial feedback on Trigger Aware Analyses in Release 12  Work towards ultimate offline efficiencies and rejections, to be compared to corresponding figures for trigger selections  Especially relevant for Combined Performance e, t, m, b-tag  Follow up on minimum-bias event selection and analysis  Follow up on menu for L = 1 × cm -2 s -1 (items and rates)  Small group formed to study the issue and propose a strategy  Ideas shown need to be developed into menus for commissioning and for early physics (optimisation for two purposes somewhat different)  Follow up on physics with b-jets  Efficiency to identify the jets for tagging in LVL2 in various physics scenarios - e.g. rates and RoI multiplicities  “Hot topics”  End-cap LVL2 muons  Forward-jet trigger  ETmiss rates (including beam backgrounds)  Next week end of October

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 29Planning… Schedula in revisione alla luce delle nuove informazioni su LHC: discussioni durante la TDAQ Week di Settembre: cioè adesso, in perfetta sovrapposizione con la nostra CSN1 (Ho il volo per Londra domani mattina)

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 30 ROS plan  Prima tranche dei ROS già pagata al CERN (275 KCHF)  Seconda tranche e resto delle network card ordinata e consegna ultimata di recente, pagamento previsto entro la fine dell’anno (275 KCHF)  Questo esaurisce il nostro contributo CORE al Read-Out System  Impegno importante previsto in ME per contributi all’installazione e commissioning dell’intero sistema  Sezioni coinvolte Pavia, Roma1, …

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 31 Data Collection plan  Other DataFlow PCs  Price enquiry finished for SFI, DFM, L2SV and pROS  Evaluations nearly complete  Choose which company(ies) to buy from  SFOs - specification to be finalised in ~1 month  2007 CORE expected from Italy: 50 KCHF  Try and setup system capable of deliverying steady-state nominal SFO output from day 1  Better position to exploit early running trigger and detector studies  No correlation with total resources available in HLT farms  e.g. send LVL1 output directly to off-line

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 32 Networks and Online plan  Switch Procurements (Chassis + Pizza Boxes)  Plan finalised, shopping list produced for 2006  Sharing between FA’s agreed  File servers and Monitoring PC specs finalised  135 KHCF su CORE 2006, inclusi monitoring PC e local file server: end of contributions

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 33 Infrastructure plan  SDX Infrastructure  Orders started - cable ladders, nuts & bolts, shelves for switches  About to decide on power distribution  Coolers for SDX  Order placed for 43 racks (all of upper level)  2007 CORE expected from Italy: 80 KCHF

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 34 HLT farms plan  HLT Processors  IT Market Survey completed  Specifications has been drafted: similar (but not identical) to ATLAS needs  Not possible to use a blanket contract for 2006: buy our own  Lively discussion about spending profile for this items  First iteration ended Saturday September 3:44 (Big Ben time)  Propose today our best approach based on  Need to cover basic needs for commissioning, calibrations, first run period  Avoid buying anything that is not absolutely necessary  95 KCHF back to CSN1 in 2006  Other HLT/DAQ Items  First bills are appearing here - Rack coolers, inter-rack fibres, etc TDAQ Resource Committee

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 35 Revised Cost Profile (KCHF) Total Pre-series Detector R/O LVL2 Proc Event Builder Event Filter Online Infrastructure INFN Total TDR Total

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 36 Richieste ME (DAQ/HLT)  Per le responsabilità vedi la prima trasparenza  Installazione e commissioning ROS e HLT  4 mesi uomo a Roma1  2 mesi uomo a Pavia  Commissioning DAQ muoni/pixel  4 mesi uomo a Roma1 (incluso ROD Crate DAQ)  2 mesi uomo a Pavia (incluso sviluppo Monitoring)  9 mesi uomo a Genova (incluso HLT steering)  Event Building, SFI e SFO  8 mesi uomo a LNF (inclusi Large Scale Tests)  Deployment degli algoritmi di selezione on-line  2 mesi uomo a Pavia

CSN1 Trieste Settembre 2006 V. Vercesi - INFN Pavia 37Conclusioni  Stato attuale del progetto HLT/DAQ ben allineato con le scadenze future di ATLAS e di LHC previste nel 2007  Il progetto è certamente complesso e anche le responsabilità e i finanziamenti italiani coprono diversi settori  Il sistema di HLT/DAQ ha legami profondi con tutte le altre aree di sviluppo: rivelatori, software online e offline, networking, performance di fisica, etc  Sono necessari maggiori contributi alla forza lavoro per la parte di installazione e test  Impegno fondamentale di consentire a tutti i rivelatori una fase efficiente di commissioning  Inserimento degli algoritmi nei test di cosmici  Realizzazione di catene complete di read-out e successivamente di Event Building  Le slice HLT hanno raggiunto uno stadio di maturità avazata, sono integrate nell’ambiente on-line e costituiscono gli ingredienti per la costruzione e lo studio dei Menu di Trigger  Il Progetto è in fase di revisione (management)  Necessità di adatttarsi alla nuova fase dell’esperimento  Coerenza con l’ATLAS Operational Model  Impegno perché gli italiani mantengano i ruoli di visibilità che si sono meritati