BaBar Tier A Administration Workshop CCR, Paestum Giugno 2003 Alberto Crescente, INFN Sez. Padova

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – Necessary Administration Informations ➢ Log Files ➢ Computer Model ➢ Connections Informations ➢ Computer Location ➢ Monitoring Informations ➢ Machine's Informations ➢ Temperatures, Fans ➢ CPU Load, I/O Load ➢ Hardware Errors ➢ Messages ➢ Mail ➢ Mysql

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – Used Tools ➢ Machines Database (WebDB) Machine Informations (CPU, RAM, etc.) Repair Informations Machine Connections Machine Location ➢ Log Analizer (sysAlarm) Syslog analizer ➢ Performance Monitor (perfMC) CPU Load I/O Load Temperatures

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – General Data Flow WebDB HTML MySQL PHP WebDB to xCAT PHP Extreme Cluster Administration Toolkit Configuration Files Clients & Servers syslogs SysAlarm Management Server syslog Script Perl /var/log/messag es SysAlarm PerfMC web server HTML/XML Language C PerfMC pooler RR D SNMP Farmcfg

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – sysAlarm Data Flow Clients & Servers syslogs SysAlarm Management Server syslog MySQL Script Perl /var/log/message s SysAlarm configuration file Query

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – sysAlarm Configuration File logPath./log fileToMonitor /var/log/messages # Specify the type of file to monitor ('syslog' or 'chslog') monitorType syslog debug 1 # # You can have multiple matchLine lines # matchLine rx buffer matchLine timeout waiting matchLine System Error # # this is the "basename" for the logfile # sysAlarm # Notification recipients # program # Default message for the subject # mailBin /bin/mail Subject sysAlarm Notification # # database that will store log messages... # #sqlserv bbr-user sqlserv sqluser xxx sqlpwd yyy sqldb myAlarm sqltable syslog # # Choose what to do with notification messages # send 0 writeSql 1

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – WebDB Features ➢ Clients and Servers Archive Informations Machine Name CPU RAM etc. ➢ Assistance Calls ➢ Event Log Viewer ➢ Machines Location

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – WebDB Data Flow Farmcfg WebDB HTML SysAlarm MySQL PHP WebDB to xCAT PHP Script Extreme Cluster Administration Toolkit Configuration Files

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – WebDB Home Page

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – WebDB Machine Details

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – WebDB Machine Details

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – WebDB Farms Structure

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – WebDB Rack Machines Location

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – WebDB Rack Machines Location

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – WebDB Assistence Calls

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – WebDB Syslog

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – PerfMC Features ➢ It allows the control of all the devices that interface with SNMP ➢ It allows the execution of simultaneous requests ➢ It allows a high sampling frequency ➢ It doesn't affect the cluster performances ➢ It permits to build a user interface thanks to the integrated web server and PHP support.

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – PerfMC Data Flow Farmcfg PerfMC web server HTML/XML MySQL Language C PerfMC poller RRD SNMP

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – PerfMC

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – PerfMC

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – PerfMC

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – PerfMC

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – PerfMC

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – PerfMC New Interface

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – PerfMC New Interface

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – PerfMC New Interface

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – PerfMC Future Implementations ➢ Integration of a system alarm that notifies malfunctions through SMS and ➢ Integration with HTTPS protocol to crypt informations ➢ Integration with the monitoring environmental parameters

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – Assistance Calls 0,15% CPU time/year lost for hardware failures Failures NumberComponent's Number% FailuresComponent ,09IDE Disk 21401,43SCSI Disk ,21Mainboard 53001,67Power Supply 43841,04CPU 45760,69RAM 10 Other TOTALE74 Failures in 1 year

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – Assistance Calls

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – Assistance Calls New Machines Installed

Alberto Crescente, servizio calcolo Padova Workshop CCR, Paestum Giugno 2003 Farm BaBar – References BaBar Farm Home Page WebDB PerfMC Farm System Administration