High-level tools for Grid catalogue access, I/O and file transfer operations Flavia Donno INFN and CERN Scuola per utenti INFN della Grid Bologna, 27 November 2007

Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 2 Outline The LFC Grid File Catalogue The LFC CLI Example of LFC usage DM tools: lcg-utils and GFAL Examples of lcg-utils usage The GFAL interface ROOT file access File transfer operations: globus-url-copy The Grid File Transfer Service (FTS): Main concepts Examples of FTS usage Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 2

The LFC Grid File Catalogue Keeps track of the location of copies (replicas) of files on the Grid LFC holds the mapping LFN-GUID-SURL LFN 1 LFN i : SURL j GUID TURL j1 TURL jl TURL 11 TURL 1k SURL 1 LFC Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 3

The LFC Grid File Catalogue Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 4

The LFC Grid File Catalogue LFN acts as main key in the database. It has: Symbolic links to it (additional LFNs) Unique Identifier (GUID) System metadata Information on replicas One field of user metadata Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 5

The LFC Grid File Catalogue Two kinds of LFC Central LFC For each VO, one site on the grid will publish a global catalog. This will record entries (file replicas or dataset entities) across the whole Grid. Local LFC Local catalogs record the file replicas stored at that site's SEs only. Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 6

Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 7 Features Namespace operations All names are in a hierarchical namespace mkdir(), opendir(), etc… Also chdir() GUID attached to every directory and file Defined by the user LFC Namespace LFC has a directory tree structure /grid/<VO_name>/ <you create it> Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 7

Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 8 Features Security – GSI Authentication and Authorization Mapping done from Client DN to uid/gid pair Authorization done in terms of uid/gid VOMS will be integrated (collaboration with INFN/NIKHEF) VOMS roles appear as a list of gids Ownership of files is stored in catalog Permissions implemented Unix (user, group, all) permissions POSIX ACLs (group and users) Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 8

Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 9 The LFC Interfaces LCG UTIL LFC SERVER GFAL Python LFC CLIENT C API DLI WMS CLI lfc-ls, lfc-mkdir, lfc-setacl, … Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 9

Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 10 The LFC CLI Add/replace a comment lfc-setcomment Set file/directory access control lists lfc-setacl Remove a file/directory lfc-rm Rename a file/directory lfc-rename Create a directory lfc-mkdir List file/directory entries in a directory lfc-ls Make a symbolic link to a file/directory lfc-ln Get file/directory access control lists lfc-getacl Delete the comment associated with the file/directory lfc-delcomment Change owner and group of the LFC file-directory lfc-chown Change access mode of the LFC file/directory lfc-chmod Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 10

Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 11 Finding the catalogue lcg-infosites Example: finding an SE: > lcg-infosites --vo atlas se Avail Space(Kb) Used Space(Kb) Type SEs ---------------------------------------------------------- 1320000000 n.a n.a gb-se-ams.els.sara.nl 1320000000 n.a n.a gb-se-wur.els.sara.nl 536868064 2848 n.a se.grid.rug.nl 104856555 1044 n.a srm.grid.sara.nl Example: finding an LFC > lcg-infosites --vo lhcb lfc lfc.grid.sara.nl Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 11

Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 12 Finding the catalogue lcg-info For more advanced searches: For example, finding out where to put your files >lcg-info --vo tutor --list-se -- query='SE=srm.grid.sara.nl' --attrs=Path - SE: srm.grid.sara.nl - Path /pnfs/grid.sara.nl/data/tutor Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 12

Interaction with the WMS The InputSandbox and OutputSandbox should only be used for small amounts of data. Large files should be on SEs The RB can locate Grid files: allows for data-based match-making Jdl file: InputData = "lfn:/grid/tutor/MyFile"; The lfn’s / guid’s needed by the job as an input to the process Tells RB to schedule job on CE close to SE holding the file glite-brokerinfo getInputData returns list of files in InputData attribute OutputSE=“storm-fe.cr.cnaf.infn.it”; location of a SE where the output data will be stored DataAccessProtocol=“gsiftp”; The list of protocols that the application is able to “speak” for accessing files listed in the InputData Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 13

Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 14 Data Management Tools The WLCG supported data management high-level tools and APIs are lcg-utils and the Grid File Access Library (GFAL) Lcg-utils provides both an API library and a Command Line Interface (CLI) Lcg-utils provides a high-level view of the SRM interface hiding the details of the implementations and the differences between the protocol versions (v1 vs v2) The current releases of lcg-utils/GFAL not yet released are respectively v1.6.4 and v1.10.5. These are the releases that offer support for SRM v2.2 Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 14

Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 15 Data Management Tools http://grid-deployment.web.cern.ch/grid-deployment/flavia/man/ Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 15

Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 16 Data Management Tools lcg-gt [ -h,--help ] [ -t timeout ] [ -v,--verbose ] [ -i,--insecure ] [ -D,--defaultsetype defaulttype ] [ -T,--setype setype ] [ -b,--nobdii ] [ -s,--st " spacetokendesc" ] surl protocol lcg-gt –b –T srmv2 \ srm://lxshare0303.cern.ch:8443/srm/managerv2?SFN=/castor/cern.ch/home/dteam/test rfio   rfio://lxshare0303.cern.ch//data/dt/stage/test.2415     939325759     TURL Request ID FileID Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 16

Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 17 Data Management Tools lcg-del [ -a ] [ --config config_file ] [ -h,--help ] [ -i,--insecure ] [ -D,--defaultsetype defaulttype ] [ -T,--setype setype ] [ -b,--nobdii ] [ -s,--se se ] [ -v,--verbose ] [ -t timeout ] [ --vo virtual_organization ] file lcg-del --vo dteam \       srm://lxshare0282.cern.ch//pnfs/cern.ch/data/dteam/testjpb16    lcg-del --vo dteam -a lfn:/grid/dteam/tests/testjpb_rf10 SRM v1.1 ?? Not yet from VOMS proxy Operations on the file catalogue Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 17

Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 18 Data Management Tools lcg-ls [ -h,--help ] [ -t timeout ] [ -D,--defaultsetype defaulttype ] [ -T,--setype setype ] [ -b,--nobdii ] [ -l,--long ] [ -d,--dir ] surl [...] lcg-ls srm://lxpdm101.cern.ch/dpm/cern.ch/home     /dpm/cern.ch/home/alice     /dpm/cern.ch/home/atlas     /dpm/cern.ch/home/cms     /dpm/cern.ch/home/dteam     /dpm/cern.ch/home/gin     /dpm/cern.ch/home/gin.ggf.org   /dpm/cern.ch/home/lhcb     /dpm/cern.ch/home/ops lcg-ls -b -T srmv2 -l srm://srm-v2.cern.ch:8443/srm/managerv2\?SFN=/castor/cern.ch/grid/dteam Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 18

Grid File Access Library It offers C and python wrappers Not all calls have an equivalent CLI Gfal_prestage allows to prestage a file from tape Gfal_pin extends the lifetime of the file copy on disk Gfal_release allows a user to release a file (lcg-sd) A thread-safe version is under development Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 19

Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 20 ROOT file access The ROOT framework offers many plug-ins to access files using various file access protocols in addition to the native root file protocol The basic call is the following: root [6] f=TFile::Open("rfio://castorlhcb.cern.ch:9002/?svcClass=lhcbdata&castorVersion=2&p ath=/castor/cern.ch/grid/lhcb/production/DC06/phys-v2-lumi2/00001650/DST/00”) ROOT can be used to access files via rfio and dcap ROOT has also a GFAL plugin: GFAL is invoked to handle the TURL passed in input root [8] f=TFile::Open("gfal:srm://srm-durable- lhcb.cern.ch:8443/castor/cern.ch/grid/lhcb/production/DC06/phys-v2- lumi2/00001650/DST/0000/00001650_00000055_5.dst") ATTENTION: there are problem with handling of TURLs provided by different storage services. For details check: https://twiki.cern.ch/twiki/bin/view/LHCb/FileAccess https://twiki.cern.ch/twiki/bin/view/Atlas/IssuesWithPosixIO Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 20

Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 21 globus-url-copy Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 21

Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 22 File Transfer Service FTS is a low level data movement service Why is it needed? Improves reliability for transfers Provides asynchronous file transfer schedule transfers when resources are available Provides control of transfer properties (channel concept) No catalogue interactions  users have to handle SURL Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 22

Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 23 File Transfer Service Transfer Job A set of source/destination pairs specifying files to transfer Submitted to FTS for processing Channel A job is assigned to a channel after submission Represents a point-to-point network link Catch all channels are possible: any-to-me, me-to-any Similar to a queue where you can specify VO share for the queue Number of concurrent file transfer Number of concurrent streams (gridFTP) Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 23

Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 24 FTS architecture All components are decoupled from each other Each interacts only with the database Experiments interact via web-service User: FileTransfer Admin: ChannelManagement VO agents assign jobs to channels Channel agents manage assigned file transfers Monitoring and statistics can be collected via the DB (FTM is under development) Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 24

Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 25 FTS clients glite-transfer- submit Submit a transfer job : needs at least source and destination SURL glite-transfer- status Given one or more job ID, query about their status glite-transfer- cancel Delete the transfer with the give Job ID glite-transfer-list Query about status of all user’s jobs; support options for query restrictions glite-transfer- channel-list Show all available channel; detailed info only if user has admin privileges -s option allows user to specify space token descriptions for destination No space tokens are foreseen for get operations FTS does not implement staging capabilities: a file must be in cache in order To be transferred succescully Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 25

Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 26