High-level tools for Grid catalogue access, I/O and file transfer operations Flavia Donno INFN and CERN Scuola per utenti INFN della Grid Bologna, 27 November.

High-level tools for Grid catalogue access, I/O and file transfer operations
Flavia Donno INFN and CERN Scuola per utenti INFN della Grid Bologna, 27 November 2007

Scuola per utenti INFN della Grid, Bologna, 27 November 2007 - 2
Outline The LFC Grid File Catalogue The LFC CLI Example of LFC usage DM tools: lcg-utils and GFAL Examples of lcg-utils usage The GFAL interface ROOT file access File transfer operations: globus-url-copy The Grid File Transfer Service (FTS): Main concepts Examples of FTS usage Scuola per utenti INFN della Grid, Bologna, 27 November

The LFC Grid File Catalogue
Keeps track of the location of copies (replicas) of files on the Grid LFC holds the mapping LFN-GUID-SURL LFN 1 LFN i : SURL j GUID TURL j1 TURL jl TURL 11 TURL 1k SURL 1 LFC Scuola per utenti INFN della Grid, Bologna, 27 November

Scuola per utenti INFN della Grid, Bologna, 27 November

LFN acts as main key in the database. It has: Symbolic links to it (additional LFNs) Unique Identifier (GUID) System metadata Information on replicas One field of user metadata Scuola per utenti INFN della Grid, Bologna, 27 November

Two kinds of LFC Central LFC For each VO, one site on the grid will publish a global catalog. This will record entries (file replicas or dataset entities) across the whole Grid. Local LFC Local catalogs record the file replicas stored at that site's SEs only. Scuola per utenti INFN della Grid, Bologna, 27 November

Features Namespace operations All names are in a hierarchical namespace mkdir(), opendir(), etc… Also chdir() GUID attached to every directory and file Defined by the user LFC Namespace LFC has a directory tree structure /grid/<VO_name>/ <you create it> Scuola per utenti INFN della Grid, Bologna, 27 November

Features Security – GSI Authentication and Authorization Mapping done from Client DN to uid/gid pair Authorization done in terms of uid/gid VOMS will be integrated (collaboration with INFN/NIKHEF) VOMS roles appear as a list of gids Ownership of files is stored in catalog Permissions implemented Unix (user, group, all) permissions POSIX ACLs (group and users) Scuola per utenti INFN della Grid, Bologna, 27 November

The LFC Interfaces LCG UTIL LFC SERVER GFAL Python LFC CLIENT C API DLI WMS CLI lfc-ls, lfc-mkdir, lfc-setacl, … Scuola per utenti INFN della Grid, Bologna, 27 November

The LFC CLI Add/replace a comment lfc-setcomment Set file/directory access control lists lfc-setacl Remove a file/directory lfc-rm Rename a file/directory lfc-rename Create a directory lfc-mkdir List file/directory entries in a directory lfc-ls Make a symbolic link to a file/directory lfc-ln Get file/directory access control lists lfc-getacl Delete the comment associated with the file/directory lfc-delcomment Change owner and group of the LFC file-directory lfc-chown Change access mode of the LFC file/directory lfc-chmod Scuola per utenti INFN della Grid, Bologna, 27 November

Finding the catalogue lcg-infosites Example: finding an SE: > lcg-infosites --vo atlas se Avail Space(Kb) Used Space(Kb) Type SEs n.a n.a gb-se-ams.els.sara.nl n.a n.a gb-se-wur.els.sara.nl n.a se.grid.rug.nl n.a srm.grid.sara.nl Example: finding an LFC > lcg-infosites --vo lhcb lfc lfc.grid.sara.nl Scuola per utenti INFN della Grid, Bologna, 27 November

Finding the catalogue lcg-info For more advanced searches: For example, finding out where to put your files >lcg-info --vo tutor --list-se -- query='SE=srm.grid.sara.nl' --attrs=Path - SE: srm.grid.sara.nl - Path /pnfs/grid.sara.nl/data/tutor Scuola per utenti INFN della Grid, Bologna, 27 November

Interaction with the WMS
The InputSandbox and OutputSandbox should only be used for small amounts of data. Large files should be on SEs The RB can locate Grid files: allows for data-based match-making Jdl file: InputData = "lfn:/grid/tutor/MyFile"; The lfn’s / guid’s needed by the job as an input to the process Tells RB to schedule job on CE close to SE holding the file glite-brokerinfo getInputData returns list of files in InputData attribute OutputSE=“storm-fe.cr.cnaf.infn.it”; location of a SE where the output data will be stored DataAccessProtocol=“gsiftp”; The list of protocols that the application is able to “speak” for accessing files listed in the InputData Scuola per utenti INFN della Grid, Bologna, 27 November

Data Management Tools The WLCG supported data management high-level tools and APIs are lcg-utils and the Grid File Access Library (GFAL) Lcg-utils provides both an API library and a Command Line Interface (CLI) Lcg-utils provides a high-level view of the SRM interface hiding the details of the implementations and the differences between the protocol versions (v1 vs v2) The current releases of lcg-utils/GFAL not yet released are respectively v1.6.4 and v These are the releases that offer support for SRM v2.2 Scuola per utenti INFN della Grid, Bologna, 27 November

Data Management Tools Scuola per utenti INFN della Grid, Bologna, 27 November

Data Management Tools lcg-gt [ -h,--help ] [ -t timeout ] [ -v,--verbose ] [ -i,--insecure ] [ -D,--defaultsetype defaulttype ] [ -T,--setype setype ] [ -b,--nobdii ] [ -s,--st " spacetokendesc" ] surl protocol lcg-gt –b –T srmv2 \ srm://lxshare0303.cern.ch:8443/srm/managerv2?SFN=/castor/cern.ch/home/dteam/test rfio rfio://lxshare0303.cern.ch//data/dt/stage/test.2415 TURL Request ID FileID Scuola per utenti INFN della Grid, Bologna, 27 November

Data Management Tools lcg-del [ -a ] [ --config config_file ] [ -h,--help ] [ -i,--insecure ] [ -D,--defaultsetype defaulttype ] [ -T,--setype setype ] [ -b,--nobdii ] [ -s,--se se ] [ -v,--verbose ] [ -t timeout ] [ --vo virtual_organization ] file lcg-del --vo dteam \ srm://lxshare0282.cern.ch//pnfs/cern.ch/data/dteam/testjpb16 lcg-del --vo dteam -a lfn:/grid/dteam/tests/testjpb_rf10 SRM v1.1 ?? Not yet from VOMS proxy Operations on the file catalogue Scuola per utenti INFN della Grid, Bologna, 27 November

Data Management Tools lcg-ls [ -h,--help ] [ -t timeout ] [ -D,--defaultsetype defaulttype ] [ -T,--setype setype ] [ -b,--nobdii ] [ -l,--long ] [ -d,--dir ] surl [...] lcg-ls srm://lxpdm101.cern.ch/dpm/cern.ch/home /dpm/cern.ch/home/alice /dpm/cern.ch/home/atlas /dpm/cern.ch/home/cms /dpm/cern.ch/home/dteam /dpm/cern.ch/home/gin /dpm/cern.ch/home/gin.ggf.org /dpm/cern.ch/home/lhcb /dpm/cern.ch/home/ops lcg-ls -b -T srmv2 -l srm://srm-v2.cern.ch:8443/srm/managerv2\?SFN=/castor/cern.ch/grid/dteam Scuola per utenti INFN della Grid, Bologna, 27 November

Grid File Access Library
It offers C and python wrappers Not all calls have an equivalent CLI Gfal_prestage allows to prestage a file from tape Gfal_pin extends the lifetime of the file copy on disk Gfal_release allows a user to release a file (lcg-sd) A thread-safe version is under development Scuola per utenti INFN della Grid, Bologna, 27 November

ROOT file access The ROOT framework offers many plug-ins to access files using various file access protocols in addition to the native root file protocol The basic call is the following: root [6] f=TFile::Open("rfio://castorlhcb.cern.ch:9002/?svcClass=lhcbdata&castorVersion=2&p ath=/castor/cern.ch/grid/lhcb/production/DC06/phys-v2-lumi2/ /DST/00”) ROOT can be used to access files via rfio and dcap ROOT has also a GFAL plugin: GFAL is invoked to handle the TURL passed in input root [8] f=TFile::Open("gfal:srm://srm-durable- lhcb.cern.ch:8443/castor/cern.ch/grid/lhcb/production/DC06/phys-v2- lumi2/ /DST/0000/ _ _5.dst") ATTENTION: there are problem with handling of TURLs provided by different storage services. For details check: Scuola per utenti INFN della Grid, Bologna, 27 November

globus-url-copy Scuola per utenti INFN della Grid, Bologna, 27 November

File Transfer Service FTS is a low level data movement service Why is it needed? Improves reliability for transfers Provides asynchronous file transfer schedule transfers when resources are available Provides control of transfer properties (channel concept) No catalogue interactions  users have to handle SURL Scuola per utenti INFN della Grid, Bologna, 27 November

File Transfer Service Transfer Job A set of source/destination pairs specifying files to transfer Submitted to FTS for processing Channel A job is assigned to a channel after submission Represents a point-to-point network link Catch all channels are possible: any-to-me, me-to-any Similar to a queue where you can specify VO share for the queue Number of concurrent file transfer Number of concurrent streams (gridFTP) Scuola per utenti INFN della Grid, Bologna, 27 November

FTS architecture All components are decoupled from each other Each interacts only with the database Experiments interact via web-service User: FileTransfer Admin: ChannelManagement VO agents assign jobs to channels Channel agents manage assigned file transfers Monitoring and statistics can be collected via the DB (FTM is under development) Scuola per utenti INFN della Grid, Bologna, 27 November

FTS clients glite-transfer- submit Submit a transfer job : needs at least source and destination SURL glite-transfer- status Given one or more job ID, query about their status glite-transfer- cancel Delete the transfer with the give Job ID glite-transfer-list Query about status of all user’s jobs; support options for query restrictions glite-transfer- channel-list Show all available channel; detailed info only if user has admin privileges -s option allows user to specify space token descriptions for destination No space tokens are foreseen for get operations FTS does not implement staging capabilities: a file must be in cache in order To be transferred succescully Scuola per utenti INFN della Grid, Bologna, 27 November

High-level tools for Grid catalogue access, I/O and file transfer operations Flavia Donno INFN and CERN Scuola per utenti INFN della Grid Bologna, 27 November.

Presentazioni simili

Presentazione sul tema: "High-level tools for Grid catalogue access, I/O and file transfer operations Flavia Donno INFN and CERN Scuola per utenti INFN della Grid Bologna, 27 November."— Transcript della presentazione:

Presentazioni simili

Sul progetto

Feed-back

Entrare

Autorizzarsi attraverso i social network:

High-level tools for Grid catalogue access, I/O and file transfer operations Flavia Donno INFN and CERN Scuola per utenti INFN della Grid Bologna, 27 November.

Presentazioni simili

Presentazione sul tema: "High-level tools for Grid catalogue access, I/O and file transfer operations Flavia Donno INFN and CERN Scuola per utenti INFN della Grid Bologna, 27 November."— Transcript della presentazione:

Presentazioni simili

Sul progetto

Feed-back