Globus is a robust online tool that simplifies the process of transferring large (GB or TB) datasets to and from RCC systems. This service enables you to transfer data from your personal workstations, share it with other researchers, or move data among different HPC providers.
Example use cases:
- Reliably transfer a large amount of data from the RAS, Panasas, Lustre to another machine or server.
- Reliably transfer a large amount of data from another machine or server onto the RAS, Panasas, Lustre.
- Setup automated replication of data between the RAS, Panasas, Lustre and another server or computer.
Setting up a Globus Account and Endpoint
Before you can transfer data to and from RCC systems, you must setup a Globus Account and configure the target computer (personal computer or server) as a Globus Endpoint.
- Visit http://globus.org and sign up for an account. Then sign in.
- Browse to the Manage Endpoints page.
- Click the Add Globus Connect Personal link.
- When prompted, create an Endpoint name.
- Click Generate Setup Key, and yake note of the key when it appears.
- Click the appropriate client download button for your operating system.
- Install the client and follow the on-screen instructions. You'll need your setup key for this step.
Once you have setup an account and endpoint, you can transfer data anytime.
- With the Globus Client running, browse to the Transfer Files page at http://globus.org.
- On the right-hand side of the screen, enter the name of your endpoint (or click the "..." button to find your endpoint in a list)
- Select the path in your file tree that you wish to transfer your files to or from.
- On the left-hand site, enter:
- fsurcc#panfs to transfer data from Panasas
- fsurcc#lustre to transfer data from Lustre
- fsurcc#archival to transfer data from the Research Archival Storage
- Click Go
- In the authorization dialog box that appears, enter your RCC username and password.
- Browse to and select the files and folders you wish to transfer. Then, click the large arrow in the direction that you wish to transfer files.
- Your files will now be queued for transfer. You can monitor the progress of your transfer by viewing the Activity on the Globus website.
Setting up CLI Access (no client needed!)
You can use the Globus CLI client to transfer data to and from a remote server using SSH. This is a good strategy to use when trying to reliably move data from a remote, firewall-protected server to RCC. You do not need admin privileges on the server for this method. You will need a public/private keypair.
- Ensure you have signed up for an account and created an endpoint (see Setting up a Globus Account and Endpoint above).
- On the Globus website, browse to the Manage Identities page.
- Click the Add Linked Identity link, and select Add SSH Public Key.
- Upload your Public Key.
Using CLI Access
Activate your endpoint: ssh -t [GLOBUS-USERNAME]@cli.globusonline.org [ENDPOINT#NAME]
Supply your Globus password when prompted
Transfer files using the following syntax:
ssh [GLOBUS-USERNAME]@cli.globusonline.org scp [ENDPOINT#NAME]:/path/to/files [FSURCC#FILESYSTEM]:~/destination/path
For example, to transfer a file named foobar.txt from your home directory to your Lustre space:
ssh email@example.com scp john-doe#laptop:~/foobar.txt fsurcc#lustre:~
There is more you can do with Globus. Refer to the official documentation for more information.