File Transfer

From CLSP Wiki
Jump to: navigation, search

Sometimes it's nice to transfer files to and from your local machine and the grid.

Sharing files via the web

If you (temporarily) move a file to /export/user_uploads/vfsrv/people/share/FILENAME (where FILENAME is the unique name of your file) on the CLSP grid, then anyone can download it from Do not use this folder for very large files that will generate high network traffic.

Note, you may need to update the file's permissions before it is accessible on the web (chmod 644 /export/user_uploads/vfsrv/people/share/FILENAME).

Consider using mktemp -p /export/user_uploads/vfsrv/people/share to automatically generate and print a unique, randomly named file under /export/user_uploads/vfsrv/people/share. For example, add the following to your ~/.bash_profile and run share foo.txt to use mktemp to create a randomly named file under /export/user_uploads/vfsrv/people/share and copy the contents of foo.txt to that file:

 share() {
     local basename=`basename "$1"`
     local temp_path=`mktemp -p /export/user_uploads/vfsrv/people/share --suffix ".$basename"`
     local temp_basename=`basename "$temp_path"`
     echo "$temp_path"
     cp -p "$1" "$temp_path"
     echo "$temp_basename"


scp -r local-directory

If transferring large files, use the -l flag to limit the transfer rate (in Kb/s):

scp -r -l 8000 local-directory


 rsync -avz --progress --partial local-directory/

For huge files, include the bwlimit flag, which limits the transfer rate (in KB/s):

 rsync -avz --bwlimit=1000 --progress --partial local-directory/

Diff between File on CLSP and File on Local Computer

Add the following bash function to your ~/.bash_profile, log out and back in, and run, for example, remotediff makefile projects/kbvn/src/kbmvlsa/makefile to print a diff between makefile on your local machine and .

 function remotediff () {
   # Diff between local and remote file.                       
   local localf="$1"
   local remotef="$2"
   local remote="$3"
   ssh $remote "cat $remotef" | diff - $localf