File Transfer

From CLSP Wiki
Jump to: navigation, search

Sometimes it's nice to transfer files to and from your local machine and the grid.

Sharing files via the web

If you (temporarily) move a file to /export/user_uploads/vfsrv/people/share/FILENAME (where FILENAME is the unique name of your file) on the CLSP grid, then anyone can download it from http://www.clsp.jhu.edu/vfsrv/people/share/FILENAME. Do not use this folder for very large files that will generate high network traffic.

Note, you may need to update the file's permissions before it is accessible on the web (chmod 644 /export/user_uploads/vfsrv/people/share/FILENAME).

Consider using mktemp -p /export/user_uploads/vfsrv/people/share to automatically generate and print a unique, randomly named file under /export/user_uploads/vfsrv/people/share. For example, add the following to your ~/.bash_profile and run share foo.txt to use mktemp to create a randomly named file under /export/user_uploads/vfsrv/people/share and copy the contents of foo.txt to that file:

 share() {
     local basename=`basename "$1"`
     local temp_path=`mktemp -p /export/user_uploads/vfsrv/people/share --suffix ".$basename"`
     local temp_basename=`basename "$temp_path"`
     echo "$temp_path"
     cp -p "$1" "$temp_path"
     echo "http://www.clsp.jhu.edu/vfsrv/people/share/$temp_basename"
 }

scp

scp -r local-directory login.clsp.jhu.edu:~/path-on-the-grid

If transferring large files, use the -l flag to limit the transfer rate (in Kb/s):

scp -r -l 8000 local-directory login.clsp.jhu.edu:~/path-on-the-grid

Rsync

 rsync -avz --progress --partial local-directory/ login.clsp.jhu.edu:~/path-on-the-grid

For huge files, include the bwlimit flag, which limits the transfer rate (in KB/s):

 rsync -avz --bwlimit=1000 --progress --partial local-directory/ login.clsp.jhu.edu:~/path-on-the-grid

Diff between File on CLSP and File on Local Computer

Add the following bash function to your ~/.bash_profile, log out and back in, and run, for example, remotediff makefile projects/kbvn/src/kbmvlsa/makefile prastog3@login.clsp.jhu.edu to print a diff between makefile on your local machine and prastog3@login.clsp.jhu.edu:projects/kbvn/src/kbmvlsa/makefile .

 function remotediff () {
   # Diff between local and remote file.                       
   local localf="$1"
   local remotef="$2"
   local remote="$3"
   ssh $remote "cat $remotef" | diff - $localf
 }