TensorFlow on the Grid

From CLSP Wiki
Jump to: navigation, search

Setup

The CPU version of TensorFlow does not need any special setup -- the commands for installing on Ubuntu should also work on the grid.

For GPU, our current version of CUDA on the grid is 9.1, while the compiled versions expect CUDA9.0. To get around this, follow the instructions to install the built versions at https://github.com/mind/wheels/releases.

Tips

If your program crashes with a warning stating that the GPU is already in use, this may be because you have a lot of CPU-based computation before using the GPU (assuming you reserved a GPU according to GPUs on the Grid). To get around this, you can claim a GPU earlier by making a tf.Session object as soon as possible in your code.

TensorBoard

To use tensorboard, you'll need to tunnel (port forward) through both `login` and the node itself with the `-L` flag and ensure that the environment has tensorboard installed. Here's an example (using ssh keys):


anywhere:~$ ssh login -L 1234:localhost:1234
login.clsp.jhu.edu:~$ ssh c05 -L 1234:localhost:1234
c05:~$ source ~/env/with/tensorboard/bin/activate
(tensorboard) c05~$: tensorboard --port=1234 --logdir=.


Next, go to localhost:1234 in your browser. More information on exact Tensorboard usage or port forwarding can be found elsewhere online.