Transferring Files

MLeRP does not have a dedicated data transfer node. We recommend that you use the login node when managing your files.

Jupyter Lab

The easiest option to move files into the MLeRP environment is to use a Jupyter Lab job through Strudel2 since it requires no additional setup. You should be able to drag and drop files into the filetree on the left though it comes with some limitations.

Jupyter Lab does not support transferring folders, so you will need to either zip your files first or create your desired structure first and drop in the files later. You can still select multiple files and drag and drop all of them at once. It also only supports dropping files into the directory you are viewing.

VS Code

If you have already gone through the steps to connect your VS Code you can drag and drop files into the filetree just like with Jupyter Lab, but VS Code will allow you to drop whole folders at once, including nested folders. You can control where the files and folders end up by hovering over the desired folder.

Wget

If you’re looking for an application or dataset that is publically available from the internet, you will be able to download the data directly into the MLeRP environment with a wget command:

wget -o destination URL

rsync

If you’re familiar with the terminal you can use rsync to synchronise file systems and to transfer large amounts of files, with the ability to stop and restart the file transfers. rsync will replicate all files in a folder from one spot to another. It first analyses both file systems to find the difference and then transfers only the changes.

A typical command to synchronise files from a local folder to MLeRP is:

For MLeRP QCIF:

rsync -auv -e ssh adirectory <username>@mlerp-login0.mlerp.cloud.edu.au:~/destinationdirectory/

For MLeRP Monash:

rsync -auv -e ssh adirectory <username>@monash-mlerp-login0.mlerp.cloud.edu.au:~/destinationdirectory/

rsync is very powerful and has many options to help transfer data. For example it can delete unwanted files with --delete, compress data before transfer -z or can you let you see what command options might do without actually executing them --dry-run. For more info on rsync try:

man rsync