Dask Offloading

Testing with a LocalCluster

Once you’ve written your function and are ready to move things over to . Working with Dask workers introduces another layer of complexity where things can go wrong, which make Dask LocalClusters the easiest way to prepare your code for offloading. This will mean that code will execute in the notebook session, just like running your function straight, allowing you to view print statements and debug errors normally rather than dealing with remote code execusion before we’re ready. Once you’re satisfied with your code you can switch over to a SLURMCluster to accelerate with GPU.

from distributed import Client, LocalCluster

cluster = LocalCluster()
client = Client(cluster)

We can submit our function to the cluster with the client.submit method. This will return a future which can be unpacked with its result using future.result(). We can see the outputs of print statements while we’re using a LocalCluster. Print statements will not be visible when executing remotely with SLURMCluster. Similarly, the full stack trace is still visible when an error or assertion is raised within the function.

def client_test(input1, input2, error=False, test=False):
    # Force an error
    if error:
        assert 0 == 1
    
    # Stop after one batch when testing        
    if test: 
        print("When running in a local cluster you can see print statements!")

    return input1, input2

future = client.submit(client_test, "input1", "input2", test=True)
future.result()

When running in a local cluster you can see print statements!

('input1', 'input2')

future = client.submit(client_test, "input1", "input2", error=True)
future.result()

2024-02-09 03:58:33,152 - distributed.worker - WARNING - Compute Failed
Key:       client_test-4a09713a18d9ca3a2cc865eed9a8b248
Function:  client_test
args:      ('input1', 'input2')
kwargs:    {'error': True}
Exception: 'AssertionError()'

AssertionError: 

client.shutdown()

Running on a SLURMCluster

We can pass in extra SLURM requirements in job_extra_directives to request a GPU for our jobs. To read more about configuring the SLURMCluster to interact with the SLURM queue, go to Dask’s jobqueue documentation.

from dask_jobqueue import SLURMCluster
from distributed import Client
cluster = SLURMCluster(
    memory="128g", processes=1, cores=16, job_extra_directives=["--gres=gpu:1"]
)

cluster.scale(1)
client = Client(cluster)

Since this code is executing remotely we won’t see our print statements

client.submit(client_test, "input1", "input2", test=True).result()

('input1', 'input2')

Dask will raise any errors that the process triggers locally, even when executing remotely - but you may not get the full stack trace

client.submit(client_test, "input1", "input2", error=True).result()

AssertionError: 

If you’re working with any objects that are particularly memory intensive, you can consider using the client.scatter method to scatter large objects out to our workers ahead of time for more efficient execution.

large_object = "Let's pretend that this string is actually a really big object like your dataset"
input1_future = client.scatter(large_object)
client.submit(client_test, input1_future, "input2").result()

("Let's pretend that this string is actually really big", 'input2')

client.shutdown()

In the next tutorial we’ll see how we can apply this kind of offloading to training a model in PyTorch.

cluster = SLURMCluster(
    memory="128g", processes=1, cores=16, job_extra_directives=["--gres=gpu:40gb:1"]
)

client = Client(cluster)

client.submit(client_test, "input1", "input2", test=True).result()

KeyboardInterrupt: 

cluster = SLURMCluster(
    memory="128g", processes=1, cores=16, job_extra_directives=["--gres=gpu:3g.20gb:1", "--qos=lion"]
)

cluster.scale(1)
client = Client(cluster)

/apps/mambaforge/envs/dsks_2023.10/lib/python3.10/site-packages/distributed/node.py:182: UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 46635 instead
  warnings.warn(