Distributed jobs versus parallel programming

Matlab provides a capability called distributed jobs, which is really designed to be run from a personal computer. The Matlab running on the PC is, in their terminology, the client. The design is that you work at your PC, you set up your program so that there are fully independent steps to be computed, and then you use the distributed jobs capability to contact the cluster and submit each of those steps to the cluster as an independent job. Matlab provides functions for you to query the cluster for job status, collect the results, and then collate and continue your work.

Why ARC discourages distributed jobs

We discourage the use of distributed jobs because we do not allow your PC to connect directly to the cluster to submit jobs. The distributed jobs scheme is basically using Matlab in place of the cluster’s resource manager directly, which simply adds an additional layer of complexity when attempting to isolate any problems and complicates recovery if only a few of the jobs fails to complete successfully.

There may be times when it is appropriate to use, so we do provide this capability, but we do not recommend its use.