drmaa Package

A python package for DRM job submission and control.

This package is an implementation of the DRMAA 1.0 Python language binding specification (http://www.ogf.org/documents/GFD.143.pdf). The source is hosted on GitHub: https://github.com/pygridtools/drmaa-python Releases are available from PyPi: https://pypi.python.org/pypi/drmaa Documentation is hosted on ReadTheDocs: http://drmaa-python.readthedocs.org/

author:Enrico Sirola (enrico.sirola@statpro.com) author:Dan Blanchard (dan.blanchard@gmail.com)

class drmaa.JobInfo(jobId, hasExited, hasSignal, terminatedSignal, hasCoreDump, wasAborted, exitStatus, resourceUsage)

Bases: tuple

exitStatus

Alias for field number 6

hasCoreDump

Alias for field number 4

hasExited

Alias for field number 1

hasSignal

Alias for field number 2

jobId

Alias for field number 0

resourceUsage

Alias for field number 7

terminatedSignal

Alias for field number 3

wasAborted

Alias for field number 5

class drmaa.JobTemplate(**kwargs)

Bases: object

A job to be submitted to the DRM.

HOME_DIRECTORY = u'$drmaa_hd_ph$'

Home directory placeholder.

PARAMETRIC_INDEX = u'$drmaa_incr_ph$'

Parametric index (for job arrays / bulk jobs) placeholder.

WORKING_DIRECTORY = u'$drmaa_wd_ph$'

Working directory placeholder.

args

The job’s command argument list.

attributeNames

The list of supported DRMAA scalar attribute names.

This is apparently useless now, and should probably substituted by the list of attribute names of the JobTemplate instances.

blockEmail = False

False if this job should send an email, True otherwise.

deadlineTime = u''

The job deadline time, a partial timestamp string.

delete()

Deallocate the underlying DRMAA job template.

email

email addresses to whom send job completion info.

errorPath = u''

The path to a file representing job’s stderr.

hardRunDurationLimit

A DRMAA attribute, to be managed with scalar C DRMAA attribute management functions.

hardWallclockTimeLimit

‘Hard’ Wallclock time limit, in seconds.

The job will be killed by the DRM if it takes more than ‘hardWallclockTimeLimit’ to complete.

inputPath = u''

The path to a file representing job’s stdin.

jobCategory = u''

The job category.

jobEnvironment

The job’s environment dict.

jobName = u''

The job Name.

jobSubmissionState = u''

The job status.

joinFiles = False

True if stdin and stdout should be merged, False otherwise.

nativeSpecification = u''

A (DRM-dependant) opaque string to be passed to the DRM representing other directives.

outputPath = u''

The path to a file representing job’s stdout.

remoteCommand = u''

The command to be executed.

softRunDurationLimit

A DRMAA attribute, to be managed with scalar C DRMAA attribute management functions.

softWallclockTimeLimit

‘Soft’ Wallclock time limit, in seconds.

The job will be signaled by the DRM if it takes more than ‘hardWallclockTimeLimit’ to complete.

startTime = u''

The job start time, a partial timestamp string.

transferFiles = u''

True if file transfer should be enabled, False otherwise.

This option might require specific DRM configuration (it does on SGE).

workingDirectory = u''

The job working directory.

class drmaa.Session(contactString=None)

Bases: object

The DRMAA Session.

This class is the entry point for communicating with the DRM system

JOB_IDS_SESSION_ALL = 'DRMAA_JOB_IDS_SESSION_ALL'

JOB_IDS_SESSION_ANY = 'DRMAA_JOB_IDS_SESSION_ANY'

TIMEOUT_NO_WAIT = 0

TIMEOUT_WAIT_FOREVER = -1

contact = u''

a comma delimited string list containing the contact strings available from the default DRMAA implementation, one element per DRM system available. If called after initialize(), this method returns the contact String for the DRM system to which the session is attached. The returned strings are implementation dependent.

static control(jobId, operation)

Used to hold, release, suspend, resume, or kill the job identified by jobId.

Parameters:

jobId : string

if jobId is Session.JOB_IDS_SESSION_ALL then this routine acts on all jobs submitted during this DRMAA session up to the moment control() is called. The legal values for action and their meanings are

operation : string

possible values are:

JobControlAction.SUSPEND

stop the job

JobControlAction.RESUME

(re)start the job

JobControlAction.HOLD

put the job on-hold

JobControlAction.RELEASE

release the hold on the job

JobControlAction.TERMINATE

kill the job

To avoid thread races in multithreaded applications, the DRMAA implementation user should explicitly synchronize this call with any other job submission calls or control calls that may change the number of remote jobs.

This method returns once the action has been acknowledged by the DRM system, but does not necessarily wait until the action has been completed. Some DRMAA implementations may allow this method to be used to control jobs submitted external to the DRMAA session, such as jobs submitted by other DRMAA session in other DRMAA implementations or jobs submitted via native utilities.

static createJobTemplate()

Allocates a new job template.

The job template is used to set the environment for jobs to be submitted. Once the job template has been created, it should also be deleted (via deleteJobTemplate()) when no longer needed. Failure to do so may result in a memory leak.

static deleteJobTemplate(jobTemplate)

Deallocate a job template.

Parameters:

jobTemplate : JobTemplate

the job temptare to be deleted

This routine has no effect on running jobs.

drmaaImplementation = u''

If called before initialize(), this method returns a comma delimited list of DRMAA implementations, one element for each DRMAA implementation provided. If called after initialize(), this method returns the selected DRMAA implementation. The returned String is implementation dependent and may contain the DRM system as a component.

drmsInfo = u''

If called before initialize(), this method returns a comma delimited list of DRM systems, one element per DRM system implementation provided. If called after initialize(), this method returns the selected DRM system. The returned String is implementation dependent.

static exit()

Used to disengage from DRM.

This routine ends the current DRMAA session but doesn’t affect any jobs (e.g., queued and running jobs remain queued and running). exit() should be called only once, by only one of the threads. Additional calls to exit() beyond the first will throw a NoActiveSessionException.

static initialize(contactString=None)

Used to initialize a DRMAA session for use.

Parameters:

contactString : string or None

implementation-dependent string that may be used to specify which DRM system to use

This method must be called before any other DRMAA calls. If contactString is None, the default DRM system is used, provided there is only one DRMAA implementation available. If there is more than one DRMAA implementation available, initialize() throws a NoDefaultContactStringSelectedException. initialize() should be called only once, by only one of the threads. The main thread is recommended. A call to initialize() by another thread or additional calls to initialize() by the same thread with throw a SessionAlreadyActiveException.

static jobStatus(jobId)

returns the program status of the job identified by jobId.

The possible values returned from this method are:

• JobState.UNDETERMINED: process status cannot be determined,
• JobState.QUEUED_ACTIVE: job is queued and active,
• JobState.SYSTEM_ON_HOLD: job is queued and in system hold,
• JobState.USER_ON_HOLD: job is queued and in user hold,

• JobState.USER_SYSTEM_ON_HOLD: job is queued and in user and

  system hold,

• JobState.RUNNING: job is running,
• JobState.SYSTEM_SUSPENDED: job is system suspended,
• JobState.USER_SUSPENDED: job is user suspended,
• JobState.DONE: job finished normally, and
• JobState.FAILED: job finished, but failed.

The DRMAA implementation should always get the status of the job from the DRM system unless the status has already been determined to be FAILED or DONE and the status has been successfully cached. Terminated jobs return a FAILED status.

static runBulkJobs(jobTemplate, beginIndex, endIndex, step)

Submit a set of parametric jobs, each with attributes defined in the job template.

Parameters:

jobTemplate : JobTemplate

the template representng jobs to be run

beginIndex : int

index of the first job

endIndex : int

index of the last job

step : int

the step between job ids

The returned job identifiers are Strings identical to those returned from the underlying DRM system. The JobTemplate class defines a JobTemplate.PARAMETRIC_INDEX placeholder for use in specifying paths. This placeholder is used to represent the individual identifiers of the tasks submitted through this method.

static runJob(jobTemplate)

Submit a job with attributes defined in the job template.

Parameters:

jobTemplate : JobTemplate

the template representing the job to be run

The returned job identifier is a String identical to that returned from the underlying DRM system.

static synchronize(jobIds, timeout=-1, dispose=False)

Waits until all jobs specified by jobList have finished execution.

Parameters:

jobIds

If jobIds contains Session.JOB_IDS_SESSION_ALL, then this method waits for all jobs submitted during this DRMAA session up to the moment synchronize() is called

timeout : int

maximum time (in seconds) to be waited for the completion of a job.

The value Session.TIMEOUT_WAIT_FOREVER may be specified to wait indefinitely for a result. The value Session.TIMEOUT_NO_WAIT may be specified to return immediately if no result is available.

dispose : bool

specifies how to treat the reaping of the remote job’s internal data record, which includes a record of the job’s consumption of system resources during its execution and other statistical information. If set to True, the DRM will dispose of the job’s data record at the end of the synchronize() call. If set to False, the data record will be left for future access via the wait() method. It is the responsibility of the application to make sure that either synchronize() or `wait()`is called for every job. Not doing so creates a memory leak. Note that calling synchronize() with dispose set to true flushes all accounting information for all jobs in the list.

To avoid thread race conditions in multithreaded applications, the DRMAA implementation user should explicitly synchronize this call with any other job submission calls or control calls that may change the number of remote jobs.

If the call exits before the timeout has elapsed, all the jobs have been waited on or there was an interrupt. If the invocation exits on timeout, an ExitTimeoutException is thrown. The caller should check system time before and after this call in order to be sure of how much time has passed.

version = Version(major=10L, minor=10L)

a Version object containing the major and minor version numbers of the DRMAA library. For DRMAA 1.0, major is 1 and minor is 0.

static wait(jobId, timeout=-1)

Wait for a job with jobId to finish execution or fail.

Parameters:

jobId : str

The job id to wait completion for.

If the special string, Session.JOB_IDS_SESSION_ANY, is provided as the jobId, this routine will wait for any job from the session

timeout : float

The timeout value is used to specify the desired behavior when a result is not immediately available.

The value Session.TIMEOUT_WAIT_FOREVER may be specified to wait indefinitely for a result. The value Session.TIMEOUT_NO_WAIT may be specified to return immediately if no result is available. Alternatively, a number of seconds may be specified to indicate how long to wait for a result to become available

This routine is modeled on the wait3 POSIX routine. If the call exits before timeout, either the job has been waited on successfully or there was an interrupt. If the invocation exits on timeout, an ExitTimeoutException is thrown. The caller should check system time before and after this call in order to be sure how much time has passed. The routine reaps job data records on a successful call, so any subsequent calls to wait() will fail, throwing an InvalidJobException, meaning that the job’s data record has been already reaped. This exception is the same as if the job were unknown. (The only case where wait() can be successfully called on a single job more than once is when the previous call to wait() timed out before the job finished.)

exception drmaa.AlreadyActiveSessionException

Bases: drmaa.errors.DrmaaException

exception drmaa.AuthorizationException

Bases: drmaa.errors.DrmaaException

exception drmaa.ConflictingAttributeValuesException

Bases: drmaa.errors.DrmaaException, exceptions.AttributeError

exception drmaa.DefaultContactStringException

Bases: drmaa.errors.DrmaaException

exception drmaa.DeniedByDrmException

Bases: drmaa.errors.DrmaaException

exception drmaa.DrmCommunicationException

Bases: drmaa.errors.DrmaaException

exception drmaa.DrmsExitException

Bases: drmaa.errors.DrmaaException

exception drmaa.DrmsInitException

Bases: drmaa.errors.DrmaaException

exception drmaa.ExitTimeoutException

Bases: drmaa.errors.DrmaaException

exception drmaa.HoldInconsistentStateException

Bases: drmaa.errors.DrmaaException

exception drmaa.IllegalStateException

Bases: drmaa.errors.DrmaaException

exception drmaa.InternalException

Bases: drmaa.errors.DrmaaException

exception drmaa.InvalidAttributeFormatException

Bases: drmaa.errors.DrmaaException, exceptions.AttributeError

exception drmaa.InvalidContactStringException

Bases: drmaa.errors.DrmaaException

exception drmaa.InvalidJobException

Bases: drmaa.errors.DrmaaException

exception drmaa.InvalidJobTemplateException

Bases: drmaa.errors.DrmaaException

exception drmaa.NoActiveSessionException

Bases: drmaa.errors.DrmaaException

exception drmaa.NoDefaultContactStringSelectedException

Bases: drmaa.errors.DrmaaException

exception drmaa.ReleaseInconsistentStateException

Bases: drmaa.errors.DrmaaException

exception drmaa.ResumeInconsistentStateException

Bases: drmaa.errors.DrmaaException

exception drmaa.SuspendInconsistentStateException

Bases: drmaa.errors.DrmaaException

exception drmaa.TryLaterException

Bases: drmaa.errors.DrmaaException

exception drmaa.UnsupportedAttributeException

Bases: drmaa.errors.DrmaaException, exceptions.AttributeError

exception drmaa.InvalidArgumentException

Bases: drmaa.errors.DrmaaException, exceptions.AttributeError

exception drmaa.InvalidAttributeValueException

Bases: drmaa.errors.DrmaaException, exceptions.AttributeError

exception drmaa.OutOfMemoryException

Bases: drmaa.errors.DrmaaException, exceptions.MemoryError

drmaa.control_action_to_string(code)

drmaa.job_state(code)

class drmaa.JobControlAction

Bases: object

HOLD = u'hold'

RELEASE = u'release'

RESUME = u'resume'

SUSPEND = u'suspend'

TERMINATE = u'terminate'

class drmaa.JobState

Bases: object

DONE = u'done'

FAILED = u'failed'

QUEUED_ACTIVE = u'queued_active'

RUNNING = u'running'

SYSTEM_ON_HOLD = u'system_on_hold'

SYSTEM_SUSPENDED = u'system_suspended'

UNDETERMINED = u'undetermined'

USER_ON_HOLD = u'user_on_hold'

USER_SUSPENDED = u'user_suspended'

USER_SYSTEM_ON_HOLD = u'user_system_on_hold'

USER_SYSTEM_SUSPENDED = u'user_system_suspended'

class drmaa.JobSubmissionState

Bases: object

ACTIVE_STATE = u'drmaa_active'

HOLD_STATE = u'drmaa_hold'

drmaa.status_to_string(status)

drmaa.string_to_control_action(operation)

drmaa.submission_state(code)