A brief intro to Resource Broker

The current version of Resource Broker (Version 2 Beta 1) is being developed at the same time as GSub Beta 2 by the same team. Therefore, there are a lot of similarities between the two tools, both in terms of the external look-and-feel and the internal implementation. So, if a user learns to use any one of the two, he/she almost automatically knows the other.

Compared to GSub, the Resource Broker is easier to configure, as there is no need to set any certificate for SSH job submission.

When a user first opens the Resource Broker, a blank window comes up as there is no application configured yet. The User can start to configure his/her application by clicking the “Preference” menu item under the File menu.

In the “Application Configuration” window, the user first should go to File-> Config Server to set up a server he/she wants to connect to.

Click the “New” button in the Server Configuration window.

Enter a hostname or “Localhost” for the local machine (For more advanced user: set hostname to 127.0.0.1 does not work for this case, as there is no underline SSH connection for local machine. Jobs are generated through the EXEC system call. However, if there is a SSH server currently running on the localhost, 127.0.0.1 might work. However, we have not tested this case.) For any remote machine, the user needs to provide all the common login information, i.e., username, password, authentication method, etc. We currently do not provide a method to determine the valid authentication method for a remote machine, user could find this information by inquiring the system admin or determine it by using some third party program, e.g., Putty. If user wants to configure PBS execution for this server, he/she needs to 1) check the PBS box in the Job Manager, 2) Set queue names for that server, and 3) set the path to QSub. As for the queue name part, there is no need to give all queue names available on the sever. Providing queues the user interested to use is enough.

Once a server is configured, the user should be able to see it in the Server drop-down box in the Application Configuration window. Now the user can start to configure an application for this server by clicking “Add Application” under the File menu. For an application, the user needs to provide a script and an execution directory. For Localhost, the Resource Broker takes regular Windows command line script. The only exception is, in general, there is only one binary execution allowed in this script. Put in another word, user are free to use all default DOS command like cd, dir, mkdir, del, set, etc., as many times as he/she wants, as those commands are interpreted by the command.com shell. However, only one “.exe” file execution is allowed in this script as anything after invoking this executable will be ignored by the shell. For remote host, we have tested scripts running under C-Shell extensively. In general, any regular C-Shell scripts are accepted and run smoothly from the Resource Broker if it does not set shell variables from another shell script, i.e., no “source” command is used. This only adds some amount inconvenience to the user as all parameters set in the separated script can be set in the same script without any technical barrier. The $EXEDIR and $JOBDIR work exactly the same way as they are in GSub. The $JOBDIR is always set to be the UNIX system time at the job submission time and user define $EXEDIR to be a place where he/she wants to execute this application. The actual application execution happens at $EXEDIR/$JOBDIR. For example, if the user set his $EXEDIR to be “/home/USERNAME” and submitted a job at Oct. 16, 2006 3:25:XX MT, then this job runs at “/home/USERNAME/1161033934703”.

Just like in GSub, we also have a variable-value mapping mechanism built-in for the Resource Broker, i.e. user can set variable names to represent values which can only be defined at submission time. User is free to use three types of variables in scripting: input, output and argument. Input variables take the format of $INNUM, where NUM is a numerical value from 0 to 9. Output variables take the format of $OUTNUM, and argument variables take the format of $ARGNUM. User can give a short description for any of the variables he/she defined to help him/her to distinguish between different variables. To summarize, user first clicks “Add Application” in the file menu to start a new application, he then configs the script he needs to run the job, and then clicks the “Config Parameter” button to set parameters, at last, he needs to define the $EXEDIR variable to be a directory where he wants to execute his job.

Once the user finishes config an application, he is can run it from the Execution Window (The main/first window in Resource Broker). The user first needs to choose an application he wants to run under the Application drop-down box. Then all servers which has this application configured show up in the “Resource Table”. In the table, user can select the server and running mode, i.e., Interactive or PBS, if the user wants to do a PBS submission, he also needs to select a queue name for his submission. Based on the application user selected, he may need to select input file(s) for the run. This can be done through selecting files using the CSE-Online Global File Explore (GFE) by clicking the “Browse” button on the right panel. After these steps, the user is ready to click the “Submit Job” button. If the job is successfully submitted, the user sees a new entry appears in the Job Status window where all basic information about this job is listed. At the bottom of the execution window, there are three buttons: Update Job State, Output and Remove Job. User can use these features by first select a row in the Job Status table and then click the corresponding button.

For users who travel a lot, he can carry around the job list he submitted with him by save the job list into file. When he moves to a new computer which has CSE-Online installed, he can then load his job list in to the Resource Broker running on that machine and manages jobs he submitted before. This feature is implemented by clicking the “Save Task List” and “Load Task List” from the File menu. There is no need to re-config any of his/her applications on the new machine, only necessary server authentication method is required.

Configuration Sharing

Knowing the difficulty of composing a job script for non-computer scientists, Resource Broker provides two script sharing mechanism for users to share their job scripts: share-by-file and share-online. In the share-by-file mode, one user composes a job script, exports it to a file and hands it to another user. In the share-online mode, authorized users, e.g., system administrators, upload job scripts to the CSE-Online application configuration repository. Other users can then download them for their own use.

To use the share-online, click the "Import Application Configuration Online" from the "File" menu. Then select an application to import.

Step by Step Examples

Example 1:

 Config new server and application to run "ps -ef" command and get the result ResourceBrokerExample1.