Xenon



For your information, we have GoogleAnalytics enabled for this tutorial. This is to help show our funders that the work we do is useful to others.




Getting started

On your system, start VirtualBox.

All tutorials at RSE2017 use the same virtual machine. In case you don’t have a copy of the virtual machine, you can download it from GoogleDrive here (5.5 Gb). After the download finishes, click File in VirtualBox, then Import appliance, then select the file you downloaded.

During the import, you’ll see an initialization wizard. Make sure that the virtual machine is configured with two CPUs.

Start the Fedora virtual machine and log in with password tutorial.

Once the system has booted, Click Activities and then start both a terminal and Firefox by clicking their respective icons. Use Firefox to navigate to the tutorial text at http://xenonrse2017.readthedocs.io.

In the terminal, change directory to /home/tutorial/xenon and confirm that the xenon command line interface program can be found on the system:

cd xenon
xenon --help



Interacting with filesystems

Essentially, xenon can be used to manipulate files and to interact with schedulers, where either one can be local or remote. Let’s start simple and see if we can do something with local files. First, check its help:

xenon filesystem --help

The usage line suggests we need to pick one from {file,ftp,sftp,webdav}. Again, choose what seems to be the simplest option (file), and again, check its help.

xenon filesystem file --help

xenon filesystem file‘s usage line seems to suggest that I need to pick one from {copy,list,mkdir,remove,rename}. Simplest one is probably list, so:

xenon filesystem file list --help

So we need a path as final argument.

In case you hadn’t noticed the pattern, stringing together any number of xenon subcommands and appending --help to it will get you help on the particular combination of subcommands you supplied.

The focus of this tutorial is on using Xenon’s command line interface, but be aware that other programming interfaces are available through gRPC. Where relevant, we have included equivalent code snippets, written in Java and Python, as separate tabs.

Let’s try listing the contents of /home/tutorial/xenon.

xenon filesystem file list /home/tutorial/xenon
import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;
import nl.esciencecenter.xenon.filesystems.PathAttributes;

public class DirectoryListing {

    public static void main(String[] args) throws Exception {

        String adaptor = "file";
        FileSystem filesystem = FileSystem.create(adaptor);

        Path dir = new Path("/home/tutorial/xenon");
        boolean recursive = false;

        Iterable<PathAttributes> listing = filesystem.list(dir, recursive);

        for (PathAttributes elem : listing) {
            if (!elem.isHidden()) {
                System.out.println(elem.getPath());
            }
        }

    }
}
import xenon
from xenon import Path, FileSystem

xenon.init()

filesystem = FileSystem.create(adaptor='file')
path = Path("/home/tutorial/xenon")

listing = filesystem.list(path, recursive=False)

for entry in listing:
    if not entry.path.is_hidden():
        print(entry.path)

filesystem.close()

The result should be more or less the same as that of ls -1.

xenon filesystem file list has a few options that let you specify the details of the list operation, e.g. --hidden

xenon filesystem file list --hidden /home/tutorial/xenon
import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;
import nl.esciencecenter.xenon.filesystems.PathAttributes;

public class DirectoryListingShowHidden {

    public static void main(String[] args) throws Exception {

        String adaptor = "file";
        FileSystem filesystem = FileSystem.create(adaptor);

        Path dir = new Path("/home/tutorial/xenon");
        boolean recursive = false;

        Iterable<PathAttributes> listing = filesystem.list(dir, recursive);

        for (PathAttributes elem : listing) {
            System.out.println(elem.getPath());
        }
    }
}
import xenon
from xenon import Path, FileSystem

xenon.init()

filesystem = FileSystem.create(adaptor='file')
path = Path("/home/tutorial/xenon")

listing = filesystem.list(path, recursive=False)

for entry in listing:
    print(entry.path)

filesystem.close()

and --recursive

xenon filesystem file list --recursive /home/tutorial/xenon
import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;
import nl.esciencecenter.xenon.filesystems.PathAttributes;

public class DirectoryListingRecursive {

    public static void main(String[] args) throws Exception {

        String adaptor = "file";
        FileSystem filesystem = FileSystem.create(adaptor);

        Path dir = new Path("/home/tutorial/xenon");
        boolean recursive = true;

        Iterable<PathAttributes> listing = filesystem.list(dir, recursive);

        for (PathAttributes elem : listing) {
            if (!elem.isHidden()) {
                System.out.println(elem.getPath());
            }
        }
    }
}
import xenon
from xenon import Path, FileSystem

xenon.init()

filesystem = FileSystem.create(adaptor='file')
path = Path("/home/tutorial/xenon")

listing = filesystem.list(path, recursive=True)

for entry in listing:
    if not entry.path.is_hidden():
        print(entry.path)

filesystem.close()

Now let’s create a file and try to use xenon to copy it:

cd /home/tutorial/xenon
echo 'some content' > thefile.txt

Check the relevant help

xenon filesystem file --help
xenon filesystem file copy --help

So, the copy subcommand takes a source path and a target path:

xenon filesystem file copy /home/tutorial/xenon/thefile.txt /home/tutorial/xenon/thefile.bak
import nl.esciencecenter.xenon.filesystems.CopyMode;
import nl.esciencecenter.xenon.filesystems.CopyStatus;
import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;

public class CopyFileLocalToLocalAbsolutePaths {

    public static void main(String[] args) throws Exception {

        // use the local file system adaptor to create a file system representation
        String adaptor = "file";
        FileSystem filesystem = FileSystem.create(adaptor);

        // create Paths for the source and destination files, using absolute paths
        Path sourceFile = new Path("/home/tutorial/xenon/thefile.txt");
        Path destFile = new Path("/home/tutorial/xenon/thefile.bak");

        // create the destination file only if the destination path doesn't exist yet
        CopyMode mode = CopyMode.CREATE;
        boolean recursive = false;

        // perform the copy and wait 1000 ms for the successful or otherwise
        // completion of the operation
        String copyId = filesystem.copy(sourceFile, filesystem, destFile, mode, recursive);
        long timeoutMilliSecs = 1000;
        CopyStatus copyStatus = filesystem.waitUntilDone(copyId, timeoutMilliSecs);

        // print any exceptions
        if (copyStatus.getException() != null) {
            System.out.println(copyStatus.getException().getMessage());
        }
    }
}
import xenon
from xenon import Path, FileSystem, CopyRequest, CopyStatus

xenon.init()

filesystem = FileSystem.create(adaptor='file')
source_file = Path("/home/daisycutter/tmp/home/tutorial/xenon/thefile.txt")
dest_file = Path("/home/daisycutter/tmp/home/tutorial/xenon/thefile.bak")

# start the copy operation; no recursion, we're just copying a file
copy_id = filesystem.copy(source_file, filesystem, dest_file, mode=CopyRequest.CREATE, recursive=False)

# wait this many milliseconds for copy operations to complete
WAIT_DURATION = 1000 * 60 * 10

# wait for the copy operation to complete (successfully or otherwise)
copy_status = filesystem.wait_until_done(copy_id, WAIT_DURATION)
assert copy_status.done

# rethrow the Exception if we got one
assert copy_status.error_type == CopyStatus.NONE, copy_status.error_message

filesystem.close()

Note that the source path may be standard input, and that the target path may be standard output:

# read from stdin:
cat thefile.txt | xenon filesystem file copy - mystdin.txt

# write to stdout:
xenon filesystem file copy thefile.txt - 1> mystdout.txt

xenon filesystem file has a few more subcommands, namely mkdir, rename and remove. You can experiment a bit more with those or move on to the next section.




Interacting with schedulers

Now let’s see if we can use schedulers, starting with SLURM. For this part, we need access to a machine that is running SLURM. To avoid problems related to network connectivity, we won’t try to connect to a physically remote SLURM machine, but instead, we’ll use a dockerized SLURM installation. This way, we can mimic whatever infrastructure we need. The setup will thus be something like this:

babushka


A copy of the SLURM Docker image (nlesc/xenon-slurm:17) has been included in the RSE 2017 virtual machine. Bring it up with:

docker run --detach --publish 10022:22 --hostname slurm17 nlesc/xenon-slurm:17

Use docker ps to check the state of the container

docker ps

Once the status is healthy, see if we can ssh into it on port 10022 as user xenon with password javagat:

ssh -p 10022 xenon@localhost

# if that works, exit again
exit

Check the help to see how the slurm subcommand works:

xenon scheduler slurm --help

Let’s first ask what queues the SLURM scheduler has. For this, we need to specify a location, otherwise xenon does not know who to ask for the list of queues. According to the help, LOCATION is any location format supported by ssh or local scheduler. Our dockerized SLURM machine is reachable as localhost:10022. We’ll also need to provide a --username and --password for that location, as follows:

xenon scheduler slurm --location localhost:10022 --username xenon --password javagat queues
# returns:
Available queues: mypartition, otherpartition
Default queue: mypartition
import nl.esciencecenter.xenon.credentials.PasswordCredential;
import nl.esciencecenter.xenon.schedulers.Scheduler;

public class SlurmQueuesGetter {

    public static void main(String[] args) throws Exception {

        String adaptor = "slurm";
        String location = "localhost:10022";

        // make the password credential for user 'xenon'
        String username = "xenon";
        char[] password = "javagat".toCharArray();
        PasswordCredential credential = new PasswordCredential(username, password);

        // create the SLURM scheduler
        Scheduler scheduler = Scheduler.create(adaptor, location, credential);

        // ask SLURM for its queues
        System.out.println("Available queues (starred queue is default):");
        for (String queueName : scheduler.getQueueNames()) {
            if (queueName == scheduler.getDefaultQueueName()) {
                System.out.println(queueName + "*");
            } else {
                System.out.println(queueName);
            }
        }
    }
}
import xenon
from xenon import Scheduler, PasswordCredential

xenon.init()

credential = PasswordCredential(username='xenon',
                                password='javagat')

scheduler = Scheduler.create(adaptor='slurm',
                             location='localhost:10022',
                             password_credential=credential)

default_queue = scheduler.get_default_queue_name()

print("Available queues (starred queue is default):")
for queue_name in scheduler.get_queue_names():
    if queue_name == default_queue:
        print("{}*".format(queue_name))
    else:
        print(queue_name)

scheduler.close()

In case you are reluctant to type plaintext passwords on the command line, for example because of logging in ~/.bash_history, know that you can supply passwords from a file, as follows:

xenon scheduler slurm --location localhost:10022 --username xenon --password @password.txt queues

in which the file password.txt should contain the password. Since everything about the user xenon is public knowledge anyway, such security precautions are not needed for this tutorial, so we’ll just continue to use the --password PASSWORD syntax.

Besides queues, other slurm subcommands are exec, submit, list, remove, and wait. Let’s try to have xenon ask SLURM for its list of jobs in each queue, as follows:

xenon scheduler slurm --location localhost:10022 --username xenon --password javagat list
# should work, but we don't have any jobs yet

Now, let’s try to submit a job using slurm submit. Its usage string suggests that we need to provide (the path of) an executable. Note that the executable should be present inside the container when SLURM starts its execution. For the moment, we’ll use /bin/hostname as the executable. It should return the hostname slurm17 of the Docker container, or whatever hostname you specified for it when you ran the docker run command earlier:

# check the slurm submit help for correct syntax
xenon scheduler slurm submit --help

# let xenon submit a job with /bin/hostname as executable
xenon scheduler slurm --location localhost:10022 --username xenon --password javagat \
submit /bin/hostname

# add --stdout to the submit job to capture its standard out so we know it worked:
xenon scheduler slurm --location localhost:10022 --username xenon --password javagat \
submit --stdout hostname.stdout.txt /bin/hostname

# check to see if the output was written to file /home/xenon/hostname.stdout.txt
ssh -p 10022 xenon@localhost ls -l
# see what's in it
ssh -p 10022 xenon@localhost cat hostname.stdout.txt

Below are a few more examples of slurm submit:

# executables that take options prefixed with '-' need special syntax, e.g. 'ls -la'
xenon scheduler slurm --location localhost:10022 --username xenon --password javagat \
submit --stdout /home/xenon/ls.stdout.txt ls -- -la

# check to see if the output was written to file /home/xenon/ls.stdout.txt
ssh -p 10022 xenon@localhost ls -l
# see what's in it
ssh -p 10022 xenon@localhost cat ls.stdout.txt

# submit an 'env' job with environment variable MYKEY, and capture standard out so we know it worked
xenon scheduler slurm --location localhost:10022 --username xenon --password javagat \
submit --stdout env.stdout.txt --env MYKEY=myvalue env

# check to see if the output from 'env' was written to file /home/xenon/env.stdout.txt
ssh -p 10022 xenon@localhost ls -l
# see what's in it
ssh -p 10022 xenon@localhost cat env.stdout.txt



Combining filesystems and schedulers

So far, we’ve used xenon to manipulate files on the local filesystem, and to run system executables on the remote machine. In typical usage, however, you would use xenon to run executables or scripts of your own, which means that we need to upload such files from the local system to the remote system.

A typical workflow may thus look like this:

  1. upload input file(s)
  2. submit job
  3. download generated output file(s)

Use an editor to create a file sleep.sh with the following contents (the RSE 2017 virtual machine comes with gedit, or you can install an other editor from the repositories if you like):

#!/usr/bin/env bash
echo `date`': went to bed'
sleep $1
echo `date`': woke up'

You can test if your file is correct by:

# last argument is the sleep duration in seconds
bash sleep.sh 5

We need to upload sleep.sh to the remote machine. We can’t use xenon filesystem file like we did before, because we’re copying between file systems, so let’s look at what other options are available:

xenon filesystem --help

# let's try sftp protocol
xenon filesystem sftp --help

# we're interested in 'upload' for now
xenon filesystem sftp upload --help

We’ll also need to tell xenon what location we want to connect to, and what credentials to use. The SLURM Docker container we used before is accessible via SFTP using the same location, username and password as before, so let’s use that:

# step 1: upload input file(s)
xenon filesystem sftp --location localhost:10022 --username xenon --password javagat \
upload /home/tutorial/xenon/sleep.sh /home/xenon/sleep.sh
import nl.esciencecenter.xenon.credentials.PasswordCredential;
import nl.esciencecenter.xenon.filesystems.CopyMode;
import nl.esciencecenter.xenon.filesystems.CopyStatus;
import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;

public class UploadFileLocalToSftpAbsolutePaths {

    public static void main(String[] args) throws Exception {

        // use the local file system adaptor to create a file system representation
        String adaptorLocal = "file";
        FileSystem filesystemLocal = FileSystem.create(adaptorLocal);

        // define what file to upload
        Path fileLocal = new Path("/home/tutorial/xenon/sleep.sh");

        // use the sftp file system adaptor to create a file system representation; the remote
        // filesystem requires credentials to log in, so we'll have to create those too.
        String adaptorRemote = "sftp";
        String location = "localhost:10022";
        String username = "xenon";
        char[] password = "javagat".toCharArray();
        PasswordCredential credential = new PasswordCredential(username, password);
        FileSystem filesystemRemote = FileSystem.create(adaptorRemote, location, credential);

        // define which file to upload to
        Path fileRemote = new Path("/home/xenon/sleep.sh");

        // create the destination file only if the destination path doesn't exist yet
        CopyMode mode = CopyMode.CREATE;

        // no need to recurse, we're just downloading a file
        boolean recursive = false;

        // perform the copy/upload and wait 1000 ms for the successful or otherwise
        // completion of the operation
        String copyId = filesystemLocal.copy(fileLocal, filesystemRemote, fileRemote, mode, recursive);
        long timeoutMilliSecs = 1000;
        CopyStatus copyStatus = filesystemLocal.waitUntilDone(copyId, timeoutMilliSecs);

        // print any exceptions
        if (copyStatus.getException() != null) {
            System.out.println(copyStatus.getException().getMessage());
        } else {
            System.out.println("Done");
        }
    }
}
import xenon
from xenon import FileSystem, PasswordCredential, CopyRequest, Path, CopyStatus

xenon.init()

# use the local file system adaptor to create another file system representation
local_fs = FileSystem.create(adaptor='file')

# use the sftp file system adaptor to create a file system
# representation; the remote filesystem requires credentials to log in,
# so we'll have to create those too.
credential = PasswordCredential(username='xenon',
                                password='javagat')

remote_fs = FileSystem.create(adaptor='sftp',
                              location='localhost:10022',
                              password_credential=credential)

# define which file to upload
local_file = Path('/home/daisycutter/tmp/home/tutorial/xenon/sleep.sh')
remote_file = Path('/home/xenon/sleep.sh')

# create the destination file only if the destination path doesn't exist yet
mode = CopyRequest.CREATE

# no need to recurse, we're just uploading a file
recursive = False

# perform the copy/upload and wait 1000 ms for the successful or
# otherwise completion of the operation
copy_id = local_fs.copy(local_file, remote_fs, remote_file,
                        mode=mode, recursive=recursive)

copy_status = local_fs.wait_until_done(copy_id, timeout=1000)

assert copy_status.done

# rethrow the Exception if we got one
assert copy_status.error_type == CopyStatus.NONE, copy_status.error_message

# remember to close the FileSystem instances
remote_fs.close()
local_fs.close()

print('Done')

Now that the script is in place, we can submit a bash job using xenon scheduler slurm submit like before, taking the newly uploaded sleep.sh file as input to bash, and using a sleep duration of 60 seconds:

# step 2: submit job
xenon scheduler slurm --location localhost:10022 --username xenon --password javagat \
submit --stdout sleep.stdout.txt bash sleep.sh 60

# (should return an identifier for the job)

With the job running, let’s see if it shows up in any of the SLURM queues:

xenon scheduler slurm --location localhost:10022 --username xenon --password javagat list
# should have the job identifier in it that was printed on the command line
import nl.esciencecenter.xenon.credentials.PasswordCredential;
import nl.esciencecenter.xenon.schedulers.Scheduler;

public class SlurmJobListGetter {

    public static void main(String[] args) throws Exception {

        String adaptor = "slurm";
        String location = "localhost:10022";

        // make the password credential for user 'xenon'
        String username = "xenon";
        char[] password = "javagat".toCharArray();
        PasswordCredential credential = new PasswordCredential(username, password);

        // create the SLURM scheduler
        Scheduler scheduler = Scheduler.create(adaptor, location, credential);

        // ask SLURM for its queues and list the jobs in each
        for (String queueName : scheduler.getQueueNames()) {
            System.out.println("List of jobs in queue '" + queueName + "' for SLURM at " + location + "");
            String[] jobs = scheduler.getJobs(queueName);
            if (jobs.length == 0) {
                System.out.println("(No jobs)");
            } else {
                for (String job : jobs) {
                    System.out.println(job);
                }
            }
        }
    }
}
import xenon
from xenon import PasswordCredential, Scheduler

xenon.init()

credential = PasswordCredential(username='xenon',
                                password='javagat')

location = 'localhost:10022'

scheduler = Scheduler.create(adaptor='slurm',
                             location=location,
                             password_credential=credential)

for queue_name in scheduler.get_queue_names():
    print("List of jobs in queue {queue_name} for SLURM at {location}"
          .format(queue_name=queue_name, location=location))
    jobs = scheduler.get_jobs([queue_name])
    if not jobs:
        print("(No jobs)")
    else:
        for job in jobs:
            print("    {}".format(job))

scheduler.close()

When we submitted, we did not specify any queues, so the default queue mypartition was used:

xenon scheduler slurm --location localhost:10022 --username xenon --password javagat list --queue mypartition
# should have the job identifier in it that was printed on the command line

xenon scheduler slurm --location localhost:10022 --username xenon --password javagat list --queue otherpartition
# this queue is empty

With step 1 (upload) and step 2 (submit) covered, step 3 (download) remains:

# step 3: download generated output file(s)
xenon filesystem sftp --location localhost:10022 --username xenon --password javagat \
download /home/xenon/sleep.stdout.txt /home/tutorial/xenon/sleep.stdout.txt
import nl.esciencecenter.xenon.credentials.PasswordCredential;
import nl.esciencecenter.xenon.filesystems.CopyMode;
import nl.esciencecenter.xenon.filesystems.CopyStatus;
import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;

public class DownloadFileSftpToLocalAbsolutePaths {

    public static void main(String[] args) throws Exception {

        // use the sftp file system adaptor to create a file system representation; the remote
        // filesystem requires credentials to log in, so we'll have to create those too.
        String adaptorRemote = "sftp";
        String location = "localhost:10022";
        String username = "xenon";
        char[] password = "javagat".toCharArray();
        PasswordCredential credential = new PasswordCredential(username, password);
        FileSystem filesystemRemote = FileSystem.create(adaptorRemote, location, credential);

        // define which file to download
        Path fileRemote = new Path("/home/xenon/sleep.stdout.txt");

        // use the local file system adaptor to create a file system representation
        String adaptorLocal = "file";
        FileSystem filesystemLocal = FileSystem.create(adaptorLocal);

        // define what file to download to
        Path fileLocal = new Path("/home/tutorial/xenon/sleep.stdout.txt");

        // create the destination file only if the destination path doesn't exist yet
        CopyMode mode = CopyMode.CREATE;

        // no need to recurse, we're just downloading a file
        boolean recursive = false;

        // perform the copy/download and wait 1000 ms for the successful or otherwise
        // completion of the operation
        String copyId = filesystemRemote.copy(fileRemote, filesystemLocal, fileLocal, mode, recursive);
        long timeoutMilliSecs = 1000;
        CopyStatus copyStatus = filesystemRemote.waitUntilDone(copyId, timeoutMilliSecs);

        // print any exceptions
        if (copyStatus.getException() != null) {
            System.out.println(copyStatus.getException().getMessage());
        } else {
            System.out.println("Done");
        }
    }
}
import xenon
from xenon import FileSystem, PasswordCredential, CopyRequest, Path, CopyStatus

xenon.init()

# use the sftp file system adaptor to create a file system
# representation; the remote filesystem requires credentials to log in,
# so we'll have to create those too.
credential = PasswordCredential(username='xenon',
                                password='javagat')

remote_fs = FileSystem.create(adaptor='sftp',
                              location='localhost:10022',
                              password_credential=credential)

# use the local file system adaptor to create another file system representation
local_fs = FileSystem.create(adaptor='file')

# define which file to download
remote_file = Path('/home/xenon/sleep.stdout.txt')
local_file = Path('/home/daisycutter/tmp/home/tutorial/xenon/sleep.stdout.txt')

# create the destination file only if the destination path doesn't exist yet
mode = CopyRequest.CREATE

# no need to recurse, we're just uploading a file
recursive = False

# perform the copy/download and wait 1000 ms for the successful or
# otherwise completion of the operation
copy_id = remote_fs.copy(remote_file, local_fs, local_file,
                        mode=mode, recursive=recursive)

copy_status = remote_fs.wait_until_done(copy_id, timeout=1000)

assert copy_status.done

# rethrow the Exception if we got one
assert copy_status.error_type == CopyStatus.NONE, copy_status.error_message

# remember to close the FileSystem instances
remote_fs.close()
local_fs.close()

print('Done')

By this time you may start to consider putting those 3 commands in a script, as follows:

#!/usr/bin/env bash

# step 1: upload input file(s)
xenon filesystem sftp --location localhost:10022 --username xenon --password javagat \
upload /home/tutorial/xenon/sleep.sh /home/xenon/sleep.sh

# step 2: submit job
xenon scheduler slurm --location localhost:10022 --username xenon --password javagat \
submit --stdout sleep.stdout.txt bash sleep.sh 60

# step 3: download generated output file(s)
xenon filesystem sftp --location localhost:10022 --username xenon --password javagat \
download /home/xenon/sleep.stdout.txt /home/tutorial/xenon/sleep.stdout.txt
import nl.esciencecenter.xenon.credentials.PasswordCredential;
import nl.esciencecenter.xenon.filesystems.CopyMode;
import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;
import nl.esciencecenter.xenon.schedulers.JobDescription;
import nl.esciencecenter.xenon.schedulers.Scheduler;

public class AllTogetherNowWrong {

    public static void main(String[] args) throws Exception {

        /*
         * step 1: upload input file(s)
         */

        // create the local filesystem representation
        String fileAdaptorLocal = "file";
        FileSystem filesystemLocal = FileSystem.create(fileAdaptorLocal);

        // the remote system requires credentials, create them here:
        String username = "xenon";
        char[] password = "javagat".toCharArray();
        PasswordCredential credential = new PasswordCredential(username, password);

        // create the remote filesystem representation and specify the executable's path
        String fileAdaptorRemote = "sftp";
        String filesystemRemoteLocation = "localhost:10022";
        FileSystem filesystemRemote = FileSystem.create(fileAdaptorRemote,
                filesystemRemoteLocation, credential);

        {
            // specify the behavior in case the target path exists already
            CopyMode copyMode = CopyMode.CREATE;

            // no recursion, we're just copying a file
            boolean recursive = false;

            // specify the path of the script file on the local and on the remote
            Path fileLocal = new Path("/home/tutorial/xenon/sleep.sh");
            Path fileRemote = new Path("/home/xenon/sleep.sh");

            // start the copy operation
            filesystemLocal.copy(fileLocal, filesystemRemote, fileRemote, copyMode, recursive);

        }



        /*
         * step 2: submit job and capture its job identifier
         */

        // create the SLURM scheduler representation
        String schedulerAdaptor = "slurm";
        String schedulerLocation = "localhost:10022";
        Scheduler scheduler = Scheduler.create(schedulerAdaptor, schedulerLocation, credential);

        // compose the job description:
        JobDescription jobDescription = new JobDescription();
        jobDescription.setExecutable("bash");
        jobDescription.setArguments("sleep.sh", "60");
        jobDescription.setStdout("sleep.stdout.txt");

        scheduler.submitBatchJob(jobDescription);


        /*
         *  step 3: download generated output file(s)
         */

        {
            // specify the behavior in case the target path exists already
            CopyMode copyMode = CopyMode.CREATE;

            // no recursion, we're just copying a file
            boolean recursive = false;

            // specify the path of the stdout file on the remote and on the local machine
            Path fileRemote = new Path("/home/xenon/sleep.stdout.txt");
            Path fileLocal = new Path("/home/tutorial/xenon/sleep.stdout.txt");

            // start the copy operation
            filesystemRemote.copy(fileRemote, filesystemLocal, fileLocal, copyMode, recursive);

        }

        System.out.println("Done.");

    }
}

However, if you create the script above and run it, you’ll find that:

  1. Xenon complains about some destination paths already existing.
  2. The script finishes suspiciously quickly;

The first error is easily avoided by adding a --replace optional argument after upload and download, but that does not address the real issue: that of Xenon not waiting for the completion of our sleep job.

Not to worry though, we can use xenon scheduler slurm wait to wait for jobs to finish. In order to make this work, we do need to capture the identifier for a specific job, otherwise we don’t know what to wait for.

Adapt the script as follows and run it:

#!/usr/bin/env bash

# step 1: upload input file(s)
xenon filesystem sftp --location localhost:10022 --username xenon --password javagat \
upload --replace /home/tutorial/xenon/sleep.sh /home/xenon/sleep.sh

# step 2: submit job and capture its job identifier
JOBID=$(xenon scheduler slurm --location localhost:10022 --username xenon --password javagat \
submit --stdout sleep.stdout.txt bash sleep.sh 60)

# add blocking wait
xenon scheduler slurm --location localhost:10022 --username xenon --password javagat \
wait $JOBID

# step 3: download generated output file(s)
xenon filesystem sftp --location localhost:10022 --username xenon --password javagat \
download --replace /home/xenon/sleep.stdout.txt /home/tutorial/xenon/sleep.stdout.txt
import nl.esciencecenter.xenon.credentials.PasswordCredential;
import nl.esciencecenter.xenon.filesystems.CopyMode;
import nl.esciencecenter.xenon.filesystems.CopyStatus;
import nl.esciencecenter.xenon.filesystems.FileSystem;
import nl.esciencecenter.xenon.filesystems.Path;
import nl.esciencecenter.xenon.schedulers.JobDescription;
import nl.esciencecenter.xenon.schedulers.JobStatus;
import nl.esciencecenter.xenon.schedulers.Scheduler;

public class AllTogetherNow {

    public static void main(String[] args) throws Exception {

        /*
         * step 1: upload input file(s)
         */

        // create the local filesystem representation
        String fileAdaptorLocal = "file";
        FileSystem filesystemLocal = FileSystem.create(fileAdaptorLocal);

        // the remote system requires credentials, create them here:
        String username = "xenon";
        char[] password = "javagat".toCharArray();
        PasswordCredential credential = new PasswordCredential(username, password);

        // create the remote filesystem representation and specify the executable's path
        String fileAdaptorRemote = "sftp";
        String filesystemRemoteLocation = "localhost:10022";
        FileSystem filesystemRemote = FileSystem.create(fileAdaptorRemote,
                filesystemRemoteLocation, credential);

        // when waiting for jobs or copy operations to complete, wait indefinitely
        final long WAIT_INDEFINITELY = 0;

        {
            // specify the behavior in case the target path exists already
            CopyMode copyMode = CopyMode.REPLACE;

            // no recursion, we're just copying a file
            boolean recursive = false;

            // specify the path of the script file on the local and on the remote
            Path fileLocal = new Path("/home/tutorial/xenon/sleep.sh");
            Path fileRemote = new Path("/home/xenon/sleep.sh");

            // start the copy operation
            String copyId = filesystemLocal.copy(fileLocal, filesystemRemote, fileRemote, copyMode, recursive);

            // wait for the copy operation to complete (successfully or otherwise)
            CopyStatus status = filesystemLocal.waitUntilDone(copyId, WAIT_INDEFINITELY);

            // rethrow the Exception if we got one
            if (status.hasException()) {
                throw status.getException();
            }

        }



        /*
         * step 2: submit job and capture its job identifier
         */

        // create the SLURM scheduler representation
        String schedulerAdaptor = "slurm";
        String schedulerLocation = "localhost:10022";
        Scheduler scheduler = Scheduler.create(schedulerAdaptor, schedulerLocation, credential);

        // compose the job description:
        JobDescription jobDescription = new JobDescription();
        jobDescription.setExecutable("bash");
        jobDescription.setArguments("sleep.sh", "60");
        jobDescription.setStdout("sleep.stdout.txt");

        String jobId = scheduler.submitBatchJob(jobDescription);

        // wait for the job to finish before attempting to copy its output file(s)
        JobStatus jobStatus = scheduler.waitUntilDone(jobId, WAIT_INDEFINITELY);

        // rethrow the Exception if we got one
        if (jobStatus.hasException()) {
            throw jobStatus.getException();
        }


        /*
         *  step 3: download generated output file(s)
         */

        {
            // specify the behavior in case the target path exists already
            CopyMode copyMode = CopyMode.REPLACE;

            // no recursion, we're just copying a file
            boolean recursive = false;

            // specify the path of the stdout file on the remote and on the local machine
            Path fileRemote = new Path("/home/xenon/sleep.stdout.txt");
            Path fileLocal = new Path("/home/tutorial/xenon/sleep.stdout.txt");

            // start the copy operation
            String copyId = filesystemRemote.copy(fileRemote, filesystemLocal, fileLocal, copyMode, recursive);

            // wait for the copy operation to complete (successfully or otherwise)
            CopyStatus status = filesystemRemote.waitUntilDone(copyId, WAIT_INDEFINITELY);

            // rethrow the Exception if we got one
            if (status.hasException()) {
                throw status.getException();
            }

        }

        System.out.println("Done.");

    }
}
import xenon
from xenon import FileSystem, PasswordCredential, CopyRequest, Path, CopyStatus, Scheduler, JobDescription, JobStatus


def upload():

    # define which file to upload
    local_file = Path('/home/daisycutter/tmp/home/tutorial/xenon/sleep.sh')
    remote_file = Path('/home/xenon/sleep.sh')

    # create the destination file only if the destination path doesn't exist yet
    mode = CopyRequest.CREATE

    # no need to recurse, we're just uploading a file
    recursive = False

    # perform the copy/upload and wait 1000 ms for the successful or
    # otherwise completion of the operation
    copy_id = local_fs.copy(local_file, remote_fs, remote_file,
                            mode=mode, recursive=recursive)

    copy_status = local_fs.wait_until_done(copy_id, timeout=5000)

    assert copy_status.done

    # rethrow the Exception if we got one
    assert copy_status.error_type == CopyStatus.NONE, copy_status.error_message

    print('Done uploading.')


def submit():

    description = JobDescription(executable='bash',
                                 arguments=['sleep.sh', '60'],
                                 stdout='sleep.stdout.txt')

    scheduler = Scheduler.create(adaptor='slurm',
                                 location='localhost:10022',
                                 password_credential=credential)

    job_id = scheduler.submit_batch_job(description)

    print('Done submitting.')

    # wait for the job to finish before attempting to copy its output file(s)
    job_status = scheduler.wait_until_done(job_id, 10*60*1000)

    assert job_status.done

    # rethrow the Exception if we got one
    assert job_status.error_type == JobStatus.NONE, job_status.error_message

    print('Done executing on the remote.')

    # make sure to synchronize the remote filesystem
    job_id = scheduler.submit_batch_job(JobDescription(executable='sync'))
    scheduler.wait_until_done(job_id)

    scheduler.close()


def download():

    # define which file to download
    remote_file = Path('/home/xenon/sleep.stdout.txt')
    local_file = Path('/home/daisycutter/tmp/home/tutorial/xenon/sleep.stdout.txt')

    # create the destination file only if the destination path doesn't exist yet
    mode = CopyRequest.CREATE

    # no need to recurse, we're just uploading a file
    recursive = False

    # perform the copy/download and wait 1000 ms for the successful or
    # otherwise completion of the operation
    copy_id = remote_fs.copy(remote_file, local_fs, local_file,
                             mode=mode, recursive=recursive)

    copy_status = remote_fs.wait_until_done(copy_id, timeout=5000)

    assert copy_status.done

    # rethrow the Exception if we got one
    assert copy_status.error_type == CopyStatus.NONE, copy_status.error_message

    print('Done downloading.')


xenon.init()

# use the local file system adaptor to create a file system representation
local_fs = FileSystem.create(adaptor='file')

# use the sftp file system adaptor to create another file system representation;
# the remote filesystem requires credentials to log in, so we'll have to
# create those too.
credential = PasswordCredential(username='xenon',
                                password='javagat')

remote_fs = FileSystem.create(adaptor='sftp',
                              location='localhost:10022',
                              password_credential=credential)

upload()
submit()
download()

# remember to close the FileSystem instances
remote_fs.close()
local_fs.close()

print('Done')

After about 60 seconds, you should have a local copy of sleep.stdout.txt, with the correct contents this time.

Congratulations – you have successfully completed the tutorial!




What’s next?

If you want, you can continue reading about relevant subjects, or try some of the suggested exercises.

Further reading

Suggested exercises

  • Repeat selected exercises, but test against a physically remote system instead of a Docker container. Requires credentials for the remote system.
  • Repeat selected exercises using WebDAV instead of SFTP. We included the Docker container nlesc/xenon-webdav as part of the virtual machine for testing.
  • Update xenon-cli to the latest version, so you can use the s3 file adaptor to connect to Amazon’s Simple Storage Service. Requires downloading of the Docker container nlesc/xenon-s3 (300MB) from DockerHub for testing on your own machine, or an Amazon Web Services account for testing against a physically remote system.