Advanced configuration

So far, we have relied on the icat.config module to provide configuration variables for us (such as url or idsurl). However, programs may also define their own custom configuration variables.

Custom configuration variables

Let’s add the option to redirect the output of our example program to a file. The output file path shall be passed via the command line as a configuration variable. To set this up, we can use the add_variable() method:

#! /usr/bin/python

import sys
import icat
import icat.config

config = icat.config.Config(ids="optional")
config.add_variable("outfile", ("-o", "--outputfile"),
                    dict(help="output file name or '-' for stdout"),
                    default="-")
client, conf = config.getconfig()
client.login(conf.auth, conf.credentials)

if conf.outfile == "-":
    out = sys.stdout
else:
    out = open(conf.outfile, "wt")

print("Login to %s was successful." % (conf.url), file=out)
print("User: %s" % (client.getUserName()), file=out)

out.close()

This adds a new configuration variable outfile. It can be specified on the command line as -o OUTFILE or --outputfile OUTFILE and it defaults to the string - if not specified. We can check this on the list of available command line options:

$ python config-custom.py -h
usage: config-custom.py [-h] [-c CONFIGFILE] [-s SECTION] [-w URL]
                        [--idsurl IDSURL] [--no-check-certificate]
                        [--http-proxy HTTP_PROXY] [--https-proxy HTTPS_PROXY]
                        [--no-proxy NO_PROXY] [-a AUTH] [-u USERNAME] [-P]
                        [-p PASSWORD] [-o OUTFILE]

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIGFILE, --configfile CONFIGFILE
                        config file
  -s SECTION, --configsection SECTION
                        section in the config file
  -w URL, --url URL     URL to the web service description
  --idsurl IDSURL       URL to the ICAT Data Service
  --no-check-certificate
                        don't verify the server certificate
  --http-proxy HTTP_PROXY
                        proxy to use for http requests
  --https-proxy HTTPS_PROXY
                        proxy to use for https requests
  --no-proxy NO_PROXY   list of exclusions for proxy use
  -a AUTH, --auth AUTH  authentication plugin
  -u USERNAME, --user USERNAME
                        username
  -P, --prompt-pass     prompt for the password
  -p PASSWORD, --pass PASSWORD
                        password
  -o OUTFILE, --outputfile OUTFILE
                        output file name or '-' for stdout

This new option is optional, so the program can be used as before:

$ python config-custom.py -s myicat_jdoe
Login to https://icat.example.com:8181 was successful.
User: db/jdoe

If we add the option on the command line, it has the expected effect:

$ python config-custom.py -s myicat_jdoe -o out.txt
$ cat out.txt
Login to https://icat.example.com:8181 was successful.
User: db/jdoe

Alternatively, we may also specify the option in the configuration file as follows:

[myicat_jdoe]
url = https://icat.example.com:8181
auth = db
username = jdoe
password = secret
idsurl = https://icat.example.com:8181
#checkCert = No
outfile = out.txt

Flag configuration variables

Instead of passing a string value to our program, we can also define different variable types using the type parameter. Among other things, this allows us to pass boolean/flag parameters. Let’s add another configuration variable to our example program that lets us control the output via a flag:

#! /usr/bin/python

import sys
import icat
import icat.config

config = icat.config.Config(ids="optional")
config.add_variable("outfile", ("-o", "--outputfile"),
                    dict(help="output file name or '-' for stdout"),
                    default="-")
config.add_variable("hide", ["--hide-user-name"],
                    dict(help="do not display the user after login"),
                    default=False, type=icat.config.flag)
client, conf = config.getconfig()
client.login(conf.auth, conf.credentials)

if conf.outfile == "-":
    out = sys.stdout
else:
    out = open(conf.outfile, "wt")

print("Login to %s was successful." % (conf.url), file=out)
if not conf.hide:
    print("User: %s" % (client.getUserName()), file=out)

out.close()

If we call our program normally, we get the same output as before:

$ python config-flag.py -s myicat_jdoe
Login to https://icat.example.com:8181 was successful.
User: db/jdoe

But if we pass the flag parameter, it produces a different output:

$ python config-flag.py -s myicat_jdoe --hide-user-name
Login to https://icat.example.com:8181 was successful.

A flag type configuration variable also adds a negated form of the command line flag:

$ python config-flag.py -s myicat_jdoe --no-hide-user-name
Login to https://icat.example.com:8181 was successful.
User: db/jdoe

This may look somewhat pointless at first glance as it only affirms the default. It becomes useful if we set this flag in the configuration file as in:

[myicat_jdoe]
url = https://icat.example.com:8181
auth = db
username = jdoe
password = secret
idsurl = https://icat.example.com:8181
#checkCert = No
hide = true

In that case we can override this setting on the command line with --no-hide-user-name.

Defining sub-commands

Many programs split up their functionality into sub-commands. For instance, the git program can be called as git clone, git checkout, git commit, and so on. In general, each sub-command will take their own set of configuration variables.

You can create programs like this and manage the configuration of each sub-command with icat.config using the add_subcommands() method. It adds a special ConfigSubCmds configuration variable representing the sub-command. This object provides the add_subconfig() method to register a new sub-command value. On the sub-config object in turn you can then define specific configuration variables using the familiar add_variable() method.

To put it all together, consider the following example program:

#! /usr/bin/python

import icat
import icat.config

config = icat.config.Config(ids="optional")

# add a global configuration variable 'entity' common for all sub-commands
config.add_variable("entity", ("-e", "--entity"),
                    dict(help="an entity from the ICAT schema",
                         choices=["User", "Study"]))

# add the configuration variable representing the sub-commands
subcmds = config.add_subcommands("mode")

# register three possible values for the sub-commands {list,create,delete}
subconfig_list = subcmds.add_subconfig("list",
                                       dict(help="list existing ICAT objects"))
subconfig_create = subcmds.add_subconfig("create",
                                         dict(help="create a new ICAT object"))
subconfig_delete = subcmds.add_subconfig("delete",
                                         dict(help="delete an ICAT object"))

# add two additional configuration variables 'name' and 'id' that are
# specific to the 'create' and 'delete' sub-commands respectively.
subconfig_create.add_variable("name", ("-n", "--name"),
                              dict(help="name for the new ICAT object"))
subconfig_delete.add_variable("id", ("-i", "--id"),
                              dict(help="ID of the ICAT object"))

client, conf = config.getconfig()
client.login(conf.auth, conf.credentials)

# check which sub-command (mode) was called
if conf.mode.name == "list":
    print("listing existing %s objects..." % conf.entity)
    print(client.search(conf.entity))
elif conf.mode.name == "create":
    print("creating a new %s object named %s..." % (conf.entity, conf.name))
    obj = client.new(conf.entity, name=conf.name)
    obj.create()
elif conf.mode.name == "delete":
    print("deleting the %s object with ID %s..." % (conf.entity, conf.id))
    obj = client.get(conf.entity, conf.id)
    client.delete(obj)

print("done")

If we check the available commands for the above program, our three sub-commands should be listed:

$ python config-sub-commands.py -h
usage: config-sub-commands.py [-h] [-c CONFIGFILE] [-s SECTION] [-w URL]
                              [--idsurl IDSURL] [--no-check-certificate]
                              [--http-proxy HTTP_PROXY]
                              [--https-proxy HTTPS_PROXY]
                              [--no-proxy NO_PROXY] [-a AUTH] [-u USERNAME]
                              [-P] [-p PASSWORD] [-e {User,Study}]
                              {list,create,delete} ...

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIGFILE, --configfile CONFIGFILE
                        config file
  -s SECTION, --configsection SECTION
                        section in the config file
  -w URL, --url URL     URL to the web service description
  --idsurl IDSURL       URL to the ICAT Data Service
  --no-check-certificate
                        don't verify the server certificate
  --http-proxy HTTP_PROXY
                        proxy to use for http requests
  --https-proxy HTTPS_PROXY
                        proxy to use for https requests
  --no-proxy NO_PROXY   list of exclusions for proxy use
  -a AUTH, --auth AUTH  authentication plugin
  -u USERNAME, --user USERNAME
                        username
  -P, --prompt-pass     prompt for the password
  -p PASSWORD, --pass PASSWORD
                        password
  -e {User,Study}, --entity {User,Study}
                        an entity from the ICAT schema

subcommands:
  {list,create,delete}
    list                list existing ICAT objects
    create              create a new ICAT object
    delete              delete an ICAT object

This looks good. Let’s try calling our program with the list sub-command. Of course we must also provide a section from our config file (-s SECTION) as well as the entity variable (-e {User,Study}) we defined earlier:

$ python config-sub-commands.py -s myicat_root -e User list
listing existing User objects...
[(user){
   createId = "simple/root"
   createTime = 2023-06-28 12:22:34+02:00
   id = 1
   modId = "simple/root"
   modTime = 2023-06-28 12:22:34+02:00
   affiliation = "University of Ravenna, Institute of Modern History"
   email = "acord@example.org"
   familyName = "Cordus"
   fullName = "Aelius Cordus"
   givenName = "Aelius"
   name = "db/acord"
   orcidId = "0000-0002-3262"
 }, (user){
   createId = "simple/root"
   createTime = 2023-06-28 12:22:34+02:00
   id = 2
   modId = "simple/root"
   modTime = 2023-06-28 12:22:34+02:00
   affiliation = "Goethe University Frankfurt, Faculty of Philosophy and History"
   email = "ahau@example.org"
   familyName = "Hau"
   fullName = "Arnold Hau"
   givenName = "Arnold"
   name = "db/ahau"
   orcidId = "0000-0002-3263"
 }, (user){
   createId = "simple/root"
   createTime = 2023-06-28 12:22:34+02:00
   id = 3
   modId = "simple/root"
   modTime = 2023-06-28 12:22:34+02:00
   affiliation = "Université Paul-Valéry Montpellier 3"
   email = "jbotu@example.org"
   familyName = "Botul"
   fullName = "Jean-Baptiste Botul"
   givenName = "Jean-Baptiste"
   name = "db/jbotu"
   orcidId = "0000-0002-3264"
 }, (user){
   createId = "simple/root"
   createTime = 2023-06-28 12:22:34+02:00
   id = 4
   modId = "simple/root"
   modTime = 2023-06-28 12:22:34+02:00
   email = "jdoe@example.org"
   familyName = "Doe"
   fullName = "John Doe"
   givenName = "John"
   name = "db/jdoe"
 }, (user){
   createId = "simple/root"
   createTime = 2023-06-28 12:22:34+02:00
   id = 5
   modId = "simple/root"
   modTime = 2023-06-28 12:22:34+02:00
   affiliation = "University of Nancago"
   email = "nbour@example.org"
   familyName = "Bourbaki"
   fullName = "Nicolas Bourbaki"
   givenName = "Nicolas"
   name = "db/nbour"
   orcidId = "0000-0002-3266"
 }, (user){
   createId = "simple/root"
   createTime = 2023-06-28 12:22:34+02:00
   id = 6
   modId = "simple/root"
   modTime = 2023-06-28 12:22:34+02:00
   affiliation = "Kaiser-Wilhelms-Akademie für das militärärztliche Bildungswesen"
   email = "rbeck@example.org"
   familyName = "Beck-Dülmen"
   fullName = "Rudolph Beck-Dülmen"
   givenName = "Rudolph"
   name = "db/rbeck"
   orcidId = "0000-0002-3267"
 }, (user){
   createId = "simple/root"
   createTime = 2023-06-28 12:22:34+02:00
   id = 7
   modId = "simple/root"
   modTime = 2023-06-28 12:22:34+02:00
   fullName = "Data Ingester"
   name = "simple/dataingest"
 }, (user){
   createId = "simple/root"
   createTime = 2023-06-28 12:22:34+02:00
   id = 8
   modId = "simple/root"
   modTime = 2023-06-28 12:22:34+02:00
   fullName = "IDS reader"
   name = "simple/idsreader"
 }, (user){
   createId = "simple/root"
   createTime = 2023-06-28 12:22:34+02:00
   id = 9
   modId = "simple/root"
   modTime = 2023-06-28 12:22:34+02:00
   fullName = "Pub reader"
   name = "simple/pubreader"
 }, (user){
   createId = "simple/root"
   createTime = 2023-06-28 12:22:34+02:00
   id = 10
   modId = "simple/root"
   modTime = 2023-06-28 12:22:34+02:00
   fullName = "Root"
   name = "simple/root"
 }, (user){
   createId = "simple/root"
   createTime = 2023-06-28 12:22:34+02:00
   id = 11
   modId = "simple/root"
   modTime = 2023-06-28 12:22:34+02:00
   fullName = "User Office"
   name = "simple/useroffice"
 }]
done

We see the users defined in the example content created in the previous tutorial sections. Let’s add a new user. We will use the create sub-command to do this. Earlier, we defined a configuration variable name (-n NAME) that is specific to the create sub-command. We can check this by calling:

$ python config-sub-commands.py create -h
usage: config-sub-commands.py create [-h] [-n NAME]

optional arguments:
  -h, --help            show this help message and exit
  -n NAME, --name NAME  name for the new ICAT object

Let’s create a new User object named “db/alice”. Note that we must provide the ‘global’ configuration variables (section and entity) before the sub-command, and the sub-command-specific option (name) after it:

$ python config-sub-commands.py -s myicat_root -e User create -n db/alice
creating a new User object named db/alice...
done

If we now list the User objects again, we can see a new object with name “db/alice”:

$ python config-sub-commands.py -s myicat_root -e User list
listing existing User objects...
[(user){
   createId = "simple/root"
   createTime = 2023-06-28 12:22:34+02:00
   id = 1
   modId = "simple/root"
   modTime = 2023-06-28 12:22:34+02:00
   affiliation = "University of Ravenna, Institute of Modern History"
   email = "acord@example.org"
   familyName = "Cordus"
   fullName = "Aelius Cordus"
   givenName = "Aelius"
   name = "db/acord"
   orcidId = "0000-0002-3262"
 }, (user){
   createId = "simple/root"
   createTime = 2023-06-28 12:22:34+02:00
   id = 2
   modId = "simple/root"
   modTime = 2023-06-28 12:22:34+02:00
   affiliation = "Goethe University Frankfurt, Faculty of Philosophy and History"
   email = "ahau@example.org"
   familyName = "Hau"
   fullName = "Arnold Hau"
   givenName = "Arnold"
   name = "db/ahau"
   orcidId = "0000-0002-3263"
 }, (user){
   createId = "simple/root"
   createTime = 2023-06-28 12:22:34+02:00
   id = 3
   modId = "simple/root"
   modTime = 2023-06-28 12:22:34+02:00
   affiliation = "Université Paul-Valéry Montpellier 3"
   email = "jbotu@example.org"
   familyName = "Botul"
   fullName = "Jean-Baptiste Botul"
   givenName = "Jean-Baptiste"
   name = "db/jbotu"
   orcidId = "0000-0002-3264"
 }, (user){
   createId = "simple/root"
   createTime = 2023-06-28 12:22:34+02:00
   id = 4
   modId = "simple/root"
   modTime = 2023-06-28 12:22:34+02:00
   email = "jdoe@example.org"
   familyName = "Doe"
   fullName = "John Doe"
   givenName = "John"
   name = "db/jdoe"
 }, (user){
   createId = "simple/root"
   createTime = 2023-06-28 12:22:34+02:00
   id = 5
   modId = "simple/root"
   modTime = 2023-06-28 12:22:34+02:00
   affiliation = "University of Nancago"
   email = "nbour@example.org"
   familyName = "Bourbaki"
   fullName = "Nicolas Bourbaki"
   givenName = "Nicolas"
   name = "db/nbour"
   orcidId = "0000-0002-3266"
 }, (user){
   createId = "simple/root"
   createTime = 2023-06-28 12:22:34+02:00
   id = 6
   modId = "simple/root"
   modTime = 2023-06-28 12:22:34+02:00
   affiliation = "Kaiser-Wilhelms-Akademie für das militärärztliche Bildungswesen"
   email = "rbeck@example.org"
   familyName = "Beck-Dülmen"
   fullName = "Rudolph Beck-Dülmen"
   givenName = "Rudolph"
   name = "db/rbeck"
   orcidId = "0000-0002-3267"
 }, (user){
   createId = "simple/root"
   createTime = 2023-06-28 12:22:34+02:00
   id = 7
   modId = "simple/root"
   modTime = 2023-06-28 12:22:34+02:00
   fullName = "Data Ingester"
   name = "simple/dataingest"
 }, (user){
   createId = "simple/root"
   createTime = 2023-06-28 12:22:34+02:00
   id = 8
   modId = "simple/root"
   modTime = 2023-06-28 12:22:34+02:00
   fullName = "IDS reader"
   name = "simple/idsreader"
 }, (user){
   createId = "simple/root"
   createTime = 2023-06-28 12:22:34+02:00
   id = 9
   modId = "simple/root"
   modTime = 2023-06-28 12:22:34+02:00
   fullName = "Pub reader"
   name = "simple/pubreader"
 }, (user){
   createId = "simple/root"
   createTime = 2023-06-28 12:22:34+02:00
   id = 10
   modId = "simple/root"
   modTime = 2023-06-28 12:22:34+02:00
   fullName = "Root"
   name = "simple/root"
 }, (user){
   createId = "simple/root"
   createTime = 2023-06-28 12:22:34+02:00
   id = 11
   modId = "simple/root"
   modTime = 2023-06-28 12:22:34+02:00
   fullName = "User Office"
   name = "simple/useroffice"
 }, (user){
   createId = "simple/root"
   createTime = 2023-06-28 15:06:06+02:00
   id = 12
   modId = "simple/root"
   modTime = 2023-06-28 15:06:06+02:00
   name = "db/alice"
 }]
done

Finally, let’s delete this new object using the delete sub-command. To do this, we must specify the sub-command-specific configuration variable id (-i ID). In the above output, we can see that the object’s ID is 12, so we write:

$ python config-sub-commands.py -s myicat_root -e User delete -i 12
deleting the User object with ID 12...
done

Preset configuration values in the program

It is possible to preset the value for configuration variables in the program, using the preset keyword argument to Config. Consider the following example program:

#! /usr/bin/python

import icat
import icat.config

config = icat.config.Config(ids="optional",
                            preset={"configSection": "myicat_root"})
client, conf = config.getconfig()
client.login(conf.auth, conf.credentials)

print("Login to %s was successful." % (conf.url))
print("User: %s" % (client.getUserName()))

You can call this as follows:

$ python config-preset.py
Login to https://icat.example.com:8181 was successful.
User: simple/root

Note that we did not specify the configuration section on the command line, yet the program picked the configuration values from the myicat_root section in the configuration file. Setting values using the preset keyword argument has a similar effect as setting a default for the respective configuration variable. But it also allows to override the default for configuration variables that are predefined by icat.config.

We can still override the preset configuration variable in the command line:

$ python config-preset.py -s myicat_nbour
Login to https://icat.example.com:8181 was successful.
User: db/nbour