Skip to content

This page contains tools and information for the nominated Points of Contact.

Other system-specific information is at Thomas or Michael.

These commands can all be run as thomas-command or michael-command: they run the same thing and the different names are for convenience.

Displaying user information§

thomas-show or michael-show is a tool that enables you to find a lot of information about users. Access to the database is given to points of contact individually, contact if you try to use this and get an access denied.

At the top level, --user shows all information for one user, in multiple tables. --contacts shows all points of contact - useful for getting the IDs, and --institutes is the same. --allusers will show everyone's basic info. --getmmm will show the most recently used mmm username.

thomas-show -h  
usage: thomas-show [-h] [--user username] [--contacts] [--institutes]  
                  [--allusers] [--getmmm]  
                  {recentusers,getusers,whois} ...

Show data from the Thomas database. Use [positional argument -h] for more  

positional arguments:  
   recentusers         Show the n newest users (5 by default)  
   getusers            Show all users with this project, institute, contact  
   whois               Search for users matching the given requirements

optional arguments:  
 -h, --help            show this help message and exit  
 --user username       Show all current info for this user  
 --contacts            Show all allowed values for contact  
 --institutes          Show all allowed values for institute  
 --allusers            Show all current users  
 --getmmm              Show the highest mmm username used

Show recent users§

thomas-show recentusers or michael-show recentusers shows you the most recently-added N users, default 5.

thomas-show recentusers -h  
usage: thomas-show recentusers [-h] [-n N]

optional arguments:  
 -h, --help  show this help message and exit  
 -n N

Show users with a given project, institute, contact§

thomas-show getusers or michael-show getusers will search for exact matches to the given project, institute, contact combination.

thomas-show getusers -h  
usage: thomas-show getusers [-h] [-p PROJECT] [-i INST_ID] [-c POC_ID]

optional arguments:  
 -h, --help            show this help message and exit  
 -p PROJECT, --project PROJECT  
                       Project name  
 -i INST_ID, --institute INST_ID  
                       Institute ID  
 -c POC_ID, --contact POC_ID  
                       Point of Contact ID

Search for users based on partial information§

thomas-show whois or michael-show whois can be used to search for partial matches to username, name, email fragments, including all of those in combination.

thomas-show whois -h  
usage: thomas-show whois [-h] [-u USERNAME] [-e EMAIL] [-n GIVEN_NAME]  
                        [-s SURNAME]

optional arguments:  
 -h, --help            show this help message and exit  
                       UCL username of user contains  
 -e EMAIL, --email EMAIL  
                       Email address of user contains  
                       Given name of user contains  
 -s SURNAME, --surname SURNAME  
                       Surname of user contains

Adding user information and new projects§

thomas-add or michael-add will add information to the database. Access to the database is given to points of contact individually, contact if you try to use this and get an access denied.

Please note that all options have a --debug flag that will allow you to see the query generated without committing the changes to the database - double-check that the information you are adding is correct.

thomas-add -h  
usage: thomas-add [-h] {user,project,projectuser,poc,institute} ...

Add data to the Thomas database. Use [positional argument -h] for more help.

positional arguments:  
   user                Adding a new user with their initial project  
   project             Adding a new project  
   projectuser         Adding a new user-project-contact relationship  
   poc                 Adding a new Point of Contact  
   institute           Adding a new institute/consortium

optional arguments:  
 -h, --help            show this help message and exit

Add a new user§

thomas-add user or michael-add user allows you to add a new user, with their initial project and point of contact. This does not create their account, but does email us with everything we need in order to create it. If you run this, you do not need to email us separately. The project specified must exist.

thomas-add user -h  
usage: thomas-add user [-h] -u USERNAME -n GIVEN_NAME [-s SURNAME] -e  
                      EMAIL_ADDRESS -k "SSH_KEY" -p PROJECT_ID -c POC_ID  

optional arguments:  
 -h, --help            show this help message and exit  
                       UCL username of user  
                       Given name of user  
 -s SURNAME, --surname SURNAME  
                       Surname of user (optional)  
                       Institutional email address of user  
 -k "SSH_KEY", --key "SSH_KEY"  
                       User's public ssh key (quotes necessary)  
 -p PROJECT_ID, --project PROJECT_ID  
                       Initial project the user belongs to  
 -c POC_ID, --contact POC_ID  
                       Short ID of the user's Point of Contact  
 --verbose             Show SQL queries that are being submitted  
 --debug               Show SQL query submitted without committing the change

SSH key formats§

It will verify the provided ssh key by default. Note that it has to be in the form ssh-xxx keystartshere. If someone has sent in a key which has line breaks and header items, make it into this format by adding the key type to the start and removing the line breaks from the key body.

This key:

Comment: "comment goes here"  

should be converted into

ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAlLhFLr/4LGC3cM1xgRZVxfQ7JgoSvnVXly0K7MNufZbUSUkKtVnBXAOIjtOYe7EPndyT/SAq1s9RGZ63qsaVc/05diLrgL0E0gW+9VptTmiUh7OSsXkoKQn1RiACfH7sbKi6H373bmB5/TyXNZ5C5KVmdXxO+laT8IdW7JdD/gwrBra9M9vAMfcxNYVCBcPQRhJ7vOeDZ+e30qapH4R/mfEyKorYxrvQerJWOeLKjOH4rSnAAOLcEqPmJhkLL8k6nQAAK3P/E1PeOaB2xD7NNPqfIsjhAJLZ+2wV3eUZATx9vnmVF0YafOjvzcoK2GqUrhNAvi7k0f+ihh8twkfthj==

Other types of keys (dss etc) will say what they are in the first line, and you should change the ssh-rsa appropriately. The guide linked at Creating an ssh key in Windows also shows where users can get the second format out of PuTTY.

Add a new project§

thomas-add project or michael-add project will create a new project, associated with an institution. It will not show in Gold until it also has a user in it.

A project ID should begin with your institute ID, followed by an underscore and a project name.

thomas-add project -h  
usage: thomas-add project [-h] -p PROJECT_ID -i INST_ID [--debug]

optional arguments:  
 -h, --help            show this help message and exit  
 -p PROJECT_ID, --project PROJECT_ID  
                       A new unique project ID  
 -i INST_ID, --institute INST_ID  
                       Institute ID this project belongs to  
 --debug               Show SQL query submitted without committing the change

Add a new project/user pairing§

thomas-add projectuser or michael-add projectuser will add an existing user to an existing project. Creating a new user for an existing project also creates this relationship. After a new project-user relationship is added, a cron job will pick that up within 15 minutes and create that project for that user in Gold, with no allocation.

thomas-add projectuser -h  
usage: thomas-add projectuser [-h] -u USERNAME -p PROJECT_ID -c POC_ID  

optional arguments:  
 -h, --help            show this help message and exit  
                       An existing UCL username  
 -p PROJECT_ID, --project PROJECT_ID  
                       An existing project ID  
 -c POC_ID, --contact POC_ID  
                       An existing Point of Contact ID  
 --debug               Show SQL query submitted without committing the change

Gold resource allocation§

We are currently using Gold to manage allocations on Thomas and Michael. There is one Gold database, so all the projects exist on both, but they are only active on specific clusters.

Reporting from Gold§

There are wrapper scripts for a number of Gold commands (these exist in the userscripts module, loaded by default).

These are all set to report in cpu-hours with the -h flag, as that is our main unit. If you wish to change anything about the wrappers, they live in /shared/ucl/apps/cluster-scripts/ so you can take a copy and add your preferred options.

They all have a --man option to see the man pages for that command.

Here are some basic useful options and what they do. They can all be given more options for more specific searches.

gusage -p project_name [-s start_time]  # Show the Gold usage per user in this project, in the given timeframe if specified.  
gbalance                                # Show the balance for every project, split into total, reserved and available.  
glsuser                                 # Shows all the users in Gold.  
glsproject                              # Shows all the projects and which users are in them.  
glsres                                  # Show all the current reservatioms, inc user and project. The Name column is the SGE job ID.  
gstatement                              # Produce a reporting statement showing beginning and end balances, credits and debits.

# Less useful commands  
glstxn                                  # Show all Gold transactions. Filter or it will take forever to run.  
glsalloc                                # Show all the allocations.

These can be run by any user. The date format is YYYY-MM-DD.

Eg. gstatement -p PROJECT -s 2017-08-01 will show all credits and debits for the given project since the given date, saying which user and job ID each charge was associated with.

Transferring Gold§

As the point of contact, you can transfer Gold from your allocation account into other project accounts. As before, we've put -h in the wrapper so it is always working in cpu-hours.

gtransfer --fromProject xxx_allocation --toProject xxx_subproject cpu_hours

You can also transfer in the opposite direction, from the subproject back into your allocation account.

Note that you are able to transfer your allocation into another institute's projects, but you cannot transfer it back again - only the other institute's point of contact (or rc-support) can give it back, so be careful which project you specify.

When two allocations are active§

There is now an overlap period of a week when two allocations can be active. By default, gtransfer will transfer from active allocations in the order of earliest expiring first. To transfer from the new allocation only, you need to specify the allocation id.

gtransfer -i allocation_ID --fromProject xxx_allocation --toProject xxx_subproject cpu_hours

glsalloc shows you all allocations that ever existed, and the first column is the id.

Id  Account Projects              StartTime  EndTime    Amount     Deposited  Description      
--- ------- --------------------- ---------- ---------- ---------- ---------- --------------   
87  38      UKCP_allocation       2017-08-07 2017-11-05 212800.00 3712800.00  
97  38      UKCP_allocation       2017-10-30 2018-02-04 3712800.00 3712800.00