Skip to content

MMM Points of Contact

User management tools§

This page contains tools and information for the nominated Points of Contact.

Other system-specific information is at Thomas, Michael or Young.

These commands can all be run as thomas-command or michael-command or young-command: they run the same thing and the different names are for convenience.

Displaying user information§

thomas-show, michael-show or young-show is a tool that enables you to find a lot of information about users. Access to the database is given to points of contact individually, contact rc-support@ucl.ac.uk if you try to use this and get an access denied.

At the top level, --user shows all information for one user, in multiple tables. --contacts shows all points of contact - useful for getting the IDs, and --institutes is the same. --allusers will show everyone's basic info. --getmmm will show the most recently used mmm username.

thomas-show -h  
usage: thomas-show [-h] [--user username] [--contacts] [--institutes]  
                  [--allusers] [--getmmm]  
                  {recentusers,getusers,whois} ...

Show data from the Thomas database. Use [positional argument -h] for more  
help.

positional arguments:  
 {recentusers,getusers,whois}  
   recentusers         Show the n newest users (5 by default)  
   getusers            Show all users with this project, institute, contact  
   whois               Search for users matching the given requirements

optional arguments:  
 -h, --help            show this help message and exit  
 --user username       Show all current info for this user  
 --contacts            Show all allowed values for contact  
 --institutes          Show all allowed values for institute  
 --allusers            Show all current users  
 --getmmm              Show the highest mmm username used

Show recent users§

thomas-show recentusers shows you the most recently-added N users, default 5.

thomas-show recentusers -h  
usage: thomas-show recentusers [-h] [-n N]

optional arguments:  
 -h, --help  show this help message and exit  
 -n N

Show users with a given project, institute, contact§

thomas-show getusers will search for exact matches to the given project, institute, contact combination.

thomas-show getusers -h  
usage: thomas-show getusers [-h] [-p PROJECT] [-i INST_ID] [-c POC_ID]

optional arguments:  
 -h, --help            show this help message and exit  
 -p PROJECT, --project PROJECT  
                       Project name  
 -i INST_ID, --institute INST_ID  
                       Institute ID  
 -c POC_ID, --contact POC_ID  
                       Point of Contact ID

Search for users based on partial information§

thomas-show whois can be used to search for partial matches to username, name, email fragments, including all of those in combination.

thomas-show whois -h  
usage: thomas-show whois [-h] [-u USERNAME] [-e EMAIL] [-n GIVEN_NAME]  
                        [-s SURNAME]

optional arguments:  
 -h, --help            show this help message and exit  
 -u USERNAME, --user USERNAME  
                       UCL username of user contains  
 -e EMAIL, --email EMAIL  
                       Email address of user contains  
 -n GIVEN_NAME, --name GIVEN_NAME  
                       Given name of user contains  
 -s SURNAME, --surname SURNAME  
                       Surname of user contains

Adding user information and new projects§

thomas-add will add information to the database. Access to the database is given to points of contact individually, contact rc-support@ucl.ac.uk if you try to use this and get an access denied.

Please note that all options have a --debug flag that will allow you to see the query generated without committing the changes to the database - double-check that the information you are adding is correct.

thomas-add -h  
usage: thomas-add [-h] {user,project,projectuser,poc,institute} ...

Add data to the Thomas database. Use [positional argument -h] for more help.

positional arguments:  
 {user,project,projectuser,poc,institute}  
   csv                 Add all users from the provided CSV file
   user                Adding a new user with their initial project  
   project             Adding a new project  
   projectuser         Adding a new user-project-contact relationship  
   poc                 Adding a new Point of Contact  
   institute           Adding a new institute/consortium

optional arguments:  
 -h, --help            show this help message and exit

Add a new user§

thomas-add user allows you to add a new user, with their initial project and point of contact. This does not create their account, but does email us with everything we need in order to create it. If you run this, you do not need to email us separately. The project specified must exist.

The user will be allocated the next free mmmxxxx username - you should only specify username yourself if they are an existing UCL user, or on Young if they previously had a Thomas or Michael account you should give them the same username.

You can get your poc_id by looking at thomas-show --contacts.

thomas-add user -h  
usage: thomas-add user [-h] -u USERNAME -n GIVEN_NAME [-s SURNAME] -e  
                      EMAIL_ADDRESS -k "SSH_KEY" -p PROJECT_ID -c POC_ID  
                      [--debug]

optional arguments:  
 -h, --help            show this help message and exit  
 -u USERNAME, --user USERNAME  
                       UCL username of user  
 -n GIVEN_NAME, --name GIVEN_NAME  
                       Given name of user  
 -s SURNAME, --surname SURNAME  
                       Surname of user (optional)  
 -e EMAIL_ADDRESS, --email EMAIL_ADDRESS  
                       Institutional email address of user  
 -k "SSH_KEY", --key "SSH_KEY"  
                       User's public ssh key (quotes necessary)  
 -p PROJECT_ID, --project PROJECT_ID  
                       Initial project the user belongs to  
 -c POC_ID, --contact POC_ID  
                       Short ID of the user's Point of Contact  
 --verbose             Show SQL queries that are being submitted  
 --debug               Show SQL query submitted without committing the change
SSH key formats§

It will verify the provided ssh key by default. Note that it has to be in the form ssh-xxx keystartshere. If someone has sent in a key which has line breaks and header items, make it into this format by adding the key type to the start and removing the line breaks from the key body.

This key:

---- BEGIN SSH2 PUBLIC KEY ----  
Comment: "comment goes here"  
AAAAB3NzaC1yc2EAAAABJQAAAQEAlLhFLr/4LGC3cM1xgRZVxfQ7JgoSvnVXly0K  
7MNufZbUSUkKtVnBXAOIjtOYe7EPndyT/SAq1s9RGZ63qsaVc/05diLrgL0E0gW+  
9VptTmiUh7OSsXkoKQn1RiACfH7sbKi6H373bmB5/TyXNZ5C5KVmdXxO+laT8IdW  
7JdD/gwrBra9M9vAMfcxNYVCBcPQRhJ7vOeDZ+e30qapH4R/mfEyKorYxrvQerJW  
OeLKjOH4rSnAAOLcEqPmJhkLL8k6nQAAK3P/E1PeOaB2xD7NNPqfIsjhAJLZ+2wV  
3eUZATx9vnmVF0YafOjvzcoK2GqUrhNAvi7k0f+ihh8twkfthj==  
---- END SSH2 PUBLIC KEY ----

should be converted into

ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAlLhFLr/4LGC3cM1xgRZVxfQ7JgoSvnVXly0K7MNufZbUSUkKtVnBXAOIjtOYe7EPndyT/SAq1s9RGZ63qsaVc/05diLrgL0E0gW+9VptTmiUh7OSsXkoKQn1RiACfH7sbKi6H373bmB5/TyXNZ5C5KVmdXxO+laT8IdW7JdD/gwrBra9M9vAMfcxNYVCBcPQRhJ7vOeDZ+e30qapH4R/mfEyKorYxrvQerJWOeLKjOH4rSnAAOLcEqPmJhkLL8k6nQAAK3P/E1PeOaB2xD7NNPqfIsjhAJLZ+2wV3eUZATx9vnmVF0YafOjvzcoK2GqUrhNAvi7k0f+ihh8twkfthj==

Other types of keys (ed25519 etc) will say what they are in the first line, and you should change the ssh-rsa appropriately. The guide linked at Creating an ssh key in Windows also shows where users can get the second format out of PuTTY.

Add new users in bulk from a CSV file§

young-add csv allows you to add users in bulk using a CSV file of specific format and headers.

The CSV is comma-separated with a header line of

email,given_name,surname,username,project_ID,ssh_key

You can leave username empty for it to allocate them a new username, but if they have an existing mmm username you should fill it in. It may be useful to show users with a given institute on Thomas if you are migrating users from one service to another.

You can download a CSV template here. Replace the example data.

young-add csv will try to automatically get your Point of Contact ID based on your username. If it can't, or if you have more than one, it will give you a list to choose from. (All users in one CSV upload will be added using the same Point of Contact ID).

The project you are adding the user to must already exist.

The SSH key must be formatted as shown in SSH key formats.

If you check your CSV file on the cluster with cat -v and it shows that it is beginning with M-oM-;M-? and ending with ^M you probably need to run dos2unix on it first.

Add a new project§

thomas-add project will create a new project, associated with an institution. It will not show in Gold until it also has a user in it.

A project ID should begin with your institute ID, followed by an underscore and a project name.

thomas-add project -h  
usage: thomas-add project [-h] -p PROJECT_ID -i INST_ID [--debug]

optional arguments:  
 -h, --help            show this help message and exit  
 -p PROJECT_ID, --project PROJECT_ID  
                       A new unique project ID  
 -i INST_ID, --institute INST_ID  
                       Institute ID this project belongs to  
 --debug               Show SQL query submitted without committing the change

Add a new project/user pairing§

thomas-add projectuser will add an existing user to an existing project. Creating a new user for an existing project also creates this relationship. After a new project-user relationship is added, a cron job will pick that up within 15 minutes and create that project for that user in Gold, with no allocation.

thomas-add projectuser -h  
usage: thomas-add projectuser [-h] -u USERNAME -p PROJECT_ID -c POC_ID  
                             [--debug]

optional arguments:  
 -h, --help            show this help message and exit  
 -u USERNAME, --user USERNAME  
                       An existing UCL username  
 -p PROJECT_ID, --project PROJECT_ID  
                       An existing project ID  
 -c POC_ID, --contact POC_ID  
                       An existing Point of Contact ID  
 --debug               Show SQL query submitted without committing the change

Deactivating information§

This tool is only partly functional at present. It allows you to deactivate (not delete) some entities that may no longer exist or may have been created in error.

Deactivate a projectuser§

Use this when the user should no longer be a member of the given project. It does not deactivate the user account, just their membership in this project. You can confirm the change by looking at young-show --user - it will say 'deactivated' rather than 'active' next to their listing for this project.

young-deactivate projectuser -h
usage: thomas_deactivate.py projectuser [-h] -u USERNAME -p PROJECT [--debug]

optional arguments:
  -h, --help            show this help message and exit
  -u USERNAME, --user USERNAME
                        An existing username
  -p PROJECT, --project PROJECT
                        An existing project ID
  --debug               Show SQL query submitted without committing the change

Gold resource allocation§

We are currently using Gold to manage allocations. Thomas and Michael share one Gold database, so all the projects exist on both, but they are only active on the correct cluster. Young has its own database.

Reporting from Gold§

There are wrapper scripts for a number of Gold commands (these exist in the userscripts module, loaded by default).

These are all set to report in cpu-hours with the -h flag, as that is our main unit. If you wish to change anything about the wrappers, they live in /shared/ucl/apps/cluster-scripts/ so you can take a copy and add your preferred options.

They all have a --man option to see the man pages for that command.

Here are some basic useful options and what they do. They can all be given more options for more specific searches.

gusage -p project_name [-s start_time]  # Show the Gold usage per user in this project, in the given timeframe if specified.  
gbalance                                # Show the balance for every project, split into total, reserved and available.  
glsuser                                 # Shows all the users in Gold.  
glsproject                              # Shows all the projects and which users are in them.  
glsres                                  # Show all the current reservatioms, inc user and project. The Name column is the SGE job ID.  
gstatement                              # Produce a reporting statement showing beginning and end balances, credits and debits.

# Less useful commands  
glstxn                                  # Show all Gold transactions. Filter or it will take forever to run.  
glsalloc                                # Show all the allocations.

These can be run by any user. The date format is YYYY-MM-DD.

Eg. gstatement -p PROJECT -s 2017-08-01 will show all credits and debits for the given project since the given date, saying which user and job ID each charge was associated with.

Transferring Gold§

As the point of contact, you can transfer Gold from your allocation account into other project accounts. As before, we've put -h in the wrapper so it is always working in cpu-hours.

gtransfer --fromProject xxx_allocation --toProject xxx_subproject cpu_hours

You can also transfer in the opposite direction, from the subproject back into your allocation account.

Note that you are able to transfer your allocation into another institute's projects, but you cannot transfer it back again - only the other institute's point of contact (or rc-support) can give it back, so be careful which project you specify.

When two allocations are active§

There is now an overlap period of a week when two allocations can be active. By default, gtransfer will transfer from active allocations in the order of earliest expiring first. To transfer from the new allocation only, you need to specify the allocation id.

gtransfer -i allocation_ID --fromProject xxx_allocation --toProject xxx_subproject cpu_hours

glsalloc -p xxx_allocation shows you all allocations that ever existed for your institute, and the first column is the id.

Id  Account Projects              StartTime  EndTime    Amount     Deposited  Description      
--- ------- --------------------- ---------- ---------- ---------- ---------- --------------   
87  38      UKCP_allocation       2017-08-07 2017-11-05 212800.00 3712800.00  
97  38      UKCP_allocation       2017-10-30 2018-02-04 3712800.00 3712800.00