BNL NSLS-II JupyterHub Guide

Where can I get help if I need it?

This Jupyter resource is maintained by the NSLS-II Data Science and Systems Integration (DSSI) Program. Specific questions related to beamline data or analysis should be directed first to NSLS-II beamline staff. Beamline staff can escalate issues regarding access or questions about the software environments through their Assigned Support Teams.

How do users get access?

All NSLS-II users and staff are granted access to Jupyter. This should happen automatically within an hour of account creation. (Formerly, users needed to be granted access, but this is no longer the case.)

What computational resources are available?

1. Storage Storage for large data is planned but not quite yet ready. Current you only have your home directory (quota 100 GB) which is intended to hold dotfiles and other configuration, temporary scratch files. Power users may also put custom conda environments there (under $HOME/.conda/envs) as described below.
2. Memory Each Jupyter single-user server gets 32 GB of memory. This is subject to change without notice.
3. CPUs Each Jupyter single-uesr server gets 8 CPUs.
4. GPUs GPUs are not yet accessible from Jupyter, but they are available on some of the underlying hosts and will be made available from Jupyter in the future.

Named Servers

JupyterHub enables each user to run multiple "servers" with potentially different configurations at the same time. These configurations can affect the software environment, and in the future, the computational resources (e.g. GPU access, available cores and memory). For more information about this JupyterHub feature see this section of the JupyterHub documentation.

Add or upgrade custom software

There are three different use cases:

1. Quickly install or upgrade a couple packages for immediate personal use.
2. Make additional (or upgraded) packages available to all users.
3. Create a separate, personal software environment to customize many packages.

Quickly install or upgrade a couple packages for immediate personal use

This is intended to enable easy changes to quickly experiment or work around problems. Any changes made this way are temporary. They will not persist after your Jupyter server is stopped and restarted. See other two approaches, detailed below, for ways to make a permanent change. In JupyterLab, use the menu File > New > Terminal. Run

conda install ...

or

pip install ...

as needed. There is limited space available (2 GB) for these changes. If installation resutls in an error No space left on device you have reached the limit. See one of the other approaches. Stopping and restarting your server (File > Hub Control Panel > Stop My Server) clears this space and gives you a fresh start.

Make additional (or upgraded) packages available to all users

Open a GitHub Issue or Pull Request against the repository NSLS-II/scipy-binder to update the requirements. System packages may be added to binder/apt.txt and conda or pip packages may be added to binder/environment.yml.

Create a separate, personal software environment to customize many packages

Go to File > New Launcher and open a Terminal. Deactivate the default environment (called notebook).

conda deactivate

Choose a name for the environment. It should have no spaces.

ENV_NAME=YOUR_ENVIRONMENT_NAME

Create a conda environment. This will be stored under your home directory. It will persist (unlike Approach 1) but it will only available to you (unlike Approach 2). The conda environment must contain the package ipykernel at least. Replace ... with any additional conda packages you want to install.

conda create -n $ENV_NAME ipykernel ...
conda activate $ENV_NAME

You may later add any packages you wish with conda install ... or pip install .... Some general advice:

• It's faster to install all the conda packages up front in the conda create command above.
• It's OK to install with pip as well as as conda, but it can sometimes cause problems if you install with conda after you have installed things with pip. To avoid issues, install the conda packges you need first, and once you use pip, stick to pip. If you need conda again, make a new environment.

python -m ipykernel install --user --name $ENV_NAME --display-name "$ENV_NAME"

The argument after --display-name can be anything you want; it does not have to be $ENV_NAME. It is for humans, not machines. It may have spaces. Go to File > New Launcher and you should see your new kernel among the options shown.

Appendix: Managing Kernels and Environments

To list the currently-available kernels and environments:

jupyter kernelspec list
conda env list

To remove the kernel and the environment you created above:

jupyter kernelspec remove $ENV_NAME
conda env remove --name $ENV_NAME

Where did my files go?

In Spring 2021, NSLS2 changed its user home directories. After a long period and multiple notifications to staff and users, the old home directories were archived. They are available upon request to your Assigned Support Team.