Experiment Tracking with Weights and Biases

Introduction #

In this tutorial, we show how to set up experiment tracking via Weights and Biases on Neuro Platform using Neuro Project Template.

Creating Neuro Project #

To create a new Neuro project, run:

neuro project init cd <project-slug> make setup

Authenticating W&B #

Register a W&B account (note: W&B offers free limited accounts for individual researchers, see W&B pricing policy). Then, download your API key on W&B settings page and save it to a local file ./config/wandb-token.txt (you can also use a different file name, but in this case you'd need to put the file name to the env var: export WANDB_SECRET_FILE=<file-name>).

Check that Neuro project detects the key file:

make wandb-check-auth

In case of success, this command should print something like: Weights & Biases will be authenticated via key file: '/project-path/config/wandb-token.txt'. Now, if you run a develop, train, or jupyter job, Neuro will authenticate W&B via your API key by running wandb login at job's startup.

Testing #

Run a development job and connect to the job's shell:

export PRESET=cpu-small # to avoid consuming GPU for this test make develop make connect-develop

In your job's shell, try to use wandb:

wandb status

You should see something like:

Logged in? True Current Settings { "anonymous": "false", "base_url": "https://api.wandb.ai", "entity": null, "git_remote": "origin", "ignore_globs": [], "project": null, "run": "latest", "section": "default" }

Please find a real-world example of W&B usage on Neuro Platform in our ML Recipe for Hierarchical Attention. You can also find other examples on how to use experiment tracking and other features of W&B in official W&B's examples repo.

To close remote terminal session, press ^D or type exit.

Please don't forget to terminate your job when you don't need it anymore:

make kill-develop