Working Notes: my commonplace notebook for recording & exploring ideas.
Home. Site Map. Subscribe. More at expLog.

Notes on PyTorch

torch.distributed

• this has a lot of magic functions available in the torch.distributed package

  • imported from distributed_c10.py link

• using it in a notebook as a way to use all 8 gpus on demand?

  • use the jupyter kernel as a torchrun controller equivalent
  • spawn multiple copies of the kernel remotely or locally
  • ideally this is baked into the kernel as a package instead, tentatively the way ray works?

• pytorch overrides the system excepthook… to prefix the exception with the rank number

• the magic WORLD property on groups comes from a metaclass

  • introduced in this Pull Request

— Kunal