Posts

Basic math: auto-regressive language generation Decoding methods Greedy Search Definition Example Feature Beam Search Definition examplle Feature Code Some function features of beam search Why beam search might not be good in open-ended generation Sampling issue 1 with sampling: coherent Top-K Sampling An example Good and Bad Top-p (nucleus) sampling — revised top-k sampling an example Top-K + Top-P Basic math: auto-regressive language generation auto-regressive language generation is based on the assumption that the probability distribution of a word sequence can be decomposed into the product of conditional next word distributions:

A common issue you face when switching environment using conda or different envs depends on different Java version. Here is the solution. Find where jdk installed you can find the location using whereis java and which java(which only output the dir of Java that is currently used).

Notes from txtbook Chapter 3

Buy a PC with Nvidia GPU Install Liunx Get connect to Server add a sudo user ssh connection get internet connection get IP address install ssh in the server generate rsa key pair send public to server config ssh file install dependencies Buy a PC with Nvidia GPU For CPU, you can choose AMD For GPU, Nvidia GPU is your only choice Notice that the bottleneck of your machine could most likely be the GPU memory.

2 Classes you need to self define when you implement Graph Neural Network(GNN): MyNet(pytorch.nn.Moduel) MyGraphModel(torch_geometric.nn.MessagePassing) MyNet(pytorch.nn.Moduel) In your overall model structure, you should implement: (in __init__): call a MessagePassing child class to build massage-passing model (in forward): make sure the data follows the requirement of MessagePassing child class do the “iterative massage passing"(K-times) in forward, the final output will be the node embedding you need.

To train a model, we need: loss function Optimizer model data Relation beetween Loss function and Optimizer? recall the simplest pytorch training process: dataset = My_data() loader = DataLoader(dataset) model = MyNet() criterion = torch.

To train a model, we need: loss function Optimizer model data Some facts testing is very expensive, since you have to look at whole testing set.

| First first, the forward of nn.Module is the one single most important part of the entire implementation. It may effect: other part of the nn.Module customized dataloader optimizer and loss function.

PyTorch has two primitives to work with data: torch.utils.data.DataLoader and torch.utils.data.Dataset. Dataset is an object that stores the samples and their corresponding labels there are two types of Dataset: https://pytorch.

Single Machine Data Parallel When data is too large to feed in a single GPU data is scatterred across GPUs Model is replicated on each GPUs Pytorch Gather the ouput from each GPUs, and compute loss, gradient, updates weights Code is simple, just one line: