Basic math: auto-regressive language generation Decoding methods Greedy Search Definition Example Feature Beam Search Definition examplle Feature Code Some function features of beam search Why beam search might not be good in open-ended generation Sampling issue 1 with sampling: coherent Top-K Sampling An example Good and Bad Top-p (nucleus) sampling — revised top-k sampling an example Top-K + Top-P Basic math: auto-regressive language generation auto-regressive language generation is based on the assumption that the probability distribution of a word sequence can be decomposed into the product of conditional next word distributions:

A common issue you face when switching environment using conda or different envs depends on different Java version. Here is the solution. Find where jdk installed you can find the location using whereis java and which java(which only output the dir of Java that is currently used).

Notes from txtbook Chapter 3

Buy a PC with Nvidia GPU Install Liunx Get connect to Server add a sudo user ssh connection get internet connection get IP address install ssh in the server generate rsa key pair send public to server config ssh file install dependencies Buy a PC with Nvidia GPU For CPU, you can choose AMD For GPU, Nvidia GPU is your only choice Notice that the bottleneck of your machine could most likely be the GPU memory.

2 Classes you need to self define when you implement Graph Neural Network(GNN): MyNet(pytorch.nn.Moduel) MyGraphModel(torch_geometric.nn.MessagePassing) MyNet(pytorch.nn.Moduel) In your overall model structure, you should implement: (in __init__): call a MessagePassing child class to build massage-passing model (in forward): make sure the data follows the requirement of MessagePassing child class do the “iterative massage passing"(K-times) in forward, the final output will be the node embedding you need.