\( \newcommand{\matr}[1] {\mathbf{#1}} \newcommand{\vertbar} {\rule[-1ex]{0.5pt}{2.5ex}} \newcommand{\horzbar} {\rule[.5ex]{2.5ex}{0.5pt}} \newcommand{\E} {\mathrm{E}} \)
deepdream of a sidewalk
Show Question
Math and science::INF ML AI

Transformer implementation

On the reverse side, there is a full transformer, implemented in Pytorch in vit-pytorch. Can you remember it?

Source

https://github.com/lucidrains/vit-pytorch/blob/main/vit_pytorch/vit_1d.py#L9