to use a CNN: for data with spatial structure like images or time series

Contrastive Language–Image Pre-training

CLIP embeds images and text into a shared space using contrastive learning

384-dim all-MiniLM-L6-v2 optimizes: fast sentence similarity with 6 layers

All-MiniLM-L6-v2 optimizes fast sentence similarity with 6 layers

cosine similarity is preferred over dot product for normalized embeddings

Cosine similarity measures orientation, not magnitude, making it ideal for normalized embeddings

GCN (Graph Convolutional Network) does: spectral convolution approximated by neighbor averaging

GCN approximates spectral convolution via neighbor averaging

the embedding layer does: maps discrete token IDs to dense learned vectors

Embeddings convert token IDs to dense vectors for neural network processing

ALiBi allows length extrapolation better than learned position embeddings

ALiBi uses relative positional encoding, avoiding fixed-size embeddings, enabling better handling of variable-length sequences