Player FM uygulamasıyla çevrimdışı Player FM !
[QA] When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training
Manage episode 451541829 series 3524393
AnchorAttention addresses numerical issues in BFloat16 with Rotary Positional Embedding, enhancing long-context performance and training efficiency in large language models while maintaining task capabilities.
https://arxiv.org/abs//2411.13476
YouTube: https://www.youtube.com/@ArxivPapers
TikTok: https://www.tiktok.com/@arxiv_papers
Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016
Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers
2081 bölüm
Manage episode 451541829 series 3524393
AnchorAttention addresses numerical issues in BFloat16 with Rotary Positional Embedding, enhancing long-context performance and training efficiency in large language models while maintaining task capabilities.
https://arxiv.org/abs//2411.13476
YouTube: https://www.youtube.com/@ArxivPapers
TikTok: https://www.tiktok.com/@arxiv_papers
Apple Podcasts: https://podcasts.apple.com/us/podcast/arxiv-papers/id1692476016
Spotify: https://podcasters.spotify.com/pod/show/arxiv-papers
2081 bölüm
Tüm bölümler
×Player FM'e Hoş Geldiniz!
Player FM şu anda sizin için internetteki yüksek kalitedeki podcast'leri arıyor. En iyi podcast uygulaması ve Android, iPhone ve internet üzerinde çalışıyor. Aboneliklerinizi cihazlar arasında eş zamanlamak için üye olun.