Isn’t the purpose of self attention exactly to recognize the relevance of some t... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		dcrazy 19 days ago \| parent \| context \| favorite \| on: LLM=True Isn’t the purpose of self attention exactly to recognize the relevance of some tokens over others?

kimixa 19 days ago | [–]

That may help with tokens being "ignored" while still being in the context window, but not context window size costs and limitations in the first place.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact