Couldn't you just compare the similarity of the embeddings? I imagine that would... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		pstorm on Nov 17, 2023 \| parent \| context \| favorite \| on: Show HN: Alerting in realtime RAG: spot changes to... Couldn't you just compare the similarity of the embeddings? I imagine that would work in the vast majority of cases and save a lot of LLM calls.

janchorowski on Nov 17, 2023 [–]

That's a good idea, the deduplication criterion is easy to change, using an llm is faster to get started, but after a while a corpus of decisions is created and can be used to either select another mechanism, or e.g. train one on top of bert embeddings.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact