I expect some level of caching and even request bucketing by similarity is possi... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		pharmakom on March 2, 2023 \| parent \| context \| favorite \| on: Introducing ChatGPT and Whisper APIs I expect some level of caching and even request bucketing by similarity is possible. How many users come with the same prompt?

thewataccount on March 2, 2023 [–]

In my experience running the same prompt always get's different results. Maybe they cache between different people but I'm not sure that'd be worth the cache space at that point? although 8x A100s is a lot to not have caching...

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact