> Due to the large size of the model (314B parameters), a machine with enough GP... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		gardenhedge on March 17, 2024 \| parent \| context \| favorite \| on: Grok > Due to the large size of the model (314B parameters), a machine with enough GPU memory is required to test the model with the example code What type of machine do you need to play around with this?

317070 on March 17, 2024 | [–]

Probably a machine with about 628 GB of GPU memory. (2 bytes per parameter)

So 8xH100 (80Gb each) should do it.

Marlinski on March 18, 2024 | | [–]

I suppose it can be quantizised

anigbrowl on March 17, 2024 | | [–]

'Chunky beast, needs 320 Gb VRAM likely 4 bit, likely is being run 8 bit on 8 x 80 Gb GPUs.'

-Emad

a_wild_dandan on March 18, 2024 | [–]

A single 192GB M2 Mac using a 4-bit quant would work.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact