Lots of fine tuning, attention visualisation, evaluation of embeddings and different embedding generation methods, not just LLMs though I use them a lot for deep nets of many kinds
Can you talk about the attention visualization a bit. What’s the benefit. I have been plotting attention masks and even visualizing them over sentences but it obv doesn’t result in a ton of insight due to the sentences changing from words to much more complex vectors that represent those words after being processed through attentions 32x.
I am using attention weighted pooling strategy to generate cross-attentions across N transformer heads, this allows us to visualize which parts of an output were caused by which part of the input -- this is used to debug summarization. Only works for encoder/decoder networks - i.e. ones where you have cross attentions
Thank you! The animation at the top is a GPU shader of the transition between 1s, 2p and 3d electron wave functions in a hydrogen atom, the animated crystal of sulpher I made using Maya and then animated into/out of wireframe by loading it and spinning it in WebGL, the other ones further down are just screen recordings.
You might like this too - this last weekend I finished writing a topological contour GPU shader for visualizing electron density grids, here is a molecule of buckminsterfullerene - the contour generator is running on the GPU, which is how I can keep the framerate so high.
WOW! I was undergraduate chemistry (decades ago) and if these visualizations had existed back then... I may have actually done something with my chemistry training! The `SMILES` number text input had me in flashbacks about some of the fledgling research I half-assed waybackwhen.
Thanks for providing such a wonderful resource for future chemists!
Some people have been fine-tuning mistral 7B and phi-2 on their high-end macs. Unified memory is a hell of a thing. The resulting model here is not spectacular but as a proof of concept it's pretty exciting what you get in 3.5 hours on a consumer machine.
What can a bone-stock (16GB) M2Pro Mini do? Would it be correct to assume this takes 8x longer (on account of 1/4 memory and 1/2 GPU), or is training not even possible?
There's little between the M2 and M3 at this point with regards to LLM work, and given no M3 Ultra exists yet, the M2 Ultra would be the way to go considering this task in isolation, and indeed the only option if you wanted to go up to 192GB RAM. If you needed to be portable or had budgetary concerns, then your choice may differ.
No, they have far less bandwidth. However, I am not sure if getting Apple CPUs would be a good idea. You can get some toy projects running with them, but a pay-as-you-go cloud solution would have been cheaper and faster. I don't find my M1 Max to be particularly competitive.
AMD and Intel allow their customers to choose lots of different memory configurations and so they prioritize compatibility over bandwidth. Apple's architecture has no support for external memory, they put their memory directly on package and can prioritize for bandwidth.
How is that these days? I water-cooled my system with the intention of having a quiet system sometime around 2008, but it just used liquid to transfer heat to a sink further away, which had a noisy fan.
Much better these days and easier to do. Fans are aso much quieter these days too. A few years ago I was twisted into building a water cooled GPU system because it was the only 30 series card I could get in a 6 month period. Wasn't too hard to build into a system that doesn't even have enough room for the water pump to stand vertically. Soft tubing is annoying though and will fatigue with time, restricting flow rates. So I suggest if you do that to take more time to ensure you have angled fittings where you need them and to zip tie straight paths where you can (and thicker tubing is better). Might just be easier to go hardline instead tbh.
Thermals are great though. But tbh now what I do is put my machine behind my TV and I have the side open and place the pump outside to stand vertically like old systems that had them outside the case lol. That's because if I'm not playing games on it then there's no difference in just sshing into the system because I live in the terminal anyways.
Pretty straightforward these days! Though I've been doing it for awhile. There a lot of little gotcha's like tubing choice and fittings but once you get the hang of the options it's pretty fun.
Though cooling 4090s even with water is hard - you need a good flow rate to dissipate 450W. They'll hit near 70C during training if the fans are kept below audible, which works for me. 55-60C with fans at 50%. This is with 2x420mm (3x 140mm fan) rads and one 280mm rad (2x 140mm fans).
The stock heatsinks actually were actually able to cool them just as well, just they where much louder doing so!
Power is the other tricky bit. The computer is living on the dedicated AC circuit for now... 1600W is a lot. When it was on the single outlet circuit in my house, I could see each iteration as the lights dimmed in sync with the power draw surging.
3x4090s 1xTesla A100
Lots of fine tuning, attention visualisation, evaluation of embeddings and different embedding generation methods, not just LLMs though I use them a lot for deep nets of many kinds
Both for my day job (hedge fund) and my hobby project https://atomictessellator.com
It’s summer here in NZ and I have these in servers mounted in a freestanding server rack beside my desk, and it is very hot in here XD