hardware//NVIDIA//A100

The A100 is the GPU that trained GPT-3, PaLM and most of the LLMs of 2021 to 2023. It carries 80GB of HBM2e memory and Tensor Cores for mixed-precision training.

The A100 is the GPU that trained GPT-3, PaLM and most of the LLMs of 2021 to 2023. It carries 80GB of HBM2e memory and Tensor Cores for mixed-precision training.

Clusters of thousands of A100s defined the scaling era, at roughly $10K to $15K per card, until the next generation, the H100, brought two to three times the training throughput and a dedicated Transformer Engine.