โ† Reading Library / ๐Ÿš€ Science & Technology / ๐Ÿ’ก Great Inventions That Changed the World / Neural Network AI Chip (GPU/TPU)
Expand Grade: G1 Early Reader G2 Developing G3 Fluent G4 Expanding G5 Advanced
๐Ÿ’ก INVENTIONS โš™๏ธ 1999โ€“2015 CE โ€ข Massively Parallel Silicon Architecture VA Science, Engineering & History SOL

Neural Network AI Chip (GPU/TPU)

Thousands of Tiny Cores Thinking Together

Grade 5 โ€ข Advanced โ€ข 255 words
๐Ÿง‘โ€๐Ÿ”ฌ Inventor & Origin Nvidia GPU Pioneers (1999/2006) & Google TPU Team led by Norm Jouppi (2015)
โณ Year Invented 1999 Programmable GPU; 2012 Deep Learning Spark; 2015 Custom TPU
โš™๏ธ How It Works Thousands of tiny math units multiply giant grids of numbers at the exact same instant
โšก World-Changing Spark Solves quadrillions of math steps every second to power AI vision, translation, and science
Neural Network AI Chip (GPU/TPU) (1999โ€“2015 CE โ€ข Massively Parallel Silicon Architecture)
๐Ÿคฏ Mind-Blowing Fact โ€ข Grade 5

One modern AI accelerator executing 4 petaFLOPS performs as many math operations in a single second as a human doing one calculation every second would finish in 126 million years!

โœจ Word Sparks (Grade 5 Glossary) Tap ๐Ÿ”Š to hear
tensor
Multi-dimensional array holding numbers for AI math
petaFLOPS
One quadrillion math operations finished per second
quantization
Representing weights with compact, lower-bit numbers
๐Ÿช

How do 100-billion-transistor chips etched with extreme ultraviolet light execute quadrillions of tensor operations every second?

1 Beyond Von Neumann Sequential Computing

For seventy years, classical computer architecture followed John von Neumann's blueprint: a central processing unit fetching instructions one by one across a shared memory bus. Deep neural networks shattered that paradigmโ€”demanding trillions of simultaneous linear-algebra operations (Y = W ยท X + B) across multi-dimensional arrays known mathematically as .

2 SIMT GPUs and Domain-Specific Systolic TPUs

Two complementary silicon architectures rose to meet that demand. Modern GPUs employ Single-Instruction, Multiple-Thread (SIMT) streaming multiprocessors augmented with dedicated matrix Cores, descended from Nvidia's 1999 GPU and 2006 CUDA platform that powered AlexNet's historic 2012 ImageNet breakthrough. Taking specialization further in 2015, Norm Jouppi and colleagues unveiled Google's TPU around huge 128ร—128 or 256ร—256 2D systolic arrays (inspired by H. T. Kung and Charles Leiserson's 1978 theory), holding neural weights stationary inside MAC cells while streaming activations across without touching SRAM registers.

3 Numerical Formats, Photolithography, and 3D Packaging

Etched at 3-nanometer feature scales using Extreme Ultraviolet (EUV) lithography, flagship AI accelerators pack over 100 billion transistors onto reticle-limit dies. Utilizing Brain Floating Point (`BF16`), `FP8`, and `INT4` , a single accelerator delivers multiple (10ยนโต operations per second), fed at several terabytes per second through thousands of microscopic silicon-interposer wires connected to 3D-stacked High-Bandwidth Memory (HBM).

4 Optical Interconnects and Warehouse-Scale Superpods

Training frontier foundation models exceeds any single chip's memory, so engineers network tens of thousands of GPUs or TPUs into unified 'pods' via multi-hundred-gigabit copper and optical circuit-switched interconnectsโ€”coordinating distributed gradients across entire rooms as one giant logical supercomputer.

5 Neuromorphic and Photonic Horizons

To curb megawatt data-center power budgets, chip architects now explore analog in-memory resistive crossbars, spiking neuromorphic processors modeled on biological synapses, and silicon-photonic waveguides that multiply matrices at the literal speed of laser light.

๐Ÿง โšก Brain Sparks โ€ข Quick Curiosity Check ๐ŸŒŸ 0 / 2 Sparks Lit

1. In a weight-stationary systolic array inside a TPU, how does the hardware minimize wasted energy from memory reads?

2. Why must large AI chips connect to 3D-stacked High-Bandwidth Memory (HBM) right beside the processor on a silicon interposer?

๐ŸŒฟ๐Ÿพ Explore More 1-Page Curiosity Sparks: ๐Ÿ’ก Printing Press ๐Ÿ’ก Compass ๐Ÿ’ก Papermaking ๐Ÿ’ก Mechanical Clock ๐Ÿ’ก Eyeglasses ๐Ÿ’ก Microscope ๐Ÿ’ก Refracting Telescope ๐Ÿ’ก Reflecting Telescope ๐Ÿ’ก Mercury Thermometer ๐Ÿ’ก Barometer ๐Ÿ’ก Steam Engine ๐Ÿ’ก Spinning Jenny ๐Ÿฆ… Bald Eagle ๐Ÿ”ฌ Albert Einstein ๐ŸฅŠ Muhammad Ali โšฝ Lionel Messi โšฝ Pelรฉ ๐Ÿ›๏ธ Socrates