Skip to main content

Chapter 7.1 - 1st Residual Connection

Overview

Have you ever played a video game and saved your progress right before a boss fight? That way, if you mess up, you don't lose everything! A Residual Connection is exactly like a save point for our AI.

🎯 Why we do it​

Rationale

As our AI model gets deeper and deeper, sometimes the math gets messy, and the AI forgets the original words we gave it. To make sure it never forgets the original input, we simply take the original input and ADD it to the new output!

🛠️ How we do it​

Methodology

It's literally just addition! New Answer = Old Input + Attention Output This simple plus sign makes training huge AI models incredibly stable because the original data always has a fast lane to skip the hard math.

# Our original word embeddings
original_input = [10, 20, 30]

# What the complex attention math gave us
attention_output = [2, -1, 5]

# The Save Point (Residual Connection)
safe_output = []
for i in range(3):
    safe_output.append(original_input[i] + attention_output[i])

print("Safe and sound:", safe_output)