Perceptrons & Basics | Machine Learning Guide | Ephizen

The Perceptron#

The perceptron is the simplest neural network - a single neuron that makes decisions.

Historical Note

Invented by Frank Rosenblatt in 1958, the perceptron sparked the field of neural networks.

How It Works#

Inputs → Weighted Sum → Activation → Output

x₁ ──(w₁)──┐
           │
x₂ ──(w₂)──┼──→ Σ ──→ f(x) ──→ output
           │
x₃ ──(w₃)──┘
           ↑
         bias

Mathematically:

output = f(w₁x₁ + w₂x₂ + w₃x₃ + b)

Building a Perceptron from Scratch#

python

import numpy as np

class Perceptron:
    def __init__(self, input_size, learning_rate=0.1):
        self.weights = np.zeros(input_size)
        self.bias = 0
        self.lr = learning_rate

    def activation(self, x):
        """Step function: 1 if x >= 0, else 0"""
        return 1 if x >= 0 else 0

    def predict(self, x):
        """Forward pass"""
        linear = np.dot(self.weights, x) + self.bias
        return self.activation(linear)

    def train(self, X, y, epochs=100):
        """Train using perceptron learning rule"""
        for _ in range(epochs):
            for xi, yi in zip(X, y):
                prediction = self.predict(xi)
                error = yi - prediction

                # Update weights and bias
                self.weights += self.lr * error * xi
                self.bias += self.lr * error

Training Example: OR Gate#

python

# OR gate: output 1 if any input is 1
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([0, 1, 1, 1])

perceptron = Perceptron(input_size=2)
perceptron.train(X, y, epochs=10)

# Test
for xi in X:
    print(f"{xi} -> {perceptron.predict(xi)}")

# Output:
# [0, 0] -> 0
# [0, 1] -> 1
# [1, 0] -> 1
# [1, 1] -> 1

The Limitation: XOR Problem#

Famous Limitation

A single perceptron cannot solve XOR (exclusive or). This limitation nearly killed neural network research for decades.

python

# XOR: output 1 if inputs are different
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([0, 1, 1, 0])  # XOR truth table

perceptron = Perceptron(input_size=2)
perceptron.train(X, y, epochs=100)

# This will NOT learn XOR correctly!
# Perceptron can only learn linearly separable patterns

Why XOR Fails#

Linearly Separable?

Loading chart...

The perceptron draws a straight line. XOR points can't be separated by a line.

The Solution: Multi-Layer Networks#

Stack perceptrons to learn complex patterns:

python

import torch
import torch.nn as nn

class XORNetwork(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(2, 4)   # Hidden layer
        self.layer2 = nn.Linear(4, 1)   # Output layer
        self.activation = nn.ReLU()

    def forward(self, x):
        x = self.activation(self.layer1(x))
        x = torch.sigmoid(self.layer2(x))
        return x

# This CAN learn XOR!
model = XORNetwork()

From Perceptron to Neural Network#

Single Perceptron

Can only learn linearly separable patterns

Hidden Layers

Add layers between input and output

Non-linear Activation

Replace step function with smooth functions

Backpropagation

Algorithm to train multi-layer networks

Key Components Comparison#

Feature	Component	Perceptron
Layers	Single	Multiple
Activation	Step function	ReLU, Sigmoid, etc.
Training	Perceptron rule	Backpropagation
Capability	Linear only	Any function

Key Takeaways#

➕

Weighted Sum

Inputs are multiplied by weights and summed. Core of all neural networks.

⚡

Activation Function

Introduces non-linearity. Without it, layers collapse to linear.

📚

Learning

Adjust weights based on errors. Small steps toward the goal.

🚫

Limitations

Single layer = linear only. Stack layers to learn anything.

The Foundation

Understanding the perceptron is crucial. Every neuron in modern deep learning is an evolved version of this simple idea: weighted sum, activation, output.

Inputs → Weighted Sum → Activation → Output x₁ ──(w₁)──┐ │ x₂ ──(w₂)──┼──→ Σ ──→ f(x) ──→ output │ x₃ ──(w₃)──┘ ↑ bias

Building a Perceptron from Scratch#

python

import numpy as np

class Perceptron:
    def __init__(self, input_size, learning_rate=0.1):
        self.weights = np.zeros(input_size)
        self.bias = 0
        self.lr = learning_rate

    def activation(self, x):
        """Step function: 1 if x >= 0, else 0"""
        return 1 if x >= 0 else 0

    def predict(self, x):
        """Forward pass"""
        linear = np.dot(self.weights, x) + self.bias
        return self.activation(linear)

    def train(self, X, y, epochs=100):
        """Train using perceptron learning rule"""
        for _ in range(epochs):
            for xi, yi in zip(X, y):
                prediction = self.predict(xi)
                error = yi - prediction

                # Update weights and bias
                self.weights += self.lr * error * xi
                self.bias += self.lr * error

Training Example: OR Gate#

python

# OR gate: output 1 if any input is 1
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([0, 1, 1, 1])

perceptron = Perceptron(input_size=2)
perceptron.train(X, y, epochs=10)

# Test
for xi in X:
    print(f"{xi} -> {perceptron.predict(xi)}")

# Output:
# [0, 0] -> 0
# [0, 1] -> 1
# [1, 0] -> 1
# [1, 1] -> 1

The Limitation: XOR Problem#

Famous Limitation

A single perceptron cannot solve XOR (exclusive or). This limitation nearly killed neural network research for decades.

python

# XOR: output 1 if inputs are different
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([0, 1, 1, 0])  # XOR truth table

perceptron = Perceptron(input_size=2)
perceptron.train(X, y, epochs=100)

# This will NOT learn XOR correctly!
# Perceptron can only learn linearly separable patterns

The Solution: Multi-Layer Networks#

Stack perceptrons to learn complex patterns:

python

import torch
import torch.nn as nn

class XORNetwork(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(2, 4)   # Hidden layer
        self.layer2 = nn.Linear(4, 1)   # Output layer
        self.activation = nn.ReLU()

    def forward(self, x):
        x = self.activation(self.layer1(x))
        x = torch.sigmoid(self.layer2(x))
        return x

# This CAN learn XOR!
model = XORNetwork()

Feature	Component	Perceptron	Modern NN
Layers	Single	Multiple
Activation	Step function	ReLU, Sigmoid, etc.
Training	Perceptron rule	Backpropagation
Capability	Linear only	Any function

Feature

Component

Perceptron

Modern NN

Layers

Single

Multiple

Activation

Step function

ReLU, Sigmoid, etc.

Training

Perceptron rule

Backpropagation

Capability

Linear only

Any function

Key Takeaways#

➕

Weighted Sum

Inputs are multiplied by weights and summed. Core of all neural networks.

⚡

Activation Function

Introduces non-linearity. Without it, layers collapse to linear.

📚

Learning

Adjust weights based on errors. Small steps toward the goal.

🚫

Limitations

Single layer = linear only. Stack layers to learn anything.

The Foundation

Understanding the perceptron is crucial. Every neuron in modern deep learning is an evolved version of this simple idea: weighted sum, activation, output.

The Perceptron#

How It Works#

Building a Perceptron from Scratch#

Training Example: OR Gate#

The Limitation: XOR Problem#

Why XOR Fails#

Linearly Separable?

The Solution: Multi-Layer Networks#

From Perceptron to Neural Network#

Single Perceptron

Hidden Layers

Non-linear Activation

Backpropagation

Key Components Comparison#

Key Takeaways#

Weighted Sum

Activation Function

Learning

Limitations

Ready to level up your skills?

The Perceptron#

How It Works#

Building a Perceptron from Scratch#

Training Example: OR Gate#

The Limitation: XOR Problem#

Why XOR Fails#

Linearly Separable?

The Solution: Multi-Layer Networks#

From Perceptron to Neural Network#

Single Perceptron

Hidden Layers

Non-linear Activation

Backpropagation

Key Components Comparison#

Key Takeaways#

Weighted Sum

Activation Function

Learning

Limitations

Ready to level up your skills?