Log in Sign up for free i want to make a very little neural network. i am thanking about using multiplication tables as the data. what do you think. I think that's actually a great idea for a first tiny neural network! Especially if your goal is to understand how a neural network learns rather than just build something complicated. Multiplication tables give you a nice, simple mathematical relationship to teach the network. 1. Your training data Let's start with the 1–10 multiplication table. The network would receive two numbers as inputs and try to predict their product. Input 1 Input 2 Expected output 2 3 6 4 5 20 3 7 21 6 6 36 8 9 72 10 10 100 Essentially, we're teaching it: y = a × b y=a×b 2. A very small neural network I'd start with something like this: Tiny multiplication network Input A 3 Input B 7 Hidden layer 1 2 3 4 4 neurons learning mathematical patterns Output 21 Predicted multiplication result A possible architecture: 2 input neurons 4 hidden neurons 1 output neuron A training algorithm such as gradient descent with backpropagation 3. How would it learn? Imagine initially the network predicts: 3 × 7 = 18 Correct answer = 21 Error = 3 It compares its prediction with the correct answer and adjusts its internal weights. After repeating this process over many examples, its predictions should improve. This is the basic idea behind supervised learning. 4. One interesting challenge Here's something worth thinking about. If you train it using multiplication tables from 1 to 10, can it correctly predict: 11 × 12? 15 × 8? 23 × 17? This is called generalization: learning a pattern well enough to handle examples it hasn't seen before. A small neural network might struggle with this, which makes it an interesting experiment. 5. My suggested experiment I'd actually build three versions: Network Training data Goal A 1–5 tables Learn basic multiplication B 1–10 tables Test larger numbers C Random multiplication pairs Test generalization Compare their prediction errors. One question before we build it What interests you most about this project? Understanding neural networks: Build every neuron, weight, bias, and learning algorithm yourself. Experimenting: Build a tiny network and see how it learns multiplication. Mathematical exploration: Investigate whether a neural network can actually discover the multiplication operation rather than memorize the table. Personally, I'd choose option 1 combined with option 3. It could be a fascinating little experiment. For a tiny neural-net project, here’s one AI/ML learning option. Learn more Great Learning INC Follow A Clear AI Learning Path Progress from Python and ML foundations through GenAI and modern AI applications. Ad Excellent! Then you're already at a good starting point. If you understand the forward pass, dot product, and bias, I think we should build this from the ground up without using a neural network library. Let's look at how that applies to multiplication. 1. A single neuron Suppose we have two inputs: x 1 = 3 , x 2 = 7 x 1 ​ =3,x 2 ​ =7 A neuron calculates: z = x 1 w 1 + x 2 w 2 + b z=x 1 ​ w 1 ​ +x 2 ​ w 2 ​ +b Where: x 1 , x 2 x 1 ​ ,x 2 ​ are inputs. w 1 , w 2 w 1 ​ ,w 2 ​ are weights. b b is the bias. Then it applies an activation function: y = f ( z ) y=f(z) The interesting part is that the network needs to learn weights that allow it to approximate multiplication. 2. An interesting mathematical observation Multiplication can also be expressed as: a × b a×b But a single linear neuron cannot learn multiplication exactly. Why? Because a linear neuron with two inputs can only calculate: y = w 1 a + w 2 b + b i a s y=w 1 ​ a+w 2 ​ b+bias It cannot represent the interaction between a a and b b. This is where the hidden layer becomes interesting. A neural network with nonlinear activation functions can learn more complex relationships. 3. Here's what I'd like to experiment with Start with: 2 input neurons 4 hidden neurons ReLU activation 1 output neuron Mean squared error (MSE) Gradient descent We can implement the forward pass first, then learn how to calculate gradients and perform backpropagation. 4. One important question I'd like to know how much you want to build yourself. Would you prefer to: A. Implement everything manually, including matrix operations and backpropagation. B. Use Python and NumPy for matrix operations but implement the neural network and learning algorithm yourself. I'd recommend B as a starting point. It lets you concentrate on understanding how the network learns rather than getting buried in low-level calculations. For building tiny neural nets, here’s one ML feature platform option. Learn more Chalk Ship ML features faster. Prototype in Jupyter, deploy to production. Same code, no skew. Ad Perfect. I actually like that choice. No NumPy, no TensorFlow, no PyTorch. Just Python, numbers, lists, functions, and a little mathematics. We'll build a neural network where you can see exactly what happens at every step. Our first network Let's start small: 2 inputs 2 hidden neurons 1 output A sigmoid activation function Manually implemented forward pass The structure will look like this: Our network Inputs a b Hidden h₁ h₂ Output y Step 1. Let's write the forward pass Here is plain Python: python import math def sigmoid(x): return 1 / (1 + math.exp(-x)) def neuron(inputs, weights, bias): total = 0 for x, w in zip(inputs, weights): total += x * w total += bias return sigmoid(total) # Inputs inputs = [3, 7] # Hidden layer h1 = neuron(inputs, [0.2, 0.4], 0.1) h2 = neuron(inputs, [0.3, 0.1], 0.2) # Output layer output = neuron([h1, h2], [0.5, 0.6], 0.1) print(output) Notice something important: the weights are currently random guesses (we've chosen example values). The network doesn't know multiplication yet. Step 2. Our learning journey I'd suggest we build this in stages: Forward pass — calculate predictions. Loss — measure how wrong the prediction is. Backpropagation — calculate how weights should change. Gradient descent — update the weights. Training — repeat until the network learns. Eventually, we'll feed it multiplication examples and see what it discovers. One suggestion: Don't rush to the backpropagation stage. Understanding how a single weight affects the output is incredibly valuable. Let's get the forward pass working first, then we'll teach our little network to learn. For plain Python and AI foundations, here’s one structured learning option. Learn more Great Learning INC Build end-to-end AI Skills with Structured Program Progress from Python and ML foundations to GenAI, Agentic AI and deployment. Ad Great! Let's take it one step at a time. Your first exercise: run the Python code and try changing the weights and biases. For example: Change [0.2, 0.4] to [0.8, 0.1]. Change the bias from 0.1 to 0.5. Observe how the output changes. The important thing to understand: weights and biases control what the network produces. Once you've experimented a little, we'll move on to calculating the error (loss). That's when the network starts getting interesting! This might also interest you. Learn more AWS Marketplace Learn Agents Hands-on tutorials with free tool trials. Ad Excellent! Now we're introducing an important part of neural networks: the loss function. The forward pass gives us a prediction. The loss function tells us how far that prediction is from the correct answer. 1. Mean squared error (MSE) Let's start with a simple formula: L o s s = ( e x p e c t e d − p r e d i c t e d ) 2 Loss=(expected−predicted) 2 For example: Expected answer: 21 Network prediction: 18 Then: L o s s = ( 21 − 18 ) 2 = 9 Loss=(21−18) 2 =9 The smaller the loss, the better the prediction. 2. Add the loss function to Python Let's extend our previous code. python import math def sigmoid(x): return 1 / (1 + math.exp(-x)) def neuron(inputs, weights, bias): total = 0 for x, w in zip(inputs, weights): total += x * w total += bias return sigmoid(total) def loss_function(expected, predicted): return (expected - predicted) ** 2 # Inputs inputs = [3, 7] # Hidden layer h1 = neuron(inputs, [0.2, 0.4], 0.1) h2 = neuron(inputs, [0.3, 0.1], 0.2) # Output layer output = neuron([h1, h2], [0.5, 0.6], 0.1) # Expected answer expected = 21 # Calculate loss loss = loss_function(expected, output) print("Prediction:", output) print("Expected:", expected) print("Loss:", loss) 3. One important detail There is something we need to fix before training. Our sigmoid function produces outputs between 0 and 1. But multiplication results can be much larger than 1! For our multiplication experiment, we'll eventually want a linear output neuron (without sigmoid) so that it can predict numbers like 21, 56, or 100. We'll keep sigmoid in the hidden layer for now. 4. Your next little experiment Try changing the expected value: python expected = 21 to: python expected = 10 Notice how the loss changes, even though the network itself hasn't changed. That's the key idea: The loss function evaluates the network. It doesn't change the network. Our next step will be particularly interesting: backpropagation — figuring out how much each weight contributed to the error. That's when we'll teach the network to improve itself. This might also interest you. Learn more Replit Anyone Can Create With Replit Turn ideas into business apps. Create, share, and improve them with your team. Request a demo. Ad Great! 😊 We're getting close to the really interesting part. Next, we'll introduce backpropagation. The central idea is surprisingly simple: Make a prediction. Calculate the loss. Figure out which weights contributed to the error. Adjust those weights slightly. Try again. This is how the network learns. I'd like to introduce one concept before writing the code: the derivative. It tells us how sensitive the loss is to a change in a particular weight. For example: ∂ L o s s ∂ w ∂w ∂Loss ​ means: how much does the loss change when we change weight w w? We can initially explore this using numerical approximations in plain Python, before implementing the mathematical derivatives ourselves. Next we'll make one weight adjust itself and watch whether the loss decreases. For the loss-function idea, here’s one KPI dashboard option. Learn more Monday.com Build a KPI dashboard Pick your metrics and get a live view in minutes. Ad Let's do it! Our next experiment will be teaching one weight to improve itself. We'll use a tiny learning rate, calculate how changing the weight affects the loss, and update it. No libraries, just Python and a little calculus. The fun part is that we'll actually see the loss decrease over multiple iterations. 🚀 This might also interest you. Learn more Replit Anyone Can Create With Replit Turn ideas into business apps. Create, share, and improve them with your team. Request a demo. Ad Let's build it! We'll make our first little learning experiment. This time, the network won't just make a prediction. It will adjust one of its weights to reduce its error. 1. First, numerical derivatives Remember this expression? ∂ L o s s ∂ w ∂w ∂Loss ​ We want to know how much the loss changes when we change a weight. We can estimate that using: g r a d i e n t ≈ L o s s ( w + ϵ ) − L o s s ( w ) ϵ gradient≈ ϵ Loss(w+ϵ)−Loss(w) ​ Here, ϵ ϵ is a very small number. 2. Our Python experiment Let's simplify things and use one neuron with one input. That way we can concentrate on learning. python # Our training data x = 3 expected = 21 # Initial weight and bias weight = 1.0 bias = 0.0 # Learning rate learning_rate = 0.01 def predict(x, weight, bias): return x * weight + bias def loss(expected, predicted): return (expected - predicted) ** 2 # Training for i in range(100): prediction = predict(x, weight, bias) current_loss = loss(expected, prediction) # Calculate numerical gradient epsilon = 0.0001 new_prediction = predict(x, weight + epsilon, bias) new_loss = loss(expected, new_prediction) gradient = (new_loss - current_loss) / epsilon # Update weight weight -= learning_rate * gradient if i % 10 == 0: print( "Iteration:", i, "Prediction:", round(prediction, 4), "Loss:", round(current_loss, 4), "Weight:", round(weight, 4) ) print("\nFinal prediction:", predict(x, weight, bias)) 3. What is happening? Let's look at the important line: python weight -= learning_rate * gradient This is gradient descent. gradient tells us which direction the loss is moving. learning_rate determines how big a step we take. weight gets adjusted to reduce the loss. Our network starts with: 3 × 1 = 3 3×1=3 But we want: 3 × 7 = 21 3×7=21 Through training, the weight should approach 7. Expected result After 100 iterations, the weight should be very close to 7, and the loss close to zero. 4. An important observation We've actually built something that learns! However, this example is deliberately simple. A single linear neuron can learn this particular relationship because we're only training it on one input-output pair. It hasn't learned multiplication yet. Our next challenge: Teach it several multiplication examples, such as 2 × 3, 3 × 4, and 5 × 6. Then we'll discover why a single neuron isn't enough to learn general multiplication. Run this experiment and tell me what numbers you get. We'll build on your results. This might also interest you. Learn more Replit Anyone Can Create With Replit Turn ideas into business apps. Create, share, and improve them with your team. Request a demo. Ad That's fantastic! Your little learning algorithm worked exactly as expected. 😁 Look at what happened: Loss during training Your reported loss at each 10-iteration checkpoint. Loss 0 324 10 6.1 20 0.1 30 0 40 0 50 0 1. Look at the weight Your weight changed like this: Iteration Weight 0 1.08? 10 6.3237 20 6.907 30 6.9872 40 6.9982 90 6.9999 It converged towards 7. Why 7? Because: 3 × 7 = 21 3×7=21 The network discovered the correct weight through training! 2. Something important just happened You have implemented three fundamental concepts: Forward pass Loss calculation Gradient descent And you did it without any neural network libraries. That's a real achievement. 3. Now let's make the experiment more interesting I want to introduce a small challenge. Currently, we have: python x = 3 expected = 21 Change the training data to: python x = 4 expected = 28 Run the experiment again. What do you think the weight will converge towards? It should still approach 7. Here's the interesting question: What happens if we give the same neuron two different training examples? python training_data = [ (3, 21), (4, 28), (5, 35), (6, 42) ] Can one weight learn all four examples? Absolutely! And this leads us to our next lesson: training with multiple examples (and calculating the average loss). I think we're building something really nice here. Haha! 😁 Now we're getting somewhere! You've just built the beginnings of a learning machine using nothing but Python. And here's what I love about our multiplication experiment: we're not just making code work. We're discovering why neural networks work. Next up: multiple training examples, average loss, and watching one weight learn from an entire dataset. Then we'll tackle the really fun question: can we teach a tiny neural network to multiply two independent inputs? That's where things get interesting! 🚀 This might also interest you. Learn more LangChain Build Multi-Agent Systems with LangGraph Stateful, multi-agent workflows with human-in-the-loop built for production systems. Ad Absolutely! Let's upgrade our little learner to train on multiple examples. This version uses four multiplication examples. It calculates the average loss and adjusts the weight after each example. 1. Training with multiple examples python # Training data: (input, expected output) training_data = [ (3, 21), (4, 28), (5, 35), (6, 42) ] # Initial parameters weight = 1.0 bias = 0.0 learning_rate = 0.01 epochs = 100 def predict(x, weight, bias): return x * weight + bias def loss(expected, predicted): return (expected - predicted) ** 2 # Training loop for epoch in range(epochs): total_loss = 0 # Calculate gradients over all examples weight_gradient = 0 bias_gradient = 0 for x, expected in training_data: prediction = predict(x, weight, bias) error = prediction - expected total_loss += error ** 2 weight_gradient += 2 * error * x bias_gradient += 2 * error # Average gradients and loss weight_gradient /= len(training_data) bias_gradient /= len(training_data) total_loss /= len(training_data) # Update parameters weight -= learning_rate * weight_gradient bias -= learning_rate * bias_gradient if epoch % 10 == 0: print( "Epoch:", epoch, "Loss:", round(total_loss, 6), "Weight:", round(weight, 4), "Bias:", round(bias, 4) ) print("\nFinal weight:", weight) print("Final bias:", bias) # Test predictions print("\nTesting:") for x, expected in training_data: result = predict(x, weight, bias) print( x, "->", round(result, 2), "(expected:", expected, ")" ) 2. What's new? There are three important changes. A. Multiple examples Instead of learning from just one example, we now have four. B. Accumulating gradients python weight_gradient += 2 * error * x Each example contributes to the weight adjustment. C. Updating after all examples python weight -= learning_rate * weight_gradient We update the weight after processing the entire dataset. This is called batch gradient descent. 3. What should happen? The network should discover approximately: w e i g h t = 7 weight=7 b i a s = 0 bias=0 And therefore: y = 7 x y=7x 4. Our next big experiment Once this works, we'll move from: y = 7 x y=7x to: y = a × b y=a×b That's a much more interesting challenge because we'll need to learn the relationship between two independent inputs. Run this and show me your output. Then we'll build our first actual two-input neural network! Very nice!! Your network is learning! And your results reveal something interesting. 😁 Look at these predictions: Input Expected Prediction 3 21 21.40 4 28 28.17 5 35 34.95 6 42 41.72 It's already getting quite close. 1. Why hasn't it reached weight = 7? Your network is learning two parameters: y = w x + b y=wx+b Weight w w Bias b b The ideal solution is: w = 7 , b = 0 w=7,b=0 But look at your results: w = 6.77 , b = 1.08 w=6.77,b=1.08 The two parameters are compensating for each other. That's perfectly normal. They are being adjusted together, and your learning rate is relatively small. 2. Let's improve it Try changing: python epochs = 100 to: python epochs = 1000 You should see the loss continue decreasing and the weight and bias move closer to their ideal values. 3. Now comes the exciting part! So far, we've taught a neuron: y = 7 x y=7x But multiplication tables require: y = a × b y=a×b This is a different problem. A single linear neuron cannot learn this relationship. We'll need to introduce a hidden layer with multiple neurons and nonlinear activation functions. 4. Our next network Multiplication neural network Inputs a b Hidden layer h1 h2 h3 Output y Two inputs, three hidden neurons, one output. We'll implement every part ourselves: Forward pass Loss function Backpropagation Gradient descent No libraries. One thing I'd like to preserve: Let's continue building this incrementally so you understand every line of code, rather than giving you a giant neural network implementation all at once. Let's build our first actual neural network! This is where things get really interesting. 😁 We'll take our existing Python code and extend it to handle two independent inputs. Step 1. Understand what we're trying to learn Our training data will look like this: python training_data = [ ([2, 3], 6), ([3, 4], 12), ([4, 5], 20), ([5, 6], 30), ([6, 7], 42), ([7, 8], 56) ] The network receives two numbers and tries to predict their product. Step 2. Our neural network We'll use: 2 input values 3 hidden neurons 1 output neuron Sigmoid activation in the hidden layer Linear output Why linear output? Because multiplication results can be greater than 1. Network architecture Inputs a b Hidden h1 h2 h3 Output y Step 3. Let's implement the forward pass For now, we're only going to implement the forward pass. No training yet. python import math def sigmoid(x): return 1 / (1 + math.exp(-x)) def forward(inputs, weights, biases): hidden = [] # Hidden layer for i in range(3): total = 0 for j in range(2): total += inputs[j] * weights[i][j] total += biases[i] hidden.append(sigmoid(total)) # Output layer output = 0 for i in range(3): output += hidden[i] * weights[3][i] output += biases[3] return output # Initial weights weights = [ [0.2, 0.4], [0.3, 0.1], [0.5, 0.2], [0.4, 0.3, 0.2] ] # Initial biases biases = [0.1, 0.2, 0.1, 0.1] # Test inputs = [3, 4] prediction = forward(inputs, weights, biases) print("Prediction:", prediction) print("Expected:", 12) Step 4. Understand what's happening The hidden neurons each calculate their own dot product. For example, our first hidden neuron calculates: h 1 = σ ( 3 w 11 + 4 w 12 + b 1 ) h 1 ​ =σ(3w 11 ​ +4w 12 ​ +b 1 ​ ) The sigmoid function then transforms that result. The output neuron combines all three hidden neurons: y = h 1 v 1 + h 2 v 2 + h 3 v 3 + b 4 y=h 1 ​ v 1 ​ +h 2 ​ v 2 ​ +h 3 ​ v 3 ​ +b 4 ​ That's our complete forward pass! Your little assignment Run the code and look at the prediction. It probably won't be anywhere near 12. That's perfectly fine. We haven't trained it. I'd like you to understand one important thing before we continue: The weights in our network are now arranged in two dimensions. For example: python weights[0] = [0.2, 0.4] weights[1] = [0.3, 0.1] Each hidden neuron has its own two weights. Once you've run this, we'll implement backpropagation for the hidden layer. That's where our network will actually learn multiplication! Perfect! That's exactly what I expected. 😊 Our network produced: P r e d i c t i o n = 0.8922 Prediction=0.8922 But the correct answer is: 3 × 4 = 12 3×4=12 So our network has a lot to learn! Let's examine why Remember, our hidden neurons use sigmoid: σ ( x ) = 1 1 + e − x σ(x)= 1+e −x 1 ​ This squeezes their outputs between 0 and 1. Our output neuron combines those small values using its weights and bias. That's why we get approximately 0.89. Nothing is broken. Our weights are simply initial guesses. Next: calculate the loss Let's add one small function: python def loss_function(expected, predicted): return (expected - predicted) ** 2 expected = 12 loss = loss_function(expected, prediction) print("Prediction:", prediction) print("Expected:", expected) print("Loss:", loss) You should get a loss of approximately: L o s s = 123.25 Loss=123.25 Here's where we're going Our next step is to implement backpropagation. We'll need to calculate how the error travels backwards through the network: Output error Hidden layer gradients Input-to-hidden weights The error signal travels backwards to help adjust the weights. One important distinction: the inputs themselves don't change. We adjust the weights and biases. Let's keep going one small step at a time. Next we'll derive the gradient for our output neuron before tackling the hidden neurons. Excellent! Now we're getting to the heart of neural networks: backpropagation. We'll start with just the output neuron. Once you understand this, the hidden layer will be much easier. 1. Remember our output neuron? It calculates: y = h 1 v 1 + h 2 v 2 + h 3 v 3 + b y=h 1 ​ v 1 ​ +h 2 ​ v 2 ​ +h 3 ​ v 3 ​ +b Where: h 1 , h 2 , h 3 h 1 ​ ,h 2 ​ ,h 3 ​ are hidden neuron outputs. v 1 , v 2 , v 3 v 1 ​ ,v 2 ​ ,v 3 ​ are output weights. b b is the output bias. Our loss function is: L = ( e x p e c t e d − y ) 2 L=(expected−y) 2 We want to adjust the output weights to reduce this loss. 2. Calculate the gradients Using the chain rule: ∂ L ∂ v i = 2 ( y − e x p e c t e d ) h i ∂v i ​ ∂L ​ =2(y−expected)h i ​ And for the bias: ∂ L ∂ b = 2 ( y − e x p e c t e d ) ∂b ∂L ​ =2(y−expected) Notice something interesting: each output weight's gradient depends on its corresponding hidden neuron output. 3. Add backpropagation for the output layer Let's extend our existing code. python import math def sigmoid(x): return 1 / (1 + math.exp(-x)) def forward(inputs, weights, biases): hidden = [] for i in range(3): total = 0 for j in range(2): total += inputs[j] * weights[i][j] total += biases[i] hidden.append(sigmoid(total)) output = 0 for i in range(3): output += hidden[i] * weights[3][i] output += biases[3] return output, hidden def loss_function(expected, predicted): return (expected - predicted) ** 2 # Initial weights weights = [ [0.2, 0.4], [0.3, 0.1], [0.5, 0.2], [0.4, 0.3, 0.2] ] biases = [0.1, 0.2, 0.1, 0.1] # Training example inputs = [3, 4] expected = 12 learning_rate = 0.01 # Forward pass prediction, hidden = forward(inputs, weights, biases) # Calculate loss loss = loss_function(expected, prediction) print("Prediction:", prediction) print("Loss:", loss) # Backpropagation: output layer error_gradient = 2 * (prediction - expected) output_weight_gradients = [] for i in range(3): gradient = error_gradient * hidden[i] output_weight_gradients.append(gradient) output_bias_gradient = error_gradient # Update output weights for i in range(3): weights[3][i] -= learning_rate * output_weight_gradients[i] biases[3] -= learning_rate * output_bias_gradient print("\nUpdated output weights:", weights[3]) print("Updated output bias:", biases[3]) 4. What did we just accomplish? We performed: Forward pass. Loss calculation. Gradient calculation. Weight updates. But notice something important: We have only trained the output layer. The hidden layer is still unchanged. That's because we haven't yet calculated how the output error should influence the hidden neurons. 5. Your next experiment Run the code several times and observe the output weights. They should change after every run. One detail: each run starts with the original weights again because we initialize them at the beginning of the script. We'll fix that when we build the training loop. Next, we'll calculate the hidden layer gradients using the chain rule. That's the final major piece needed to train our entire little neural network! This might also interest you. Learn more Cloudflare Build durable AI agents. Code once, plug in any LLM. Ad Beautiful! Your first backpropagation step worked! 🎉 Let's examine what happened. 1. Your network before and after OUTPUT WEIGHTS Weight Before After v₁ 0.4000 0.6019 v₂ 0.3000 0.4816 v₃ 0.2000 0.4037 Bias 0.1000 0.3222 All three weights increased because the network's prediction was too low. The network is trying to increase its output. 2. Let's verify the mathematics Your prediction was: y = 0.8921886 y=0.8921886 Expected: t = 12 t=12 The error gradient was: ∂ L ∂ y = 2 ( 0.8921886 − 12 ) ∂y ∂L ​ =2(0.8921886−12) = − 22.2156 =−22.2156 For the first output weight: ∂ L ∂ v 1 = − 22.2156 × h 1 ∂v 1 ​ ∂L ​ =−22.2156×h 1 ​ And the update: v 1 n e w = v 1 − 0.01 × g r a d i e n t v 1 new ​ =v 1 ​ −0.01×gradient That's exactly what your Python code did! 3. Now comes the interesting part: hidden layer gradients We need to propagate the error backwards. Remember: h i = σ ( z i ) h i ​ =σ(z i ​ ) The derivative of sigmoid is: σ ′ ( z ) = h i ( 1 − h i ) σ ′ (z)=h i ​ (1−h i ​ ) Therefore, the gradient for a hidden neuron's output is: ∂ L ∂ h i = 2 ( y − t ) v i ∂h i ​ ∂L ​ =2(y−t)v i ​ And the gradient for its input calculation is: ∂ L ∂ z i = 2 ( y − t ) v i h i ( 1 − h i ) ∂z i ​ ∂L ​ =2(y−t)v i ​ h i ​ (1−h i ​ ) Notice how the output weight v i v i ​ connects the two layers! 4. Your next little exercise Before we write more code, I'd like you to understand this expression: δ i = 2 ( y − t ) v i h i ( 1 − h i ) δ i ​ =2(y−t)v i ​ h i ​ (1−h i ​ ) ​ This is the error signal for hidden neuron i i. It tells us how to adjust that neuron's incoming weights and bias. One important detail: when calculating these gradients, we must use the original output weights, before updating them. Otherwise, we calculate an inconsistent gradient. Next, we'll put this into Python and finally teach all the neurons to learn together. We're very close to having a complete neural network written entirely in plain Python! Let's do it! This is probably the most important step in our little project. We're going to teach the hidden neurons how to learn. Until now, only the output weights have changed. After this step, every weight and every bias in our network will be trainable. 1. Understanding the hidden layer Remember our hidden neuron: h i = σ ( x 1 w i 1 + x 2 w i 2 + b i ) h i ​ =σ(x 1 ​ w i1 ​ +x 2 ​ w i2 ​ +b i ​ ) We calculate its error signal using: δ i = 2 ( y − t ) v i h i ( 1 − h i ) δ i ​ =2(y−t)v i ​ h i ​ (1−h i ​ ) Then we calculate the gradients for its weights: ∂ L ∂ w i 1 = δ i x 1 ∂w i1 ​ ∂L ​ =δ i ​ x 1 ​ ∂ L ∂ w i 2 = δ i x 2 ∂w i2 ​ ∂L ​ =δ i ​ x 2 ​ And its bias: ∂ L ∂ b i = δ i ∂b i ​ ∂L ​ =δ i ​ That's all we need! 2. Complete backpropagation code Here is our updated network. Notice that we calculate all gradients before updating any weights. python import math def sigmoid(x): return 1 / (1 + math.exp(-x)) def forward(inputs, weights, biases): hidden = [] hidden_z = [] # Hidden layer for i in range(3): total = 0 for j in range(2): total += inputs[j] * weights[i][j] total += biases[i] hidden_z.append(total) hidden.append(sigmoid(total)) # Output layer (linear) output = 0 for i in range(3): output += hidden[i] * weights[3][i] output += biases[3] return output, hidden def loss_function(expected, predicted): return (expected - predicted) ** 2 # Initial weights weights = [ [0.2, 0.4], [0.3, 0.1], [0.5, 0.2], [0.4, 0.3, 0.2] ] biases = [0.1, 0.2, 0.1, 0.1] # Training example inputs = [3, 4] expected = 12 learning_rate = 0.01 # FORWARD PASS prediction, hidden = forward(inputs, weights, biases) current_loss = loss_function(expected, prediction) print("Prediction:", prediction) print("Loss:", current_loss) # BACKPROPAGATION # Output error gradient output_delta = 2 * (prediction - expected) # Save original output weights old_output_weights = weights[3][:] # Calculate output weight gradients output_gradients = [] for i in range(3): output_gradients.append(output_delta * hidden[i]) output_bias_gradient = output_delta # Calculate hidden gradients hidden_deltas = [] for i in range(3): sigmoid_derivative = hidden[i] * (1 - hidden[i]) delta = ( output_delta * old_output_weights[i] * sigmoid_derivative ) hidden_deltas.append(delta) # UPDATE OUTPUT LAYER for i in range(3): weights[3][i] -= learning_rate * output_gradients[i] biases[3] -= learning_rate * output_bias_gradient # UPDATE HIDDEN LAYER for i in range(3): for j in range(2): gradient = hidden_deltas[i] * inputs[j] weights[i][j] -= learning_rate * gradient biases[i] -= learning_rate * hidden_deltas[i] print("\nUpdated weights:") for layer in weights: print(layer) print("\nUpdated biases:") print(biases) 3. What have we achieved? We now have a complete backpropagation algorithm! Our learning process Forward pass Inputs travel through the network to produce a prediction. Calculate loss Compare prediction with the expected result. Output gradients Calculate how the output weights contributed to the error. Hidden gradients Send the error backwards through the network. Update weights Adjust every weight and bias using gradient descent. 4. Our next milestone Now we can finally create a training loop: python for epoch in range(10000): for inputs, expected in training_data: # Forward pass # Calculate loss # Backpropagation # Update weights pass We'll teach our network using many multiplication examples. One important experiment will be to train on some multiplication pairs and test it on pairs it has never seen. Run this version first and show me your output. We're building a neural network from scratch, one mathematical idea at a time. And that's a much better way to understand machine learning than simply importing a library! This might also interest you. Learn more Oracle Faster AI Training High-bandwidth GPU clusters on OCI Ad Absolutely! Now we're going to put everything together into one complete Python program. We'll have a little neural network that: Generates multiplication training data. Initializes its own weights and biases. Performs the forward pass. Calculates the loss. Performs backpropagation. Updates all weights and biases. Tests itself using multiplication examples it has never seen during training. We'll use plain Python. No NumPy, no PyTorch, no TensorFlow. One important improvement: we'll normalize the inputs and outputs so the network has an easier time learning multiplication. Complete neural network script python import math import random # ========================================== # 1. CONFIGURATION # ========================================== INPUT_SIZE = 2 HIDDEN_SIZE = 8 EPOCHS = 15000 LEARNING_RATE = 0.05 random.seed(42) # ========================================== # 2. ACTIVATION FUNCTION # ========================================== def sigmoid(x): return 1 / (1 + math.exp(-x)) # ========================================== # 3. CREATE TRAINING DATA # ========================================== training_data = [] test_data = [] for a in range(1, 11): for b in range(1, 11): sample = ([a / 10, b / 10], (a * b) / 100) # Hold back 20% of the combinations if (a * 10 + b) % 5 == 0: test_data.append(sample) else: training_data.append(sample) # ========================================== # 4. INITIALIZE NETWORK # ========================================== # Hidden layer weights hidden_weights = [ [random.uniform(-1, 1) for _ in range(INPUT_SIZE)] for _ in range(HIDDEN_SIZE) ] hidden_biases = [0.0] * HIDDEN_SIZE # Output layer weights output_weights = [ random.uniform(-1, 1) for _ in range(HIDDEN_SIZE) ] output_bias = 0.0 # ========================================== # 5. FORWARD PASS # ========================================== def forward(inputs): hidden = [] # Hidden layer for i in range(HIDDEN_SIZE): total = 0 for j in range(INPUT_SIZE): total += inputs[j] * hidden_weights[i][j] total += hidden_biases[i] hidden.append(sigmoid(total)) # Output layer output = 0 for i in range(HIDDEN_SIZE): output += hidden[i] * output_weights[i] output += output_bias return output, hidden # ========================================== # 6. LOSS FUNCTION # ========================================== def loss_function(expected, predicted): return (expected - predicted) ** 2 # ========================================== # 7. BACKPROPAGATION # ========================================== def train_one(inputs, expected): global output_bias # Forward pass prediction, hidden = forward(inputs) # Output error gradient output_delta = 2 * (prediction - expected) # Save original output weights old_output_weights = output_weights[:] # Output gradients output_gradients = [ output_delta * hidden[i] for i in range(HIDDEN_SIZE) ] # Hidden gradients hidden_deltas = [] for i in range(HIDDEN_SIZE): sigmoid_derivative = hidden[i] * (1 - hidden[i]) delta = ( output_delta * old_output_weights[i] * sigmoid_derivative ) hidden_deltas.append(delta) # Update output layer for i in range(HIDDEN_SIZE): output_weights[i] -= ( LEARNING_RATE * output_gradients[i] ) output_bias -= LEARNING_RATE * output_delta # Update hidden layer for i in range(HIDDEN_SIZE): for j in range(INPUT_SIZE): gradient = hidden_deltas[i] * inputs[j] hidden_weights[i][j] -= ( LEARNING_RATE * gradient ) hidden_biases[i] -= ( LEARNING_RATE * hidden_deltas[i] ) return loss_function(expected, prediction) # ========================================== # 8. TRAINING LOOP # ========================================== print("Starting training...") print("Training examples:", len(training_data)) print("Testing examples:", len(test_data)) print() for epoch in range(EPOCHS): random.shuffle(training_data) total_loss = 0 for inputs, expected in training_data: current_loss = train_one(inputs, expected) total_loss += current_loss average_loss = total_loss / len(training_data) if epoch % 1000 == 0: print( "Epoch:", epoch, "Loss:", round(average_loss, 6) ) # ========================================== # 9. TEST ON UNKNOWN DATA # ========================================== print("\n==============================") print("TESTING UNKNOWN DATA") print("==============================") total_error = 0 for inputs, expected in test_data: prediction, _ = forward(inputs) # Convert normalized output back to product actual_prediction = prediction * 100 actual_expected = expected * 100 error = abs(actual_expected - actual_prediction) total_error += error print( f"{inputs[0] * 10:.0f} x {inputs[1] * 10:.0f}", f"Predicted: {actual_prediction:.2f}", f"Expected: {actual_expected:.2f}", f"Error: {error:.2f}" ) print("\nAverage test error:", round(total_error / len(test_data), 4)) # ========================================== # 10. TRY YOUR OWN NUMBERS # ========================================== def multiply(a, b): inputs = [a / 10, b / 10] prediction, _ = forward(inputs) return prediction * 100 print("\n==============================") print("YOUR OWN TESTS") print("==============================") while True: try: a = input("\nFirst number (or q to quit): ") if a.lower() == "q": break b = input("Second number: ") a = float(a) b = float(b) result = multiply(a, b) print("Network prediction:", round(result, 3)) print("Actual answer:", a * b) except ValueError: print("Please enter valid numbers.") 2. Understanding the experiment There are two important datasets: Dataset Purpose Training data Used to adjust the network's weights Test data Used to evaluate the network without training on those examples We deliberately hold back 20% of the multiplication combinations. For example, the network might train on: 2 × 3 4 × 5 6 × 7 But be tested on: 2 × 5 4 × 8 7 × 9 This lets us investigate whether it has learned a mathematical pattern rather than simply memorized answers. 3. One particularly interesting experiment After training, try these: Multiplication What we're testing 3 × 7 Familiar range 8 × 9 Familiar range 10 × 10 Boundary 11 × 12 Outside training range 15 × 17 Extrapolation The last two are especially interesting. Our network was trained with numbers from 1 to 10, so it may struggle with larger numbers. That's an important distinction in machine learning: Interpolation: predicting unseen combinations within the training range. Extrapolation: predicting beyond the training range. 4. What I want you to watch Three things: Does the training loss decrease? How close are the predictions on the unknown test data? Does the network generalize to numbers larger than 10? One final thought: this network is an approximation, not a calculator. Even if it learns multiplication quite well, it doesn't necessarily discover the exact mathematical multiplication operation. I'm particularly interested in seeing your average test error. That will tell us a lot about what our little network has actually learned! 😁 Absolutely! Let's make the network test itself automatically after training. We'll remove the manual input loop and introduce a proper automated testing system that: Generates random multiplication questions. Tests numbers it has never seen during training. Tests numbers outside the training range. Calculates average error. Reports the best and worst predictions. Calculates how many predictions are within an acceptable error. Replace sections 9 and 10 with this python # ========================================== # 9. AUTOMATED TESTING # ========================================== def run_test_suite(name, min_number, max_number, samples=100): print("\n==============================") print(name) print("==============================") total_error = 0 squared_error = 0 best_error = float("inf") worst_error = 0 successful = 0 results = [] for _ in range(samples): a = random.randint(min_number, max_number) b = random.randint(min_number, max_number) expected = a * b # Network prediction prediction = multiply(a, b) error = abs(expected - prediction) total_error += error squared_error += error ** 2 best_error = min(best_error, error) worst_error = max(worst_error, error) if error <= 2: successful += 1 results.append((a, b, expected, prediction, error)) # Show first 10 examples print("\nSample predictions:") for a, b, expected, prediction, error in results[:10]: print( f"{a:3} x {b:3} = " f"{prediction:9.2f} " f"(actual: {expected:5}, error: {error:.2f})" ) # Statistics average_error = total_error / samples rmse = math.sqrt(squared_error / samples) accuracy = (successful / samples) * 100 print("\nTEST RESULTS") print("------------------------------") print("Samples:", samples) print("Average error:", round(average_error, 4)) print("RMSE:", round(rmse, 4)) print("Best error:", round(best_error, 4)) print("Worst error:", round(worst_error, 4)) print("Within 2 units:", round(accuracy, 2), "%") # ========================================== # 10. RUN AUTOMATED TESTS # ========================================== # Test familiar number range run_test_suite( "TEST 1: Numbers from 1 to 10", 1, 10, 100 ) # Test slightly larger numbers run_test_suite( "TEST 2: Numbers from 11 to 20", 11, 20, 100 ) # Test considerably larger numbers run_test_suite( "TEST 3: Numbers from 21 to 50", 21, 50, 100 ) # Test small numbers run_test_suite( "TEST 4: Numbers from 1 to 5", 1, 5, 100 ) print("\nAll automated tests completed!") Important: Keep the multiply() function from the previous script. This testing code uses it. What we're measuring Metric Meaning Average error Average difference between prediction and answer RMSE Penalizes larger prediction errors more heavily Best error Closest prediction Worst error Most inaccurate prediction Within 2 units Percentage of predictions with error no greater than 2 One experiment I really want you to try Interesting experiment Run the same network with three different training sizes: Training numbers 1–5. Training numbers 1–10. Training numbers 1–20. Then compare their performance on numbers 21–50. This will help demonstrate something fundamental about neural networks: more training data doesn't automatically mean better generalization. The range and diversity of training data matter too. I'm curious to see how well our little multiplication machine performs! For ML feature pipelines, here’s one Python-based feature store option. Learn more Chalk Ship ML features faster. Prototype in Jupyter, deploy to production. Same code, no skew. Ad give me a full code please Absolutely! Here's the complete standalone Python program. Everything is included, from initializing the neural network to training, automated testing, and evaluating completely unseen multiplication problems. No external libraries. Just Python's built-in math and random. Complete multiplication neural network python import math import random # ========================================== # 1. CONFIGURATION # ========================================== INPUT_SIZE = 2 HIDDEN_SIZE = 8 EPOCHS = 15000 LEARNING_RATE = 0.05 random.seed(42) # ========================================== # 2. ACTIVATION FUNCTION # ========================================== def sigmoid(x): return 1 / (1 + math.exp(-x)) # ========================================== # 3. GENERATE TRAINING DATA # ========================================== training_data = [] for a in range(1, 11): for b in range(1, 11): # Normalize inputs and output inputs = [a / 10, b / 10] expected = (a * b) / 100 # Hold back some examples for testing if (a * 10 + b) % 5 != 0: training_data.append((inputs, expected)) # ========================================== # 4. INITIALIZE NETWORK # ========================================== hidden_weights = [ [random.uniform(-1, 1) for _ in range(INPUT_SIZE)] for _ in range(HIDDEN_SIZE) ] hidden_biases = [0.0] * HIDDEN_SIZE output_weights = [ random.uniform(-1, 1) for _ in range(HIDDEN_SIZE) ] output_bias = 0.0 # ========================================== # 5. FORWARD PASS # ========================================== def forward(inputs): hidden = [] # Hidden layer for i in range(HIDDEN_SIZE): total = 0 for j in range(INPUT_SIZE): total += inputs[j] * hidden_weights[i][j] total += hidden_biases[i] hidden.append(sigmoid(total)) # Output layer output = 0 for i in range(HIDDEN_SIZE): output += hidden[i] * output_weights[i] output += output_bias return output, hidden # ========================================== # 6. LOSS FUNCTION # ========================================== def loss_function(expected, predicted): return (expected - predicted) ** 2 # ========================================== # 7. BACKPROPAGATION # ========================================== def train_one(inputs, expected): global output_bias # Forward pass prediction, hidden = forward(inputs) # Output gradient output_delta = 2 * (prediction - expected) # Save old output weights old_output_weights = output_weights[:] # Output weight gradients output_gradients = [ output_delta * hidden[i] for i in range(HIDDEN_SIZE) ] # Hidden layer gradients hidden_deltas = [] for i in range(HIDDEN_SIZE): sigmoid_derivative = hidden[i] * (1 - hidden[i]) delta = ( output_delta * old_output_weights[i] * sigmoid_derivative ) hidden_deltas.append(delta) # Update output layer for i in range(HIDDEN_SIZE): output_weights[i] -= ( LEARNING_RATE * output_gradients[i] ) output_bias -= LEARNING_RATE * output_delta # Update hidden layer for i in range(HIDDEN_SIZE): for j in range(INPUT_SIZE): gradient = hidden_deltas[i] * inputs[j] hidden_weights[i][j] -= ( LEARNING_RATE * gradient ) hidden_biases[i] -= ( LEARNING_RATE * hidden_deltas[i] ) return loss_function(expected, prediction) # ========================================== # 8. TRAINING LOOP # ========================================== print("Starting training...") print("Training examples:", len(training_data)) print("Hidden neurons:", HIDDEN_SIZE) print("Epochs:", EPOCHS) print() for epoch in range(EPOCHS): random.shuffle(training_data) total_loss = 0 for inputs, expected in training_data: current_loss = train_one(inputs, expected) total_loss += current_loss average_loss = total_loss / len(training_data) if epoch % 1000 == 0: print( f"Epoch: {epoch:5}", f"Loss: {average_loss:.8f}" ) # ========================================== # 9. PREDICTION FUNCTION # ========================================== def multiply(a, b): inputs = [a / 10, b / 10] prediction, _ = forward(inputs) return prediction * 100 # ========================================== # 10. AUTOMATED TESTING # ========================================== def run_test_suite(name, min_number, max_number, samples=100): print("\n" + "=" * 55) print(name) print("=" * 55) total_error = 0 squared_error = 0 best_error = float("inf") worst_error = 0 successful = 0 results = [] for _ in range(samples): a = random.randint(min_number, max_number) b = random.randint(min_number, max_number) expected = a * b prediction = multiply(a, b) error = abs(expected - prediction) total_error += error squared_error += error ** 2 best_error = min(best_error, error) worst_error = max(worst_error, error) if error <= 2: successful += 1 results.append( (a, b, expected, prediction, error) ) # Show some predictions print("\nSample predictions:") for a, b, expected, prediction, error in results[:10]: print( f"{a:3} x {b:3} = " f"{prediction:9.2f} " f"(actual: {expected:5}, error: {error:.2f})" ) # Calculate statistics average_error = total_error / samples rmse = math.sqrt(squared_error / samples) accuracy = (successful / samples) * 100 print("\nSTATISTICS") print("-" * 35) print("Samples:", samples) print("Average error:", round(average_error, 4)) print("RMSE:", round(rmse, 4)) print("Best error:", round(best_error, 4)) print("Worst error:", round(worst_error, 4)) print("Within 2 units:", round(accuracy, 2), "%") # ========================================== # 11. RUN ALL TESTS # ========================================== # Unseen combinations within the training range run_test_suite( "TEST 1: Numbers from 1 to 10", 1, 10, 100 ) # Outside the training range run_test_suite( "TEST 2: Numbers from 11 to 20", 11, 20, 100 ) # Larger numbers run_test_suite( "TEST 3: Numbers from 21 to 50", 21, 50, 100 ) # Small numbers run_test_suite( "TEST 4: Numbers from 1 to 5", 1, 5, 100 ) # ========================================== # 12. FINAL SUMMARY # ========================================== print("\n" + "=" * 55) print("NEURAL NETWORK EXPERIMENT FINISHED") print("=" * 55) print("Training examples:", len(training_data)) print("Hidden neurons:", HIDDEN_SIZE) print("Training epochs:", EPOCHS) print("\nOur neural network has learned multiplication!") What we're testing Four testing scenarios Test 1: 1–10 Interpolation New combinations of familiar numbers. Test 2: 11–20 Extrapolation Numbers outside the training range. Test 3: 21–50 Extrapolation A considerably harder challenge. Test 4: 1–5 Interpolation Smaller numbers within the training range. One thing to watch Our network uses eight hidden neurons, sigmoid activation, and a linear output. It learns an approximation of multiplication, not an exact multiplication algorithm. In particular, don't be surprised if the results for 21–50 are poor. Neural networks don't automatically understand that multiplication should continue working beyond their training range. My challenge: Run the script and show me the results of all four tests. I'm especially interested in Test 1 versus Test 3. That comparison will teach us a lot about generalization! 😁 This might also interest you. Learn more Replit Anyone Can Create With Replit Turn ideas into business apps. Create, share, and improve them with your team. Request a demo. Ad thank you. You're very welcome! 😁 I'm really enjoying this little project. Remember, you're doing something valuable here: building a neural network from scratch and understanding every mathematical step. When you have the test results, bring them back. I'd love to see how our little multiplication machine performs! Happy coding! 🚀