Mathematics for Machine Learning - Day 4

Particular and General Solutions

Ever wonder, if there's an equation with two or more variables that aren't known and there's not enough information to use basic elimination method? Fear not, there's a solution.

Also, this topic will only discuss on depth regarding particular and general solutions as I found that it takes me even more than an hour to really understand this and I find it very fascinating :D

Particular Solutions

When a system has more than one solution, it's called a particular/special solution. One of the solution is if there are columns with only ones and zeros, the position of ones are unique in rows and columns, and ones are contained in all the rows.

Example:

$$Ax = B$$ $$\begin{pmatrix} 1 & 0 & 8 & 0 & -4 \\\ 0 & 1 & 2 & 0 & -12 \\\ 0 & 0 & 4 & 1 & 7 \end{pmatrix} \begin{pmatrix} x_1 \\\ x_2 \\\ x_3 \\\ x_4 \\\ x_5 \end{pmatrix} = \begin{pmatrix} 42 \\\ 8 \\\ 12 \end{pmatrix}$$

Now, you might find this to be impossible to solve, but let's break it down each matrix at a time.

Take a look at matrix A:

$$A = \begin{pmatrix} 1 & 0 & 8 & 0 & -4 \\\ 0 & 1 & 2 & 0 & -12 \\\ 0 & 0 & 4 & 1 & 7 \end{pmatrix}$$

you'll find that the 1st, 2nd, and 4th column if isolated becomes an identity matrix or as I've stated before, a unique ones in all rows and zeros.

$$A(Reordered) = \begin{pmatrix} 1 & 0 & 0 & 8 & -4 \\\ 0 & 1 & 0 & 2 & -12 \\\ 0 & 0 & 1 & 4 & 7 \end{pmatrix}$$

While B can become be decoupled to each row:

$$\begin{pmatrix} 42 \\\ 8 \\\ 12 \end{pmatrix} = 42 \begin{pmatrix} 1 \\\ 0 \\\ 0 \end{pmatrix} + 8 \begin{pmatrix} 0 \\\ 1 \\\ 0 \end{pmatrix} + 12 \begin{pmatrix} 0 \\\ 0 \\\ 1 \end{pmatrix}$$

Meaning:

If we ignore the 3rd and 5th column entirely, we can find the particular solution!

$$\begin{pmatrix} 1 & 0 & 8 & 0 & -4 \\\ 0 & 1 & 2 & 0 & -12 \\\ 0 & 0 & 4 & 1 & 7 \end{pmatrix} \begin{pmatrix} 42 \\\ 8 \\\ 0 \\\ 12 \\\ 0 \end{pmatrix} = \begin{pmatrix} 42 \\\ 8 \\\ 12 \end{pmatrix}$$ $$\begin{pmatrix} 42 + 0 + 0 + 0 + 0 \\\ 0 + 8 + 0 + 0 + 0 \\\ 0 + 0 + 0 + 12 + 0 \end{pmatrix} = \begin{pmatrix} 42 \\\ 8 \\\ 12 \end{pmatrix}$$

Congratulations! You've found the particular solution!

$$\therefore \begin{pmatrix} x_1 \\\ x_2 \\\ x_3 \\\ x_4 \\\ x_5 \end{pmatrix} = \begin{pmatrix} 42 \\\ 8 \\\ 0 \\\ 12 \\\ 0 \end{pmatrix}$$

---

General Solutions

Now this is where it took me an hour to understand what they were saying, so let's dive into it.

General solutions are basically adding in 0 to the equation. Instead of

$$Ax = B$$

We'll be adding in 0 to the first section based on the amount of non-identity matrix related columns (In our case it's two, the 3rd and 5th column)

$$A x + \lambda_1 \cdot 0 + \lambda_2 \cdot 0 = B$$

The lambda is used because since it's multiplied by 0, any scaler added will still remain zero, so it covers all values added to it.

First General Solution (3rd Row)

$$\begin{pmatrix} 8 \\\ 2 \\\ 4 \end{pmatrix}$$

Using the same decouple method as before:

$$\begin{pmatrix} 8 \\\ 2 \\\ 4 \end{pmatrix} = 8 \begin{pmatrix} 1 \\\ 0 \\\ 0 \end{pmatrix} + 2 \begin{pmatrix} 0 \\\ 1 \\\ 0 \end{pmatrix} + 4 \begin{pmatrix} 0 \\\ 0 \\\ 1 \end{pmatrix}$$

You'll remember that this decouple when added to x in the order of the identity matrix will become the answer. Meaning:

$$\text{If } \begin{pmatrix} x_1 \\\ x_2 \\\ x_3 \\\ x_4 \\\ x_5 \end{pmatrix} = \begin{pmatrix} 8 \\\ 2 \\\ 0 \\\ 4 \\\ 0 \end{pmatrix} \text{, then } B = \begin{pmatrix} 8 \\\ 2 \\\ 4 \end{pmatrix}$$

So how do I make it zero?

Easy! the column that's associated to the matrix (In our case is the 3rd) we make it -1.

$$\begin{pmatrix} x_1 \\\ x_2 \\\ x_3 \\\ x_4 \\\ x_5 \end{pmatrix} = \begin{pmatrix} 8 \\\ 2 \\\ -1 \\\ 4 \\\ 0 \end{pmatrix}$$

Here's the proof

$$\begin{pmatrix} 1 & 0 & 8 & 0 & -4 \\\ 0 & 1 & 2 & 0 & -12 \\\ 0 & 0 & 4 & 1 & 7 \end{pmatrix} \begin{pmatrix} 8 \\\ 2 \\\ -1 \\\ 4 \\\ 0 \end{pmatrix}$$ $$\begin{pmatrix} 8 + 0 - 8 + 0 + 0 \\\ 0 + 2 - 2 + 0 + 0 \\\ 0 + 0 - 4 + 4 + 0 \end{pmatrix} = \begin{pmatrix} 0 \\\ 0 \\\ 0 \end{pmatrix}$$

Now we add the first answer to our general solution!

$$A x + \lambda_1 \cdot \begin{pmatrix} 8 \\\ 2 \\\ -1 \\\ 4 \\\ 0 \end{pmatrix} + \lambda_2 \cdot 0 = B$$

Second General Solution (5th Row)

Surprize Quiz!

Now that you know how to find the equation for the general solution, try finding the second one yourself! The answer is at the bottom so you can check if you're correct or not.

$$\begin{pmatrix} -4 \\\ -12 \\\ 7 \end{pmatrix}$$

---

The full equation.

We've found the particular solutions that's needed to create the general solution!

For clarity, let's discuss what we did:

1. Find the particular solution to Ax = B
2. Find all the solutions to Ax = 0
3. Add steps one and two together

With these steps finished, congratulations! you've become familiar with particular and general solutions. From here on out, you know that there are cases where large unknown variables with little equations can be solved with the right conditions.

The general solution

$$\left\{ x \in \mathbb{R}^5 \mid x = \begin{pmatrix} 42 \\\ 8 \\\ 0 \\\ 12 \\\ 0 \end{pmatrix} + \lambda_1 \begin{pmatrix} 8 \\\ 2 \\\ -1 \\\ 4 \\\ 0 \end{pmatrix} + \lambda_2 \begin{pmatrix} -4 \\\ -12 \\\ 0 \\\ 7 \\\ -1 \end{pmatrix}, \; \lambda_1, \lambda_2 \in \mathbb{R} \right\}$$

P.S. Hope you got the answer right!

---

Acknowledgement

I can't overstate this: I'm truly grateful for this book being open-sourced for everyone. Many people will be able to learn and understand machine learning on a fundamental level. Whether changing careers, demystifying AI, or just learning in general, this book offers immense value even for fledgling composer such as myself. So, Marc Peter Deisenroth, A. Aldo Faisal, and Cheng Soon Ong, thank you for this book.

Source: Deisenroth, M. P., Faisal, A. A., & Ong, C. S. (2020). Mathematics for Machine Learning. Cambridge: Cambridge University Press. https://mml-book.com