Month: September 2023

The formula of the cost function is given by

J(mi)=1/2n*(sum of (Ypi-Yi)^2 till i reaches n)

where n is the number of data points

As we can see the cost function in the graph reduces till it reaches the minimum value.

This minimum value is the least error in our Linear Regression.

We need to find this minimum value

In order to find the minimum value we will use something called a Gradient Descent Algorithm or Convergence Algorithm.

So Formula for the convergence algorithm is

do{

M(new)=M(old)-(Learning Rate Alpha)*(partial derivative with respect to slope m)J(m)

Calculate cost function J(M(new)) and J(M(old))

}while(J(M(new))>J(M(old))

After the above algorithm, the least value of the cost function/Least error function is J(M(old)), and the slope for which the error is least is M(old)

Now the line will be drawn to the nearest data points with the equation

Y=M(old)Xi

where Xi is x points or an independent variable

We will be given data points let’s say Diabetes vs Obesity.

So as per the CDC data, we have the percentage of people suffering from diabetes for each country which is differentiated by FIPS number.

What we can do is to check whether obesity is affecting diabetes or not.

So obesity will be the independent variable and diabetes will be the dependent variable.

In order to plot linear regression for diabetes against obesity.

Let’s see if we have both the points for common areas

X(Obesity)={Xp1,Xp2,Xp3……..}

Y(Diabetes)={Yp1,Yp2,Yp3,……..}

Now we must find a line that best represents the above values

line equation-.

Yi=mXi+c

where c is the y-intercept. Let’s say we draw a line from the origin so c=0

Yi=mXi

Now we have to select different slopes {m1,m2,m3……mi} or select different angles from where the line is to be drawn in the first quadrant

After that, we have to calculate the mean difference of points from the line we draw that is {Y1, Y2, Y3, Y4} to points that we obtain from the data set here it is Obesity percentage {Yp1, Yp2, Yp3, Yp4…..}

This mean difference is called a Cost function/Least Error Squared Function/Residual Error

The formula of the cost function is given by

J(mi)=1/2n*(sum of (Ypi-Yi)^2 till i reaches n)

where n is the number of points

To know linear regression we need to understand why it came and what it does as a whole.

When we get any data set let’s say how many people are obese vs. inactive.

We will plot a graph where the x-axis is the percentage of people inactive whereas the y-axis is the percentage of people obese. What we will get is a graph full of points.

So linear regression is a machine learning algorithm that is used to predict future anomalies. So in order to predict we will draw a straight line based on current data and see if our line is closest to the data points that are already present. If it is then we can say that for the changing x-axis the point we will get in line will be the best possible prediction.

As it’s been seen the red line is predicted linear regression which is nearer to the data points and hence for the data points or let’s say future x value the predicted y value will be nearest to be accurate.

So in order to find this line we can use the squared error function(Cost Function) and minimize the value. The minimum value will be the line we are looking for.

Here hypothesis is the line equation we are looking for. Which looks like y=mx+c.

In order to learn the usage of matplotlib.pyplot, I have plotted a graph with Google Chrome’s CPU Usage Vs. Time

Code->

import matplotlib.pyplot as plt

x=[]

y=[]

for line in open('CPUData.dat','r'):

lines = [i for i in line.split()]

x.append(int(lines[0]))

y.append(float(lines[1]))

plt.title("CPU_UsageVSTime")

plt.xlabel("Time")

plt.ylabel("CPU_Usage")

plt.plot(x,y,marker='o',c='g')

plt.show()

5 Measures in statistics

1.  The measure of central tendency
2. Measure of dispersion
3. Gaussian Distribution
4. Z-score
5. Standard Normal Distribution

Central Tendency– Refers to the measure used to determine the center of distribution of data. To measure there are 3 terms Mean, Median, and Mode.

• Mean is the average of all the data. That is the sum of data by the number of data
• Median in statistics and probability theory, the median is the value separating the higher half from the lower half of a data sample, a population, or a probability distribution.
• Mode is the most frequent number—that is, the number that occurs the highest number of times.

The measure of Dispersion refers to how data is been scattered around the central tendency. In order to measure dispersion. We calculate two quantities: variance and standard deviation.

• Standard Deviation in statistics is a measure of the amount of variation or dispersion of a set of values. A low standard deviation indicates that the values tend to be close to the mean of the set, while a high standard deviation indicates that the values are spread out over a wider range.
• Variance is the expectation of the squared deviation from the mean of a random variable. The standard deviation is obtained as the square root of the variance.

Outliers- In statistics, an outlier is a data point that differs significantly from other observations. An outlier may be due to variability in the measurement, an indication of novel data, or maybe the result of experimental error; the latter are sometimes excluded from the data set.

For example, there  are 10 numbers

{1,1,2,2,3,3,4,5,5,6}

Mean=Sum of observations/Number of observations=3.2

Now suppose we add a number that is very large to the given set of numbers let’s say 100

{1,1,2,2,3,3,4,4,5,6,100}

Now Mean comes out to be 12

Previous Mean =3.2

Mean due to the presence of an outlier=12

As we can see due to presence of an outlier, mean value is signicantly changed. So in order to make correct calcualtions on data, outliers should be removed as far as possible. However the middle value or median won’t have any effect due to presence of an outlier.

---

Percentile It is a value below which a certain percentage of observation lie.

for example->

Dataset:2,2,3,4,5,5,5,6,7,8,8,8,8,9,9,10,11,11,12

What is the percentile of 10?

Percentile rank of 10 =16/20 * 100=80%ile

In order to remove outliers there is Five number summary

1. Minimimum
2. First Quartile(Q1)
3. Median
4. Third Quartile(Q3)
5. Maximum

Minimum and Maximum values can define the range of data set.

While, The lower quartile, or first quartile (Q1), is the value under which 25% of data points are found when they are arranged in increasing order. The upper quartile, or third quartile (Q3), is the value under which 75% of data points are found when arranged in increasing order.

In order to remove outlier’s we follow following steps

1. IQR(Interquartile Range)=Q3-Q1
2. Lower Fence=Q1-1.5(IQR)
3. Upper Fence=Q3+1.5(IQR)

So any value which above and below Lower and Upper Fence is an outlier. Which could be removed

Note- NumPy doesn’t provide calcuations of mode as mode is just counting of occurences while NumPy is used for mathematical calculations.

Link to test all the above functions

Below is the linear regression result, in which the estimated coefficients are->:
b_0 = 2.055980432211423
b_1 = 0.27828827666358774

A Python program written to obtain the above regression is