Sudipta Deb
Founder of Technical Potpourri, Co-Founder of Shrey Tech, Enterprise Cloud Architect
Scipy Stats Norm is a component of Scipy Stats utilized for manipulating the normal distribution. The normal distribution is a significant statistical distribution extensively utilized across diverse domains, including finance, physics, and engineering.
Scipy Statistics Norm offers several methods for analyzing the normal distribution, encompassing probability density functions, cumulative distribution functions, and random number generation. These tools facilitate developers' engagement with the standard distribution in Python.
This blog post will elucidate the fundamentals of Scipy Stats Norm and its effective application. I will begin by examining the probability density function and its application in calculating probabilities. Subsequently, I will examine the cumulative distribution function and its application in calculating percentiles. Ultimately, I will examine random number generation and its use in producing random numbers from a normal distribution.
The form you have selected does not exist.
Sample Code
I normally put all my codes in my personal Github repository, so that I can easily share with others. Here is the Github Code Repo where you can find the code samples used in this blog post.
What Is Normal Distribution?
The Normal Distribution is a probability distribution that is symmetric around its mean value. It is a continuous distribution with two parameters: mean (μ) and standard deviation (σ). The bell-shaped curve of the Normal Distribution shows that most of the data falls near the mean, while the tail shows that there is still a chance for data to fall from the mean.
Normal Distribution Using SciPy
Scipy Stats Norm is a submodule of Scipy library that provides methods for working with Normal Distribution. We can use it to create a Normal Distribution with given mean and standard deviation values. We can use the returned object (normal_distribution) to perform various operations related to normal distribution.
# Creating Normal Distribution
# Load SciPy's Statistical Model
from scipy import stats
# Create Mean and Standard Deviation
mean = 35
standard_deviation = 7.2
# Create Normal Distribution
# loc - Mean Value
# scale - Standard Deviation Value
normal_distribution = stats.norm(
loc = mean,
scale = standard_deviation
)Probability Of A Given Value Using pdf()
We can find out the answer to this question - What is the probability that the value is 40? The method pdf() from the norm class can help us to answer this question. It returns the probabilities for specific values from a normal distribution.
PDF = Probability Distribution Function
# Use PDF to understand what is the probability that the value is 40 in this normal distribution
pdf = normal_distribution.pdf(40)
print(pdf)The result is: 8.27133. So that means there is a 8.27133% chance that a randomly selected number has value 40.
Density Curve Using pdf()
A density curve is a graphical plot showing probabilities associated with different values of a variable. We can use pdf() to plot the density curve for the values.
First, we will pick evenly spaced 1000 numbers between 10 and 990.
# Pick some equally spaced 1000 bnumbers between 10 and 990
# Load numpy
import numpy as np
min_value = 10
max_value = 990
equally_spaced_values = np.linspace(min_value, max_value, 1000)
# Let's print the size, first 10 and last 10 values
print("Size: ", equally_spaced_values.size)
print("First 10 Values: ", equally_spaced_values[:10].round(2))
print("Last 10 Values: ", equally_spaced_values[-10:].round(2))Here comes the output:
Size: 1000
First 10 Values: [10. 10.98 11.96 12.94 13.92 14.9 15.89 16.87 17.85 18.83]
Last 10 Values: [981.17 982.15 983.13 984.11 985.1 986.08 987.06 988.04 989.02 990. ]
Now, we can get the probabilities of all these values using pdf() function.
# Probability of all values
all_probabilities = normal_distribution.pdf(x=equally_spaced_values)
print(all_probabilities)Finally, we can plot the value along X-Axis and the corresponding probabilities along Y-Axis. This will give us the density curve.
# Density Curve
# Set grapph size
from matplotlib import pyplot as plt plt.figure(figsize=(20, 12))
plt.plot(equally_spaced_values, all_probabilities)
plt.fill_between(equally_spaced_values, all_probabilities, color="r")
axes=plt.gca()
axes.set_xlim([min_value, max_value])
plt.xlabel("Values")
plt.ylabel("Probabilities")
plt.title("Density Curve")
plt.show()
Find Percent Values Below a Value
Now the question is - What is the percentage that selected number has value less than 450?
To answer this question, we need to add the probabilities of all values less than 450. The norm method cdf() will be able to do that. This method calculated the proportion of a normally distributed population that is less than or equal to a given value.
CDF = Cumulative Distribution Function.
# How many numbers has value less than 450?
less_than_450 = normal_distribution.cdf(450)
print(less_than_450)Output is: 0.21652. This means 0.21652% of records have values less than 450.
Visualize It
We can definitely visualize it by the below way
# Set graph size
plt.figure(figsize=(20, 12))
# plot density function
plt.plot(equally_spaced_values, all_probabilities)
plt.axvline(x=450, color="red")
#Shade the region where the value is less than equal to 450
x1 = np.linspace(equally_spaced_values.min(), 450, 100)
plt.fill_between(x1, norm.pdf(x1, mean, standard_deviation), color="r")
# Remove the dead space
axes = plt.gca()
axes.set_xlim([min_value, max_value])
ymin, ymax = axes.get_ylim()
axes.set_ylim([0, ymax])
plt.title("Value is less than equal 40")
plt.xlabel("Values")
plt.ylabel("Probabilities")
plt.show()
Find Percent Values Above a Value
If the question is - What is the probability that a randomly selected number has a value greater than 450?
To calculate this, we need to subtract the probability that a randomly selected number is less than or equal to 450 from 1. Why 1? The reason is the sum of all probabilities is always 1.
# How many numbers has value less than 450?
less_than_450 = normal_distribution.cdf(450)
print(less_than_450)Since the output is 0.78345, that means there is a 0.78345% chance that the selected number has a value greater than 450.
Let's Visualize It
# Set graph size
plt.figure(figsize=(20, 12))
# plot density function
plt.plot(equally_spaced_values, all_probabilities)
plt.axvline(x=450, color="red")
#Shade the region where the value is less than equal to 450
x1 = np.linspace(450, equally_spaced_values.max(), 100)
plt.fill_between(x1, norm.pdf(x1, mean, standard_deviation), color="r")
# Remove the dead space
axes = plt.gca()
axes.set_xlim([min_value, max_value])
ymin, ymax = axes.get_ylim()
axes.set_ylim([0, ymax])
plt.title("Value is less than equal 40")
plt.xlabel("Values")
plt.ylabel("Probabilities")
plt.show()
Calculate Percentile Using ppf()
Let's say this time you want to find the min value so that 90% of all values are greater than that value. In general, this is called percentile. The Nth percentile is the value that is greater than the N percent of all data points.
We can use the ppf() function to find that value.
PPF = Percent Point Function.
# Find the min value so that 90% of all values are greater than that value
percentile_90th = normal_distribution.ppf(0.9)
print(percentile_90th)Output is: 708.1939. It means 708.1939 is the min value to make sure 90% of all values are greater than this.
Visualize It
# Let's Visualize It
plt.plot(equally_spaced_values, all_probabilities)
plt.axvline(x=percentile_90th, color="red")
plt.xlabel("Values")
plt.ylabel("Probabilities")
plt.show()
Conclusion
I hope this blog post helped you gain practical skills using Python and SciPy. Let’s do a quick recap of what you’ve learned:
- Create normal distributions using the norm class from the SciPy stats module.
- Generate random samples using the norm method rvs().
- Calculate probabilities using pdf() and cdf().
- Find percentiles using ppf().
- Plot histogram and density curve for normal distributions using Matplotlib and Seaborn.
Disclaimer
This article is not endorsed by Salesforce, Google, or any other company in any way. I shared my knowledge on this topic in this blog post. Please always refer to Official Documentation for the latest information.
0 Comments