Averaging and Smoothing I Video Lecture Transcript
This transcript was automatically generated, so there may be discrepancies between the video and the text.

Hi everybody, welcome back. In this video we're going to talk about averaging and smoothing. This will be one of two videos, so there will be two videos on averaging and smoothing. So.We're going to talk about one of our first types of models are first types of forecasts that aren't just a baseline where essentially you're just going to use an average, not an entire average, but different types of averages of values at previous time steps to predict the next time step. So as a part of this notebook, we'll talk about the concept behind averaging and smoothing. We will define an average model and a weighted average model and this should actually say moving average model.We will then build upon that and learn about exponential smoothing. So we're going to end this video at the end of the weighted average model portion of the notebook, and then in the second video of this series on averaging the smoothing we'll pick up with the exponential smoothing models.So just sort of like the baseline average model and averaging or a smoothing forecast is where you're going to use some kind of average, not necessarily a uniform average, but some kind of average of some kind of collection of previous values to predict future values. So here's an example of a moving average forecast where the prediction and maybe this or this is actually the model. So it's okay to have not the hat but the actual value. So we're going to assume maybe that the.Value at the next time step is equal to the average of the values of the previous three time steps, plus some random noise. So this is a moving average forecast, and specifically this is a moving average forecast because you're shifting the points over.Which you average as T increases. So as T increases the window that you consider the points a part of your average moves along with T so in general, a moving average forecast with a window size K and let's actually I'm going to change this to be a size Q and then let me also change these to be q's as well and we'll see why we want these to be q's in the next notebook.A moving average forecast with window size Q is given as so. So for all the time points where T is less than or equal to n -, 1 T plus one is given by 1 / Q times the sum of the previous Q time points plus some random noise.And then once we get beyond the observed values, we are stuck at 1 / Q times the sum of the values of the Q most recent observations. So we're going to implement this sort of moving average model on our Google stock data. And so again, we're going to look at the last 14 observations as a horizon, or maybe two times the horizon. So this is our test set, and our training set will be all the other observations.So pandas actually has really nice functionality for computing and moving average called rolling. So sometimes you may see a moving average referred to as a rolling average. And so we're going to use the dot rolling functionality of a pandas data frame in order to do this. So here we're going to make a data frame to just show off. So the first column of our data frame is just the closing price, and then the second column is going to be the rolling average of the closing price, so dot.Closing price and we're going to use three just like we did in this example above. So you do dot rolling and then in parentheses you put the number of observations you want to consider in your windows, your window size, which for us will be 3. Then we're going to do we want to add an argument called closed equals left, and So what this does?Is it just ensure? So there's a couple different ways in practice that you could do a rolling average. So you could have your window start at a current observation and go 3 into the future. You could have your current observation be the middle of the window. And so in this case, if say you're at observation 4, you would look at observation 3-4 and five. You could have it any those few different ways. And so by making clothes equal to left that sets your window, it sets your.Current observation as the end point of the window, so the right end. So you'll be using the three most recent values for any time point, and then once we have our rolling set up, this is going to give us a nice little generator that we can then just call dot mean.And so here we can see that the first three observations have NA's because we can't do a three window size 3 rolling average for those because it doesn't have three previous observations. But now we have for the 4th observation, or row three. Since Python starts indexing at zero, we can take the average of the three most recent, and that's what we get with our moving average.And so now that we have this, we can just use this to make our forecast. And that's what this code does. I've already plotted it out, so we're just going to plot. The green dotted line is the training rolling average, so this is what the rolling average does on the training set. And then the red line with filled in circles, that's what is the prediction for the test set.And so here it looks weird, right, because we have this large gap between here, we have to remember that on the X axis or horizontal axis here is the date. These are stock closing prices and so the stock market doesn't trade on the weekend. And so on these days where we don't have dots associated with the points, that's because there was actually no closing price to compare it to on this day because it there was no trading on this day, OK.So as you you know, as you might be able to see from this, the moving average tends to lag behind with the actual data does. So here it says, you know, the green dotted line we can see that it's a little bit higher than what it actually tends to be or maybe it's a little bit lower than what it actually is. So there's a bit of a lag from the signal. So you can see that as you.Use your moving average to. If you have a lot larger and larger window, the little intricacies of the data set itself start to get smoothed out or averaged out and so you can see maybe some trends. And so remember in the baseline we used a trend is just like a straight line, but you can use this moving average model to get like a more local trend which maybe helps combat the fact.