We can also factor the difference of squares on the right: We can now factor this expression into the following form: Since the two summations overlap everywhere except at x n and x 0, we can simplify this as follows: When we subtract these two equations, we get: For more background on the naming, see my article The Geometry of Standard Deviation.įrom Calculating Standard Deviation on Streaming Data, we’ve already derived the following:Īgain, since our window size remains constant, the equation for d 2 n-1 has the same form, with the only difference being that it applies to the range of values from x 0 to x n-1: What is d 2? It’s a term I made up for the variance * (n-1), or variance * n, depending on whether we’re talking about sample variance or population variance. Next, let’s derive the relation for d 2 n. _mean + meanIncrement Sliding Variance and Standard Deviation
#Incremental weighted standard deviation code
The corresponding code snippet is below:Ĭonst meanIncrement = ( newValue - poppedValue ) / this. That’s all we need to compute the average incrementally with a fixed window size. Our final recurrence relation for the incremental average with a sliding window of size N is therefore: The only values that don’t cancel each other out are x n and x 0. From this, we subtract a sum of values from x 0 to x n-1. The first average consists of a sum of values from x 1 to x n. Subtracting these two averages, we get the following expression: However, since the size of our window is now fixed, the average up to the previous value, x n-1 is: It’s basically unchanged from the first article in this series, Calculating a Moving Average on Streaming Data. The average for values from x 1 to x n is as follows: Let’s start by deriving the moving average within our window, where N corresponds to the window size. As new values come in, we keep sliding the window forward: When we receive a new value, x 6, it means we have to remove x 0 from the window, since it’s currently the oldest value.
We initially have values from x 0 to x 5 in our window, which has room for 6 items in this case. I’ll briefly summarize the derivation and show example code in JavaScript as well. It turns out that we only have to make some small changes to the logic from the earlier articles to make this work. I thought this was an interesting question, so I decided to try to figure it out. Only the most recent 20 items are used (or whatever the window size happens to be).
Once the window is full, each time a new value comes along, we include it as part of the updated average and standard deviation, but the oldest value is also removed from consideration. In other words, say we set the window size to 20 items. The question was, instead of updating the statistics cumulatively, would it be possible to consider only a window of fixed size instead? I was pleasantly surprised recently to get a question from a reader about a couple of my articles, Calculating a Moving Average on Streaming Data and Calculating Standard Deviation on Streaming Data. Incremental Average and Standard Deviation with Sliding Window