Why does my reduce based average function return NaN?

Your program didn't work because, a has the accumulated value from the previous function call. The first time, first two values of the array will be used. So sum will become 17 (2 + 15). Since you are not returning anything from the function, undefined will be returned, by default, and that will be used as the value for a, in the next call. So, the evaluation goes like this

a: 2,          b: 15   => 17
a: undefined,  b: 7    => NaN

So, sum will have NaN, since undefined + 7 makes it so. Any numeric operation on NaN, will always give NaN, that is why NaN / this.length, gives you NaN. You can fix your program, just by returning the current value of sum whenever the function is called, so that on the next function call, a will have proper accumulated value.

Array.prototype.average = function() {
    var sum = 0;
    this.reduce(function(a, b) {
        sum = a + b;
        return sum;
    });
    return sum / this.length;
};

But we are not making use of the power and flexibility of reduce here. Here are two important points to consider when using reduce.

1. reduce accepts a second parameter which says the initial value to be used. Whenever possible, specify that.

2. The first parameter in the function passed to reduce accumulates the result and that will be returned finally, make use of that. No need to use a separate variable to keep track of the results.

So your code would look better like this

Array.prototype.average = function() {

    var sum = this.reduce(function(result, currentValue) {
        return result + currentValue
    }, 0);

    return sum / this.length;

};

console.log([2, 15, 7].average());
# 8

reduce actually works like this. It iterates through the array and passes the current value as the second parameter to the function and the current accumulated result as the first parameter and the value returned from the function will be stored in the accumulated value. So, the sum is actually found like this

result: 0 , currentValue: 2   => 2    (Initializer value `0`)
result: 2 , currentValue: 15  => 17
result: 17, currentValue: 7   => 24

Since it ran out of values from the array, 24 will be returned as the result of the reduce, which will be stored in sum.