What is Stationarity? Definition, Examples and Worked Solutions

Stationarity

Stationarity means the probability distribution of a stochastic process does not change over time. 

Examples:

• Probability on the dice is the same every day on every roll
• Probability on the roulette wheel does not change over time
• Probability on a deck of cards is the same on every draw

In mathematics and statistics, a stationary process (or a strict/strictly stationary process or strong/strongly stationary process) is a stochastic process whose unconditional joint probability distribution does not change when shifted in time. Consequently, parameters such as mean and variance also do not change over time.

https://en.wikipedia.org/wiki/Stationary_process

To solve linear models, we use weak stationary since we are only going to examine the mean and variance of the distribution. Weak stationary occurs if just the first and second moments are stationary.

In other words, the mean and variance stay the same and 2-point functions (like the covariance function) depend only on the difference in time indices, not on the exact location of a particular point in time (refer to Example 1 below).

Here’s a video that shows how to prove MA(1) is weakly stationary.

Example 1

Given that w is a constant and z and z’ are two independent, normalized random variables, prove that 

X_t = z cos(wt) + z’ sin(wt) 

is weakly stationary.

Solution

Note that cos(wt) and sin(wt) are constants for any given t. This is in contrast to z_t for instance, which is random and changes even for the same t.

Mean
= E[X_t] 
= E[z cos(wt)] + E[z’ sin(wt)] 
= cos(wt) E[z] + sin(wt) E[z’]
= cos(wt)*0 + sin(wt)*0
= 0

Variance
= E[X_t²] – E[X_t]²
= E[X_t²] – 0
= E[(z cos(wt) + z’ sin(wt))²]
= E[z²cos²(wt) + 2zz’cos(wt)sin(wt) + z’²sin²(wt)]
= cos²(wt)*E[z²] + 2cos(wt)sin(wt)E[zz’] + sin²(wt)E[z’²]
= cos²(wt)*1 + 0 + sin²(wt)*1
= cos²(wt) + sin²(wt)
= 1

Covariance
= E[X_tX_s]
= E[(z cos(wt) + z’ sin(wt))(z cos(ws) + z’ sin(ws))]
= E[z²cos(wt)cos(ws) + zz’cos(wt)sin(ws) + z’zsin(wt)cos(ws) + z’²sin(wt)sin(ws)]
= cos(wt)cos(ws)*E[z²] + cos(wt)sin(ws)*E[zz’] + sin(wt)cos(ws)*E[z’z] + sin(wt)sin(ws)*E[z’²]
= cos(wt)cos(ws)*1 + 0 + 0 + sin(wt)sin(ws)*1
= cos [w(t-s)]

Since mean and variance are constant, and covariance only depends on the difference between two points in time (i.e. t-s), X_t is weakly stationary.

Example 2

Let X_t = z_t + θ*z_t-2, where

E[z_t] = 0
E[z_t²] = σ²
E[z_tz_s] = σ² if t = s, 0 otherwise

Is X_t stationary? Find the mean, variance and covariance of X_t.

If A = 0.25(X₁ + X₂ + X₃ + X₄), find the mean and variance of A.

Solution

Note: 

For this question, because the value of E[z_tz_s] depends on whether t equals s, we should consider s as t-k to make it easier to simplify our solution later.

Is X_t stationary?

Yes, because we have expressed things recursively. If we shift things in time, we still have the same relative relationship.

Find the mean, variance and covariance of X_t.

Mean
= E[X_t]
= E[z_t + θ*z_t-2]
= E[z_t] + θE[z_t-2]
= 0

Variance
= E[X_t²] – E[X_t]²
= E[(z_t + θ*z_t-2)²] – 0
= E[z_t²] + 2θE[z_tz_t-2] + θ²E[z_t-2]²
= 1 + θ²

Covariance
= E[X_tX_t-k], where k not equals 0
= E[(z_t + θ*z_t-2)(z_t-k + θ*z_t-k-2)]
= E[z_tz_t-k] + θ*E[z_tz_t-k-2] + θ*E[z_t-2z_t-k] + θ²E[z_t-2z_t-k-2]
= 0 + θE[z_tz_t-k-2] + θE[z_t-2z_t-k] + 0 (refer to Note 1) 
= (θ*0 + θ*1) if k = 2, 
   (θ*1 + θ*0) if k = -2
   0 otherwise

If A = 0.25(X₁ + X₂ + X₃ + X₄), find the mean and variance of A.

Mean 
= E[A]
= 0.25*(E[X₁] + E[X₂] + E[X₃] + E[X₄])
= 0

Variance
= E[A²] – E[A]²
= 0.0625*E[(X₁ + X₂ + X₃ + X₄)²]
= 0.0625*(E[X₁²] + E[X₂²] + E[X₃²] + E[X₄²] + E[X₁X₂] + E[X₁X₃] + .. + E[X₃X₄])
= 0.0625*(E[X₁²] + E[X₂²] + E[X₃²] + E[X₄²] + 2E[X₁X₃] + 2E[X₂X₄]) (refer to Note 2)
= 0.0625*[4*(1 + θ²) + 2θ + 2θ]
= 0.25(1 + θ + θ²)

Note 1

Since k not equals 0, t not equal t – k.

Therefore,
E[z_tz_t-k]
= E[z_t-2z_t-k-2]
= 0

Note 2

E[X_tX_t-k] = θ, where k = +/- 2

Therefore, only E[X₁X₃], E[X₂X₄], E[X₃X₁] and E[X₁X₃] matter, the rest are zeros.