(Proof) Markov Recurrence by Sum Divergence

let the indicators for (1) state being $i$ at step $n$ and (2) number of times process visits $i$:

$$I_n=\{\begin{array}{ll}1& \text{if }{X}_n=i\\ 0& \text{if }{X}_n\ne i\end{array},N_i=\sum _{\forall n}^{\infty }{I}_n$$

First we have:

$$\begin{array}{rlr}{f}_i& =\mathbb{P}(\bigcup _{n=1}^{\infty }\{I_n=1\}\mid X_0=i)& \\ & =\mathbb{P}(N\ge 2\mid X_0=i)& \text{From }\mathbb{E}[N_i]=\mathbb{E}\left[\sum _{\forall n}^{\infty }{I}_n\right]\\ & =\mathbb{P}(N\ge k\mid N\ge k-1,X_0=i),\forall k\ge 2& \text{Strong Markov}\end{array}$$

The last line holds due to the strong markov property of process $\{X_t\}$. Let stopping time $T$ the time of the $n-1$st visit of $i$. Then:

$$\begin{array}{rlr}& \mathbb{P}(N\ge n|N\ge n-1,X_0=i)& \\ & =\mathbb{P}(\underset{\text{ visits once more... }}{\underbrace{\exists t:X_{T+t}=i}}|\stackrel{\text{ ...after n−1 times }}{\overbrace{{\mathcal{F}}_t}})& \\ & =\mathbb{P}(\exists t:X_{T+t}=i|X_T=X_0)& \text{Str. Mkv.}\\ & =:f_i& \end{array}$$

Thus we showed that the probability of process will be in state $i$ for exactly $m$ times will be:

$$\mathbb{P}(N\ge m-1|X_0=i)\cdot \mathbb{P}(\nexists t:X_{T+t}=i|X_T=i)=f_i^{m-1}(1-f_i)$$

This is remeniscent of the Geometric Distribution, specifically:

$$\mathcal{L}(N|X=0)\sim \text{Geo}(p=1-f_i)$$

A Geometric Distribution with mean $\frac{1}{1-f_i}$ Now looking at the definitions again:

• recurrent if $f_i=1⟺\mathbb{E}[N|X_0=i]=\infty$
• transient if $f_i<1⟺\mathbb{E}[N|X_0=i]<\infty$ Exploring this expectation further:

$$\begin{array}{rlr}\mathbb{E}[N|X_0=i]& :=\mathbb{E}\left[\sum _{\forall n}^{\infty }{I}_n|X_0=i\right]& \\ & =\sum _{n=0}^{\infty }\mathbb{E}[I_n|X_0=i]& \text{linearity}\\ & :=\sum _{n=0}^{\infty }\mathbb{P}[X_n=i|X_0=i]& \\ & =\sum _{n=0}^{\infty }{P}_{ii}^n& \end{array}$$

• $f_i=1⟺\mathbb{E}\left[{\sum }_{n=0}^{\infty }{I}_n\mid X_0=i\right]=\infty$
• $f_i<1⟺\mathbb{E}\left[{\sum }_{n=0}^{\infty }{I}_n\mid X_0=i\right]=$ For an easier to calculate formulation, Then:

$$\begin{array}{rl}\mathbb{E}\left[\sum _{n=0}^n{I}_n\mid X_0=i\right]& =\sum _{n=0}\end{array}$$

<span style="float:right;">■</span>