Notations:
$A_t$ is a random variable, indicating the action at time step $t$.
$S_t$ is a random variable, indicating the state at time step $t$.
$R_{t}$ is a random variable, indicating the immediate the reward at time step $t$, due to the state $S_t$.
$G_{t}$ is a random variable, indicating the (estimated) future rewards at time step $t$.
$G_t = \sum_{k=t+1}^{\infty} \gamma^{k-t-1}R_k$
where $\gamma$ is a coefficient factor.
$\pi(A_t = a | S_t = s)$ is a probability (policy) indicating the probability of doing action $a$ at state $s$
State Value Function $v_{\pi}(s)$
$v_{\pi}(s) = \mathbb{E_{\pi}}[G_t | S_t = s] = \mathbb{E_{\pi}}\big[\sum_{k=0}^{\infty}\gamma^kR_{t+k-1}|S_t = s\big]$
$= \mathbb{E_{\pi}}[R_{t+1} + \gamma G_{t+1} | S_t = s]$
$= \mathbb{E_{\pi}}[R_{t+1} | S_t = s] + \mathbb{E_{\pi}}[\gamma G_{t+1} | S_t = s]$ (by the linearity of expectation).
$= \sum_{a} \pi(A_t = a | S_t = s ) * \sum_{r, s'} p( R_t = r, S_{t+1} = s' | S_t = s, A_t = t)$
$ + \sum_{s'}\mathbb{E_{\pi}} [\gamma G_{t+1} | S_{t+1} = s'] * \sum_{r}p(S_{t+1} = s', R_{t+1} = r | S_t = s, A_t = a) * \sum_a \pi(A_t = a | S_t = s)$
$ = \sum_{r, s'} p( S_{t+1} = s, R_{t} = r | A_{t} = a, S_{t} = s ) * \sum_{a} \pi(A_t = a | S_t = s ) \big( r + \gamma v_{\pi}(S_t = s') \big)$
Step 3 holds because of the Markov property of the Markov Decision Process, i.e. $P(G_t | S_{t+1}, S_{t} ) = P(G_t | S_{t+1})$.
Action State Value Function $q_{\pi}(s, a)$
$q_{\pi}(s,a) = \mathbb{E}_{\pi}[G_t | S_t = s, A_t = a] = \mathbb{E}[\sum_{k=0}^{\infty}\gamma^{k}R_{t+k+1} | S_t = s, A_t = a]$
$= \mathbb{E}[R_{t+1} | S_t = s, A_t = a ] + \gamma\mathbb{E}[\sum_{k=1}^{\infty}R_{t+k+1} | S_t = s, A_t = a]$
$ = \sum_{s', r} p(S_{t+1} = s', R_t = r | S_t = s, A_t = a) r$ + $\gamma \mathbb{E}[\sum_{k=0}^{\infty}R_{t+k+2} | S_{t+1} = s', A_{t+1} = a']$
Because
$\gamma \mathbb{E}[\sum_{k=0}^{\infty}R_{t+k+2} | S_{t} = s, A_{t} = a]$
= $\gamma \sum_{s'} \mathbb{E}[\sum_{k=0}^{\infty}R_{t+k+2} | S_{t+1} = s', A_{t+1} = a'] \times \sum_{a'} \pi(A_{t+1} = a' | S_{t+1} = s') \times \sum_{r} P(S_{t+1} = s', R_t = r| A_{t} = a, S_t = s) $
= $\gamma \sum_{s'} q_{\pi}(s',a') \times \sum_{a'} \pi(A_{t+1} = a' | S_{t+1} = s') \times \sum_{r} P(S_{t+1} = s', R_t = r| A_{t} = a, S_t = s) $
Therefore
$q_{\pi}(s,a) = \sum_{s',r}P(S_{t+1} = s', R_t = r | S_t = s, A_t = a)[r + \gamma \sum_{a'} \pi(A_{t+1} = a' | S_{t+1} = s') q_{\pi}(a',s')]$
For this part, we use the Markov property that
1. $\pi(A_{t+1} = a' | S_{t+1} = s', S_{t} = s, A_{t} = a) = \pi(A_{t+1} = a' | S_{t+1} = s')$
2. $\gamma \sum_{s'} \mathbb{E}[\sum_{k=0}^{\infty}R_{t+k+2} | S_{t+1} = s', A_{t+1} = a']$
= $\gamma \sum_{s'} \mathbb{E}[\sum_{k=0}^{\infty}R_{t+k+2} | S_{t+1} = s', A_{t+1} = a', S_t = s, A_t = a] $
and the Law of Expectation.
