Theoretical Statistics Questions

Is a pdf part of the exponential family?

Equation 3.4.1

Is a statistic complete?

  • If distribution part of the exponential family:

Theorem 6.2.25

  • If not:

Theorem 6.2.21

Is a statistic sufficient?

  • If distribution part of the exponential family:

Theorem 6.2.10

  • If not:

Factorization theorem (Theorem 6.2.6)

Is a statistic unbiased?

\(E[T] = \int_{x \in R} x g_T(X|\theta) dx\)

If \(E[g(T)] = \theta\), then unbiased.

To find the expected value, sometimes it is helpful to route thru the moment and use \(M_{\sum X_i}(t) = M_{X_i}(t)^n\); for instance, if you know the pdf of X_i but the statistic is \(\sum X_i\).

Does a statistic converge in probability as \(n \xrightarrow{} \infty\)

Convergence in probability (Definition 5.5.1)

But, we are looking at the difference of \(T\) and \(\theta\) instead.

Usually, like this:

\(P(|T - \theta| \geq \epsilon) = P((T - \theta)^2 \geq \epsilon^2) = P(Var[T] \geq \epsilon^2)\)

So, calculate the variance (\(VarT = E[T^2] - E[T]^2\)) and if this (markov’s inequality, lemma 3.8.3, applied in example 5.5.3) holds up:

\(P(|T - \theta| \geq \epsilon) \leq \lim_{n\xrightarrow{}\infty} [Var[T] / \epsilon^2] = 0\)

then it converges in probability, so \(T \xrightarrow[p]{} \theta\).

Find unique best unbiased estimator of \(\theta\).

Theorem 7.3.23

Find suff. statistic \(T\). Get its distribution \(g(t|\theta)\). Find \(E[T(\vec{x})] = \hbox{ (say) } 2n\theta\). With this, we solve for \(\theta\). This becomes \(\phi(T) = \frac{T}{2n}\). Then, verify \(E[\phi(T)] = \theta\).

Find MLE

  • Solution not solvable directly (i.e. pdf has indicator function with support containing parameter)

Look at function and think about maximum (yes.. it’s vague).

  • Normal solutions

Find \(\frac{d}{d\theta} [\ln L(\theta|\vec{x})] = 0\), confirm \(\frac{d}{d\theta^2} [\ln L(\theta=\hat{\theta}|\vec{x})] < 0\), evaluate boundary conditions \(\lim_{\theta \xrightarrow{} \hbox{ upper}} [L(\theta|\vec{x})] = \lim_{\theta \xrightarrow{} \hbox{ lower}} [L(\theta|\vec{x})] = 0\)

Find MOM

Define pdf/pmf-specific expected value:

\(\mu = E(X^1) = \int \hbox{ or } \sum = \hbox{func}(\theta)\)

Set that equal to \(\frac{1}{n} \sum_{i=1}^n x_i^1\)

solve for \(\theta\).

If solving for variance, do the same but use \(X^2\) in both places.

Find CRLB (variance bound)

7.3.10 and, if exponential fam, 7.3.11 helps.

If it’s a series of n RV’s, make sure to multiply by n (e.g. 7.3.12)

Check regularity condition: \(E[\frac{d}{d\theta} \ln p(x;\theta)] = 0\)

Find statistic at CRLB

7.3.15 (but have to prove W(X) is unbiased)

Can calculate Var(W(X)) for variance lower bound also.

Best unbiased of \(\tau(\theta)\)

Rao-Blackwell (7.3.17)

  1. Find complete suff stat \(T\) for \(\theta\)

  2. Compute \(E(T)\)

  3. If \(E(T) = a + b \tau(\theta)\), then the UMVUE of \(T(\theta)\) is \(\phi(\tau) = \frac{\tau-a}{b}\)

  4. If \(E(T) \neq a + b \tau(\theta)\). Find an unbiased estimator of \(\tau(\theta)\). Compute \(\phi(T) = E[W|T]\).

Find an LRT of size 0.05

Find the likelihood function, find MLE \(\hat{\theta}\), build \(\lambda = \frac{L(\theta_0|\vec{x})}{L(\hat{\theta}|\vec{x})}\).

evaluate boundary conditions \(\lim_{\theta \xrightarrow{} \hbox{ upper}} [L(\theta|\vec{x})] = \lim_{\theta \xrightarrow{} \hbox{ lower}} [L(\theta|\vec{x})] = 0\)

Derive level \(\alpha\) UMP test of \(H_0\) and \(H_1\).

  1. Find sufficient statistic, \(T\), (either Factorization theorem or Exponential family proxy)

  2. Find distribution \(g(T|\theta)\)

  1. the suff. stat \(T\) is the transformation from \(Y = g(X) = T(\vec{x})\). Just, remove summations from it during this initial transformation. Once transformed, usually want to find distribution it follows and then apply summations afterwards using properties of that distribution (e.g., the summation of exponentials is a gamma).

  1. Verify MLR with Definition 8.3.16, usually want to take \(\frac{d}{dt}\) and verify if always positive or negative.

  2. Apply Karlin-Rubin with Theorem 8.3.17.

If \(\theta \geq \theta_0\), then the rejection region \(R = \{ \vec{x}: T(\vec{x}) < t_0\}\) and \(\alpha = P_{\theta_0}(T(\vec{x}) < t_0 | \theta \geq \theta_0) = F_T(t|\theta \geq \theta_0) = \int_{-\infty}^{t_0} g(T|\theta) dt\).

Find a pivot quantity and its distribution

  1. Find MLE.

  2. Find pdf of MLE.

  3. Transform this pdf \(X \xrightarrow{} Z\) (where the transformation \(Z = g(X;\theta)\) is the pivot). This pivot \(Z\) should be dependent on both \(X\) and \(\theta\).

  4. Show this pdf \(f_Z(z)\) is not dependent on the parameter \(\theta\)

  5. Use equation 9.2.11.

Find a pivotal interval of \(\theta\) w/ confidence coeff \((1-\alpha)\)

\(P_\theta(a \leq Q(\vec{X},\theta) \leq b) \geq 1-\alpha\). Then, replace \(Q(\vec{X},\theta)\) with the actual pivot (e.g. \(Q(\vec{X},\theta) = \frac{X}{\theta}\) and then solve for \(\theta\). This is the confidence interval. If the interval is symmetric, you can make this problem simpler by \(P_\theta(Q(\vec{X},\theta) \leq a) = P_\theta(Q(\vec{X},\theta) \geq b) = \alpha/2\)

Find smallest pivotal interval with CI \((1-\alpha)\)

See Theorem 9.2.12 (pivoting the CDF)

Evaluate the CDF of the \(Q(\vec{X},\theta)\) (take the integral from lower to x)… let’s call this \(F_{Q(\vec{X},\theta)}(x) = \int_a^x f(t)dt\)

We have \(P(a \leq Q(\vec{X},\theta) \leq b) = 1 - \alpha\)

CI region: \(\theta\) is \(\{\theta: Q^{-1}(\vec{x},a) \leq \theta \leq Q^{-1}(\vec{x},b) \}\)

CI size: \(P(a \leq Q(\vec{x},\theta) \leq b) = F_{Q(\vec{X},\theta)}(b) - F_{Q(\vec{X},\theta)}(a) = 1 - \alpha\)

Lastly, plug in the end points.

So, if (say) \(x \in (0,1)\)

  1. Use upper bound: then plug in \(b=1\) and solve for \(a\) using the CI size expression. Plug the a and b values into the CI region expression

  2. Use lower bound: then plug in \(a=0\) and solve for \(b\) using the CI size expression. Plug the a and b values into the CI region expression

Look at the interval which is smaller. And conclude that that’s the interval to choose.