<Home>Teaching><MATH 170S (Winter 2024)>Homework 3

Homework 3

Table of Contents

Problem 1

Let X1,…,XnX_1, \ldots, X_n be a random sample from a uniform distribution on the interval (θ−1,θ+1)\p{\theta-1, \theta+1}, where −∞<θ<∞-\infty < \theta < \infty.

  1. Find the method of moment estimator for θ\theta.
  2. Is your estimator in part 1 an unbiased estimator of θ\theta? Justify your answer.
  3. Given the following n=5n = 5 observations of XX, give a point estimate of θ\theta.
6.617.706.988.367.26\begin{array}{rrrrr} 6.61 & 7.70 & 6.98 & 8.36 & 7.26 \end{array}
Solution.

  1. The mean of the uniform distribution is

    E[Xi]=θ,\E\br{X_i} = \theta,

    so the method of moment estimate for θ\theta is given by x‾=θ^\mean{x} = \widehat{\theta}. Thus, the estimator is

    θ^=X‾.\boxed{\widehat{\theta} = \mean{X}}.

  2. Yes. By linearity of expectation,

    E[θ^]=E[1n∑i=1nXi]=1n∑i=1nE[Xi]=θ.\E\br{\widehat{\theta}} = \E\br{\frac{1}{n} \sum_{i=1}^n X_i} = \frac{1}{n} \sum_{i=1}^n \E\br{X_i} = \theta.

  3. We have

    θ^=x‾=7.382.\boxed{ \widehat{\theta} = \mean{x} = 7.382 }.

Problem 2

The heart rate (bpm\mathrm{bpm}) of 1010 athletes is measured. The data is given below. Assuming that the distribution from which the data was drawn is normally distributed, find a 95%95\% confidence interval for the mean heart rate.

38544236524449506250\begin{array}{rrrrrrrrrr} 38 & 54 & 42 & 36 & 52 & 44 & 49 & 50 & 62 & 50 \end{array}
Solution.

Assume that the data was pulled from N(μ,σ2)\mathcal{N}\p{\mu, \sigma^2}. Since the population standard deviation is unknown, we need to use a tt-distribution. Our test statistic is

T=X‾−μS/n∼t(n−1).T = \frac{\mean{X} - \mu}{S/\sqrt{n}} \sim t\p{n-1}.

Here, SS is the sample standard deviation and t(n−1)t\p{n-1} is the tt-distribution with n−1n-1 degrees of freedom. The endpoints for a 100(1−α)%100\p{1 - \alpha}\% confidence interval are given by

x‾±tα2(n−1)sn,\mean{x} \pm t_{\frac{\alpha}{2}}\p{n-1} \frac{s}{\sqrt{n}},

where tα2(n−1)t_{\frac{\alpha}{2}}\p{n-1} is the 100(1−α2)th100\p{1 - \frac{\alpha}{2}}^{\mathrm{th}} percentile of a t(n−1)t\p{n-1} distribution. For our problem, the parameters are

x‾=47.7,s≈7.8323,n=10,α=0.05.\begin{aligned} \mean{x} &= 47.7, \\ s &\approx 7.8323, n &= 10, \\ \alpha &= 0.05. \end{aligned}

From the textbook's tt-table,

t0.025(6)=2.262,t_{0.025}\p{6} = 2.262,

so the endpoints are

47.7±2.262⋅7.832310≈47.7±5.6.47.7 \pm 2.262 \cdot \frac{7.8323}{\sqrt{10}} \approx \boxed{47.7 \pm 5.6}.

Problem 3

Let x1,…,xnx_1, \ldots, x_n be an observed sample drawn independently from an exponential distribution, Exp⁡(λ)\operatorname{Exp}\p{\lambda}. Assume that the prior on λ\lambda is characterized by a Gamma distribution, Γ(α,β)\Gamma\p{\alpha, \beta}, with pdf

π(λ)=βαΓ(α)λα−1e−βλ,0<x<∞.\pi\p{\lambda} = \frac{\beta^\alpha}{\Gamma\p{\alpha}} \lambda^{\alpha-1} e^{-\beta\lambda}, \quad 0 < x < \infty.

Note that the pdf of an exponential distribution is given by

f(x)=λe−λx,0<x<∞.f\p{x} = \lambda e^{-\lambda x}, \quad 0 < x < \infty.
  1. Find the posterior distribution for λ\lambda, π(λ | x)\pi\p{\lambda \given x}.
  2. Calculate the mean of the posterior distribution.
Solution.

  1. The starting point for these types of problems is

    posterior∝likelihood×prior.\mathrm{posterior} \propto \mathrm{likelihood} \times \mathrm{prior}.

    Here, ∝\propto means that the two things only differ by a constant factor. Here, constant means anything that doesn't depend on λ\lambda. We don't lose any information about the distribution when dropping factors without λ\lambda in them because we can always figure out the normalization constant by integrating with respect to λ\lambda. Actually, we don't even need the normalization constant to figure out the distribution.

    The likelihood is given by

    f(x1,…,xn∣λ)=∏i=1nλe−λxi=λne−λnx‾. f\p{x_1, \ldots, x_n \mid \lambda} = \prod_{i=1}^n \lambda e^{-\lambda x_i} = \lambda^n e^{-\lambda n\mean{x}}.

    The prior is

    π(λ)=βαΓ(α)λα−1e−βλ∝λα−1e−βλ. \pi\p{\lambda} = \frac{\beta^\alpha}{\Gamma\p{\alpha}} \lambda^{\alpha-1} e^{-\beta\lambda} \propto \lambda^{\alpha-1} e^{-\beta\lambda}.

    Putting everything together,

    π(λ | x)∝λne−λnx‾λα−1e−βλ∝λα+n−1e−λ(β+nx‾).\begin{aligned} \pi\p{\lambda \given x} &\propto \lambda^n e^{-\lambda n\mean{x}}\lambda^{\alpha-1} e^{-\beta\lambda} \\ &\propto \lambda^{\alpha+n-1} e^{-\lambda\p{\beta+n\mean{x}}}. \end{aligned}

    By comparing to our known distributions, we see that the posterior distribution is

    Γ(α+n,β+nx‾).\boxed{\Gamma\p{\alpha + n, \beta + n\mean{x}}}.

  2. The mean of a Γ(α,β)\Gamma\p{\alpha, \beta} distribution (with the notation in the problem) is αβ\frac{\alpha}{\beta}, so the mean of the posterior distribution is

    α+nβ+nx‾.\boxed{\frac{\alpha + n}{\beta + n\mean{x}}}.

Problem 4

Assume y1,…,yny_1, \ldots, y_n are independent observations drawn from N(βxi,σ2)\mathcal{N}\p{\beta x_i, \sigma^2}. The xix_i's and σ2\sigma^2 are known constants, and β\beta is an unknown parameter which has prior distribution N(β0,τ2)\mathcal{N}\p{\beta_0, \tau^2}, where β0\beta_0 and τ2\tau^2 are known constants. Derive the posterior distribution of β\beta.

Solution.

We can use the same approach as in Problem 3. Here, constants are any factors that don't have a β\beta in them. The likelihood is

f(y1,…,yn | β)=∏i=1n12πσ2exp⁡(−12σ2(yi−βxi)2)∝∏i=1nexp⁡(−yi22σ2)exp⁡(βxiyiσ2−β2xi22σ2)∝exp⁡(∑i=1n(βxiyiσ2−β2xi22σ2))∝exp⁡(βnxy‾σ2−β2nx2‾2σ2),\begin{aligned} f\p{y_1, \ldots, y_n \given \beta} &= \prod_{i=1}^n \frac{1}{\sqrt{2\pi\sigma^2}} \exp\p{-\frac{1}{2\sigma^2}\p{y_i - \beta x_i}^2} \\ &\propto \prod_{i=1}^n \exp\p{-\frac{y_i^2}{2\sigma^2}} \exp\p{\frac{\beta x_iy_i}{\sigma^2} - \frac{\beta^2x_i^2}{2\sigma^2}} \\ &\propto \exp\p{\sum_{i=1}^n \p{\frac{\beta x_iy_i}{\sigma^2} - \frac{\beta^2x_i^2}{2\sigma^2}}} \\ &\propto \exp\p{\beta \frac{n\mean{xy}}{\sigma^2} - \beta^2 \frac{n\mean{x^2}}{2\sigma^2}}, \end{aligned}

where I used the shorthands

xy‾=1n∑i=1nxiyi,x2‾=1n∑i=1nxi2.\mean{xy} = \frac{1}{n} \sum_{i=1}^n x_iy_i, \quad \mean{x^2} = \frac{1}{n} \sum_{i=1}^n x_i^2.

Similarly, the prior is

π(β)=12πτ2exp⁡(−12τ2(β−β0)2)∝exp⁡(−β22τ2+2ββ02τ2)exp⁡(−β022τ2)∝exp⁡(−β22τ2+ββ0τ2).\begin{aligned} \pi\p{\beta} &= \frac{1}{\sqrt{2\pi\tau^2}} \exp\p{-\frac{1}{2\tau^2}\p{\beta - \beta_0}^2} \\ &\propto \exp\p{-\frac{\beta^2}{2\tau^2} + \frac{2\beta\beta_0}{2\tau^2}} \exp\p{-\frac{\beta_0^2}{2\tau^2}} \\ &\propto \exp\p{-\frac{\beta^2}{2\tau^2} + \frac{\beta\beta_0}{\tau^2}}. \end{aligned}

Putting these together, the posterior pdf is

π(β | y1,…,yn)∝exp⁡(βnxy‾σ2−β2nx2‾2σ2)exp⁡(−β22τ2+ββ0τ2)∝exp⁡(−12(nx2‾σ2+1τ2)β2+(nxy‾σ2+β0τ2)β)∝exp⁡(−12σ2τ2(τ2nx2‾+σ2)β2+1σ2τ2(τ2nxy‾+σ2β0)β).\begin{aligned} \pi\p{\beta \given y_1, \ldots, y_n} &\propto \exp\p{\beta \frac{n\mean{xy}}{\sigma^2} - \beta^2 \frac{n\mean{x^2}}{2\sigma^2}} \exp\p{-\frac{\beta^2}{2\tau^2} + \frac{\beta\beta_0}{\tau^2}} \\ &\propto \exp\p{-\frac{1}{2}\p{\frac{n\mean{x^2}}{\sigma^2} + \frac{1}{\tau^2}}\beta^2 + \p{\frac{n\mean{xy}}{\sigma^2} + \frac{\beta_0}{\tau^2}}\beta} \\ &\propto \exp\p{ - \frac{1}{2\sigma^2\tau^2}\p{\tau^2n\mean{x^2} + \sigma^2}\beta^2 + \frac{1}{\sigma^2\tau^2} \p{\tau^2n\mean{xy} + \sigma^2\beta_0}\beta}. \end{aligned}

Just from looking at this, we know that the posterior distribution is a normal distribution, so we only need to figure out the mean and variance. Let

a=12σ2τ2(τ2nx2‾+σ2),b=1σ2τ2(τ2nxy‾+σ2β0)\begin{aligned} a &= \frac{1}{2\sigma^2\tau^2}\p{\tau^2n\mean{x^2} + \sigma^2}, \\ b &= \frac{1}{\sigma^2\tau^2} \p{\tau^2n\mean{xy} + \sigma^2\beta_0} \end{aligned}

and recall that we can complete the square:

exp⁡(−aβ2+bβ)=exp⁡(−a(β−b2a)2+b24a)=exp⁡(−a(β−b2a)2)exp⁡(b24a)∝exp⁡(−a(β−b2a)2),,\begin{aligned} \exp\p{-a\beta^2 + b\beta} &= \exp\p{-a\p{\beta - \frac{b}{2a}}^2 + \frac{b^2}{4a}} \\ &= \exp\p{-a\p{\beta - \frac{b}{2a}}^2} \exp\p{\frac{b^2}{4a}} \\ &\propto \exp\p{-a\p{\beta - \frac{b}{2a}}^2}, \end{aligned},

so the mean will be

b2a=1σ2τ2(τ2nxy‾+σ2β0)2⋅12σ2τ2(τ2nx2‾+σ2)=τ2nxy‾+σ2β0τ2nx2‾+σ2.\begin{aligned} \frac{b}{2a} &= \frac{\frac{1}{\sigma^2\tau^2} \p{\tau^2n\mean{xy} + \sigma^2\beta_0}}{2 \cdot \frac{1}{2\sigma^2\tau^2}\p{\tau^2n\mean{x^2} + \sigma^2}} \\ &= \frac{\tau^2n\mean{xy} + \sigma^2\beta_0}{\tau^2n\mean{x^2} + \sigma^2}. \end{aligned}

For the variance σ12\sigma_1^2, notice that by looking at the pdf for a normal distribution, it must satisfy 12σ12=a\frac{1}{2\sigma_1^2} = a, so the new variance is

12a=12⋅12σ2τ2(τ2nx2‾+σ2)=σ2τ2τ2nx2‾+σ2.\begin{aligned} \frac{1}{2a} &= \frac{1}{2 \cdot \frac{1}{2\sigma^2\tau^2}\p{\tau^2n\mean{x^2} + \sigma^2}} \\ &= \frac{\sigma^2\tau^2}{\tau^2n\mean{x^2} + \sigma^2}. \end{aligned}

Thus, the posterior distribution is

N(τ2∑i=1nxiyi+σ2β0τ2∑i=1nxi2+σ2,σ2τ2τ2∑i=1nxi2+σ2).\boxed{\mathcal{N}\p{\frac{\tau^2\sum_{i=1}^n x_iy_i + \sigma^2\beta_0}{\tau^2\sum_{i=1}^n x_i^2 + \sigma^2}, \frac{\sigma^2\tau^2}{\tau^2\sum_{i=1}^n x_i^2 + \sigma^2}}}.

Problem 5

  1. Let X1,…,XnX_1, \ldots, X_n be a random sample from a continuous random variable with pdf
f(x,θ)=θxθ−1,0<x<1,0<θ.f\p{x, \theta} = \theta x^{\theta-1}, \quad 0 < x < 1, \quad 0 < \theta.
  1. Find a sufficient statistic YY for θ\theta.
  2. Show that the maximum likelihood estimator θ^\widehat{\theta} is a function of YY.
  3. Argue that θ^\widehat{\theta} is also a sufficient statistic for θ\theta.
Solution.

  1. To find a sufficient statistic, we can factor the likelihood function and eyeball it.

    f(x1,…,xn,θ)=∏i=1nθxiθ−1=θn(∏i=1nxi)θ−1.f\p{x_1, \ldots, x_n, \theta} = \prod_{i=1}^n \theta x_i^{\theta-1} = \theta^n \p{\prod_{i=1}^n x_i}^{\theta-1}.

    So a sufficient statistic is Y=∏i=1nXiY = \prod_{i=1}^n X_i. We can write down the factorization explicitly:

    φ(y,θ)=θnyθ−1,h(x1,…,xn)=1.\varphi\p{y, \theta} = \theta^n y^{\theta-1}, \quad h\p{x_1, \ldots, x_n} = 1.

  2. From our computation above, the log-likelihood function is

    ln⁡L(θ)=nln⁡θ+(θ−1)ln⁡∏i=1nxi.\ln L\p{\theta} = n\ln \theta + \p{\theta-1} \ln \prod_{i=1}^n x_i.

    Taking derivatives and solving for θ\theta gives the MLE

    θ^=−nln⁡∏i=1nXi=−nln⁡Y.\widehat{\theta} = -\frac{n}{\ln \prod_{i=1}^n X_i} = -\frac{n}{\ln Y}.

  3. We can solve for YY in terms of θ^\widehat{\theta}:

    Y=exp⁡(−nθ^),Y = \exp\p{-\frac{n}{\widehat{\theta}}},

    so plugging this into our factorization,

    f(x1,…,xn,θ)=φ(exp⁡(−nθ^),θ)h(x1,…,xn).f\p{x_1, \ldots, x_n, \theta} = \varphi\p{\exp\p{-\frac{n}{\widehat{\theta}}}, \theta} h\p{x_1, \ldots, x_n}.

    This shows that θ^\widehat{\theta} is a sufficient statistic for θ\theta.