Mathematical Statistics

The difference between statistics majors and non-majors truly lies in Mathematical Statistics

It’s undeniable that statistics play a crucial role in modern human society, with applications in every scientific field. Whether it’s hypothesis testing or analytical techniques, a thorough understanding of statistics is essential for mastery in any domain. While the door to statistics is wide open even for non-majors, and many experts use statistical methods fluently within their specific domains—sometimes even more adeptly than majors—the fundamental difference lies in the level of mathematical understanding.

Majoring in statistics doesn’t just involve skimming through a few proofs and understanding the mathematical background of certain techniques. It’s about empathizing with the overarching concepts that run through statistics, having a clear understanding of the relationships between well-known distributions, and quickly grasping the principles behind new methods. Mathematical Statistics serves as both the study and training for this, encompassing the mathematical theories that underpin the entire field of statistics.

Probability Theory

For discussions involving measure theory at the level of Probability Theory, refer to the Measure Theory category.

• Probability and the Addition Rule $P$

Univariate Random Variables

• Random Variables and Probability Distributions $X$
• Linear Combinations of Random Variables
• Probability Convergence $\overset{P}{\to}$
  • Convergence in Probability implies Convergence in Distribution
• Distribution Convergence $\overset{D}{\to}$
  • Convergence in Distribution implies Bounded in Probability
• Probability Bounded
• Proof of the Weak Law of Large Numbers
• Proof of the Central Limit Theorem

Multivariate Random Vectors

• Multivariate Probability Distributions $\mathbf{X}$
• Transformation of Multivariate Random Variables
  • Convolution Formula for Probability Density Functions
• Conditional Probability Distributions
  • Conditional Expectation Minimizes the Sum of Squares Deviation
• Independence of Random Variables $X_{1} \perp X_{2}$
  • Mutual Independence and iid Random Variables
  • Bernstein’s Distribution: Being Independent in Pairs Doesn’t Mean Mutually Independent
• Convergence of Multivariate Random Variables
• Convergence in Distribution for Multivariate Random Variables
• Principal Component Analysis (PCA)

Moments

• Expectation, Mean, Variance, Moments Definition $E X^{t}$
  • Three Representative Values in Statistics: Mode, Median, Mean
  • Mathematical Properties of Representative Values
  • Properties of Mean and Variance
  • Expectation of a Sum of Random Variables in a Function Form
• Covariance
  • Covariance Matrix
  • Pearson Correlation Coefficient $\rho$
  • No Correlation Doesn’t Imply Independence
• Skewness
• Kurtosis
• Moment Generating Function $E e^{tX}$
  • If an $n$-th Order Moment Exists, Moments of Lower Order Also Exist
• Weighted Average
• Pooled Variance $s_{p}^{2}$
• Expectation of a Random Vector

Probability Distribution Theory

The study of Probability Distribution Theory in mathematical statistics is crucial but expansive. In this blog, due to its vast scope and inclusion of topics beyond mathematical statistics, it has been made a separate category.

• Comprehensive Summary of the Addition of Random Variables with Specific Distributions
• Student’s Theorem
• Two Normally Distributed Random Variables being Independent is Equivalent to having a Covariance of $0$
• Proof of Stirling’s Approximation Formula in Mathematical Statistics
• Delta Method
• Exponential Family of Probability Distributions
• Location Family $f (x ; \theta)$
  • Sufficient Statistic and MLE for Location Family
• Scale Family $f (x ; \sigma)$

Statistical Inference

• Random Sampling
  • Sampling Univariate Random Variables
• Mean and Variance of Sample Mean
  • Distinguishing Sample Standard Deviation and Standard Error

Statistics

• Statistic and Estimator $T$
  • Difference between Monte Carlo Method and Bootstrapping
  • Confidence Interval
  • General Definition of Standard Error
• Pivot $Q(\mathbf{X}, \theta)$
• Order Statistic $X_{(k)}$
• Consistent Estimator $T_{n} \overset{P}{\to} \theta$
• Method of Moments
  • Satterthwaite’s Approximation

Unbiased Estimation

• Unbiased Estimator $ET = \theta$
  • Bias
    • Bias-Variance Trade-off
  • Why Divide by $n-1$ for Sample Variance
• Efficient Estimator
  • Cramér-Rao Lower Bound
    • Cramér-Rao Lower Bound for Unbiased Estimators
• Best Unbiased Estimator (UMVUE)
  • Uniqueness of Minimum Variance Unbiased Estimator
• Rao-Blackwell Theorem
• Lehmann-Scheffé Theorem

Sufficient Statistics

• Sufficient Statistic
• Neyman Factorization Theorem
• Minimal Sufficient Statistic
  • Variance of an Unbiased Estimator Dependent on a Minimal Sufficient Statistic is Minimized
• Ancillary Statistic
  • Ancillary Statistic for Location-Scale Family
• Complete Statistic
  • Complete Statistic for Exponential Family of Distributions
• Basu’s Theorem Proof

Likelihood Estimation

• Likelihood Function $L(\theta | \mathbf{x})$
• Maximum Likelihood Estimator (MLE)
  • Regularity Conditions
  • Fisher Information $I(\theta)$
  • Bartlett’s Identity
• Invariance Property of MLE
• Unique MLE is Dependent on a Sufficient Statistic

Hypothesis Testing

• Definition of Hypothesis Testing $H_{0} \text{ vs } H_{1}$
• Power Function $\beta (\theta)$
• Likelihood Ratio Test (LRT)
  • LRT Involving a Sufficient Statistic
• Unbiased Test Power Function and Uniformly Most Powerful Test (UMP)
  • Neyman-Pearson Lemma Proof
  • UMP Involving a Sufficient Statistic
• Monotone Likelihood Ratio (MLR)
  • Karlin-Rubin Theorem Proof
• Significance Level $p(\mathbf{X})$

Interval Estimation

• Interval Estimator $\left[ L, U \right]$
  • Confidence Set
• One-to-one Correspondence between Hypothesis Testing and Confidence Intervals
• Uniformly Most Accurate Confidence Set (UMA)
• Stochastic Increasing and Decreasing Functions and Confidence Intervals
• Shortest Confidence Interval for Unimodal Distributions

Bayesian

• Proof of Bayes’ Theorem and Prior, Posterior Distributions
  • Monty Hall Dilemma through Bayes’ Theorem
• Bayesian Paradigm
• Laplace’s Succession Rule
• Conjugate Prior
• Laplace Prior
• Jeffreys Prior
• Credible Interval
  • Difference between Credible and Confidence Intervals
  • Highest Posterior Density Credible Interval
• Hypothesis Testing using Bayes Factor

References

• Casella. (2001). Statistical Inference(2nd Edition)
• Hogg et al. (2013). Introduction to Mathematical Statistics(7th Edition)
• 김달호. (2013). R과 WinBUGS를 이용한 베이지안 통계학