Mathematical Statistics

The only difference between statistics majors and non-majors is, in fact, mathematical statistics.

It can be stated unequivocally that there is no corner of modern society where statistics is not used. No matter which field of science one pursues, to reach any significant level—whether through hypothesis testing or analytical techniques—one must learn statistics. The door to statistics is wide open even for countless non-majors, and in many cases experts use it more fluently in their own domains than statistics majors do.

How, then, are these experts distinguished from those who have formally majored in statistics? Certainly, one might point to the fact that majors learn numerous techniques in depth without being limited by the domain of the data, and that they develop a strong intuition for various types of data through experience. However, the most fundamental difference lies in the level of mathematical understanding.

Majoring in statistics is not merely about reviewing a few proofs of certain techniques and understanding the underlying mathematics. It requires an appreciation of the overarching concepts that permeate all of statistics, a clear grasp of the relationships among widely known distributions, and the ability to quickly understand the principles behind any new method encountered. Mathematical statistics is precisely the study and training geared toward developing that level of understanding, dealing with the mathematical theories that support the entirety of statistics.

Probability Theory

The level that employs measure theory🔥 has been set aside in the Probability Theory category.

• Probability and the Rule of Addition of Probabilities $P$

Univariate Random Variables

• Random Variables and Probability Distributions $X$
• Linear Combinations of Random Variables
• Convergence in Probability $\overset{P}{\to}$
  • Convergence in Probability Implies Convergence in Distribution
• Convergence in Distribution $\overset{D}{\to}$
  • Convergence in Distribution Implies Tightness
• Tightness
• Proof of the Weak Law of Large Numbers
• Proof of the Central Limit Theorem

Multivariate Random Vectors

• Multivariate Probability Distribution $\mathbf{X}$
• Transformation of Multivariate Random Variables
  • Convolution Formula for Probability Density Functions
• Conditional Probability Distribution
  • The Conditional Expectation Minimizes the Sum of Squared Deviations
• Independence of Random Variables $X_{1} \perp X_{2}$
  • Mutual Independence of Random Variables and i.i.d.
  • Bernstein’s Distribution: Pairwise Independence Does Not Imply Mutual Independence
• Convergence in Probability for Multivariate Random Variables
• Convergence in Distribution for Multivariate Random Variables
• Principal Component Analysis (PCA)
• 🔒(25/05/06)mixture distributions

Moments

• Definitions of Expected Value, Mean, Variance, and Moments $E X^{t}$
  • The Three Representative Values in Statistics: Mode, Median, Mean
  • Proof of the Mathematical Properties of Representative Values
  • Properties of the Mean and Variance
  • Expected Value of a Sum in the Form of a Function of Random Variables
• Covariance
  • Covariance Matrix
  • Pearson Correlation Coefficient $\rho$
  • Lack of Correlation Does Not Imply Independence
• Skewness
• Kurtosis
• What is a Moment Generating Function? $E e^{tX}$
  • If the n-th Moment Exists, Then All Moments of Lower Order Exist
• Weighted Mean
• Pooled Variance $s_{p}^{2}$
• Expected Value of a Random Vector

Theory of Probability Distributions

The theory of probability distributions, as taught in mathematical statistics, is extremely important; however, at Raw Shrimp Sushi House its scope has grown so vast—and now covers topics beyond mathematical statistics—that it has been given its own category.

• Summation Theorem for Random Variables with a Specific Distribution
• Student’s Theorem
• For Two Normally Distributed Random Variables, Independence is Equivalent to Zero Covariance
• A Mathematical Statistics Proof of Stirling’s Approximation
• Delta Method
• Exponential Family of Distributions
• Location Family $f(x; \theta)$
  • Sufficient Statistics and Maximum Likelihood Estimators for the Location Family
• Scale Family $f(x; \sigma)$

Quadratic Forms

• 🔒(25/04/09) Quadratic Form of a Random Vector $\mathbf{X}^{T} A \mathbf{X}$
• 🔒(25/04/13) Expected Value of a Quadratic Form of a Random Vector
• 🔒(25/04/17) Sum of Squared Deviations Represented as a Quadratic Form of a Random Vector $(n-1) S^{2} = \mathbf{X}^{T} \left( I_{n} - \frac{1}{n} J_{n} \right) \mathbf{X}$
• 🔒(25/04/21) Moment Generating Function of a Quadratic Form of a Normally Distributed Random Vector
• 🔒(25/04/25) Equivalence Conditions for the Chi-Square Property of a Quadratic Form of a Normally Distributed Random Vector
• 🔒(25/05/11) Proof of Craig’s Theorem
• 🔒(25/05/19) Proof of the Hog–Craig Theorem
• 🔒(25/05/27) Proof of Cochran’s Theorem

Statistical Inference

• Random Samples
  • How to Sample a Univariate Random Variable
• Mean and Variance of the Sample Mean from a Random Sample
  • Distinction Between Sample Standard Deviation and Standard Error

Test Statistics

• Test Statistics and Estimators $T$
  • Differences Between the Monte Carlo Method and the Bootstrap
  • Confidence Interval
  • General Definition of Standard Error
• Pivot $Q(\mathbf{X}, \theta)$
• Order Statistics $X_{(k)}$
• Consistent Estimator $T_{n} \overset{P}{\to} \theta$
• Method of Moments
  • Satterthwaite Approximation

Unbiased Estimation

• Unbiased Estimator $E T = \theta$
  • Bias
    • Bias–Variance Trade-Off
  • The Reason for Dividing the Sample Variance by $n-1$
• Efficient Estimator
  • Rao–Cramer Lower Bound
    • Rao–Cramer Lower Bound for Unbiased Estimators
• Uniformly Minimum Variance Unbiased Estimator (UMVUE)
  • Uniqueness of the Minimum Variance Unbiased Estimator
• Rao–Blackwell Theorem
• Lehmann–Scheffé Theorem

Sufficient Statistics

• Sufficient Statistics
• Neyman Factorization Theorem
• Minimal Sufficient Statistics
  • Among Unbiased Estimators Given the Minimal Sufficient Statistic, the Variance is Minimized
• Ancillary Statistics
  • Ancillary Statistics for the Location–Scale Family
• Complete Statistics
  • Complete Statistics for Exponential Family Distributions
• Proof of Basu’s Theorem

Likelihood Estimation

• Likelihood Function $L(\theta | \mathbf{x})$
• Maximum Likelihood Estimator (MLE)
  • Regularity Conditions
  • Fisher Information $I(\theta)$
  • Bartlett’s Identity
• Invariance Property of the Maximum Likelihood Estimator
• The Unique Maximum Likelihood Estimator is a Function of the Sufficient Statistic

Hypothesis Testing

• Definition of Hypothesis Testing $H_{0} \text{ vs } H_{1}$
• Power Function $\beta(\theta)$
• Likelihood Ratio Test (LRT)
  • Likelihood Ratio Test Involving Sufficient Statistics
• Unbiased Test Power Function and Uniformly Most Powerful Test (UMP)
  • Proof of the Neyman–Pearson Lemma
  • Uniformly Most Powerful Test Involving Sufficient Statistics
• Monotone Likelihood Ratio (MLR)
  • Proof of the Karlin–Rubin Theorem
• p-value $p(\mathbf{X})$

Interval Estimation

• Interval Estimator $\left[ L, U \right]$
  • Confidence Set
• One-to-One Correspondence Between Hypothesis Tests and Confidence Intervals
• Uniformly Most Accurate Confidence Set (UMA)
• Probabilistic Inversion and Confidence Intervals
• Shortest Confidence Interval for Unimodal Distributions

Bayesian

• Proof of Bayes’ Theorem and Prior and Posterior Distributions
  • The Monty Hall Dilemma Viewed Through Bayes’ Theorem
• Bayesian Paradigm
• Laplace’s Rule of Succession
• Conjugate Prior Distribution
• Laplace Prior Distribution
• Jeffreys Prior
• Credible Interval
  • Differences Between Credible and Confidence Intervals
  • Highest Posterior Density Credible Interval
• Hypothesis Testing via Bayes Factors

References

• Casella. (2001). Statistical Inference(2nd Edition)
• Hogg et al. (2013). Introduction to Mathematical Statistics(7th Edition)
• 김달호. (2013). R과 WinBUGS를 이용한 베이지안 통계학