Optimization

In mathematics, finding the maximum and minimum values of a given function $f: X \to \mathbb{R}$ is called optimization. When this objective function is directly related to problems in our lives, it receives great attention from the perspective of applied mathematics, and the importance of optimization theory that solves such problems is undeniable.

• Optimization Techniques in Mathematics
  • Optimal Values: Maximum and Minimum $\max$, $\min$
  • Optimal Solutions: Maximizer and Minimizer $\text{argmax}$, $\text{argmin}$
  • First-Order Necessary Conditions
  • Second-Order Necessary/Sufficient Conditions
• 🔒(25/11/25)Multi-Objective Optimization: Pareto Optimization
  • 🔒(25/11/29)Pareto Front
  • Alternating Optimization

Linear Programming

• Linear Programming Problem

Standard Form

• Equation Form
  • Existence of Optimal Solutions
• Basic Feasible Solutions
  • Uniqueness of Basic Feasible Solutions
  • If an Optimal Solution Exists, One of Them is a Basic Feasible Solution
• Proof of Fundamental Theorem of Linear Programming

Simplex Method

• Dictionary and Tableau
• Simplex Method
  • Initialization and Auxiliary Problem
  • Unboundedness of Objective Function
  • Cycling in Simplex Method
• Bland’s Rule for Simplex Method

Duality

• Primal and Dual Problems
• Proof of Weak Duality Theorem
• Proof of Strong Duality Theorem

Practice

• How to Solve Linear Programming Problems with Excel
• How to Solve Linear Programming Problems with Julia JuMP.jl
• How to Solve Linear Programming Problems with Python scipy
• How to Solve Linear Programming Problems with MATLAB Optimization Toolbox
• How to Solve Linear Programming Problems with R lpSolve

Nonlinear Programming

• Lagrange Multiplier Method

Gradient Descent

• Gradient Descent Method
• Stochastic Gradient Descent
• Newton’s Method
• 🔒Conjugate Gradient Method
• Subgradient
• Subgradient Method

Proximal Algorithms

• Proximal Operator $\operatorname{prox}_{\lambda f} (\mathbf{x})$
• Proximal Minimization Algorithm $\mathbf{x}^{(k+1)} = \operatorname{prox}_{\lambda f}(\mathbf{x}^{(k)})$
• Proximal Gradient Method $\mathbf{x}^{(k+1)} = \operatorname{prox}_{\lambda g}(\mathbf{x}^{(k)} - \lambda \nabla f(\mathbf{x}^{(k)}))$
  • 🔒(25/12/03)ISTA: Iterative Soft Thresholding Algorithm
  • 🔒(25/12/07)FISTA: Fast Iterative Soft Thresholding Algorithm
• PALM(Proximal Alternating Linearzied Minimization)

Heuristics

• 🔒(25/12/11)Tabu Search in Optimization Theory
• 🔒(25/12/15)What is the Population Method in Optimization Theory?

Genetic Algorithms

Particle Swarm

Main References

• Luenberger. (2021). Linear and Nonlinear Programming (5th Edition)
• Matousek. (2007). Understanding and Using Linear Programming
• Vanderbei. (2020). Linear Programming(5th Edition)
• Kochenderfer. (2025). Algorithms for Optimization(2nd Edition)