Machine Learning

Machine Learning^{machine learning} is the process of enabling a machine to learn methods for identifying features from an existing dataset so that it can effectively recognize features in new data as well.

The above definition isn’t particularly rigorous, nor does it need to be. Simply put, a machine can be understood as a computer or programming code. The dataset already in possession and used for learning is called the training set^{training set, 훈련집합}. Comparing machine learning to a student studying for exams:

• Machine: Student
• Training set: Past exam papers
• Features: Question patterns
• New data: Actual exam questions
• Learning: Solving past papers to be able to tackle actual exam questions

As of 2021, the most widely used method for implementing machine learning is deep learning^{deep learning, 심층학습}, which involves increasing the hidden layers of artificial neural networks. Artificial neural networks have recently seen dramatic improvements in performance, often showcasing the best results. Before deep learning achieved satisfactory performance, models based on statistical theories dominated machine learning.

To become proficient in machine learning, one needs to be skilled in mathematics, statistics, and programming. Mathematical and statistical knowledge is required to understand the theory, and programming skills are needed to implement it. Specifically, to deeply study machine learning theory, knowledge of linear algebra related to matrices, as well as measure theory, functional analysis, and others are necessary. Additionally, recent research is linking artificial intelligence with fields like geometry, graph theory, and partial differential equations¹.

The following articles are written to be as accessible as possible for mathematics majors.

• Free Online Resources for Artificial Intelligence, Machine Learning, and Deep Learning

Basics

Learning Concepts

• Supervised and Unsupervised Learning
• What are Training/Validation/Test Sets?
  • One-Hot Encoding
  • Data Augmentation
  • Commonly Used Datasets
    • MNIST Database
    • Iris Dataset
• Online Learning, Batch Learning, Mini-Batch Learning

Optimization

• Weights
• Loss Function
• Gradient Descent and Stochastic Gradient Descent^SGD
  • Momentum Methods, Nesterov Accelerated Gradient^NAG
  • Adaptive Learning Rates: AdaGrad, RMSProp, Adam
• Graduate Student Descent
  • Grid Search, Brute Force, Manual Effort

Sampling

• What is the Monte Carlo Method?
• Monte Carlo Integration
• Rejection Sampling
• Importance Sampling
• Markov Chain Monte Carlo^MCMC

Classical Machine Learning

• Positive Definite Kernels and Reproducing Kernel Hilbert Space $k$, $H_{k}$
• Proof of the Representer Theorem

Linear Regression Models

• Linear Regression Model
  • Gradient Descent
  • Least Squares Method
  • Maximum Likelihood
• Support Vector Machine

Linear Classification Models

• Linear Classification Models
  • Least Squares Method
• Fisher’s Linear Discriminant
• Neyman-Pearson Criterion for Binary Classification
• Bayes Risk Classifier

Clustering

• k-Means Clustering
• Hierarchical Clustering

Subfields

Reinforcement Learning

• 🔒 Mathematical Foundations for Reinforcement Learning
• 🔒 What is Reinforcement Learning
  • 🔒 Value Functions
  • 🔒 Exploration and Exploitation
• 🔒 Multi-Armed Bandit Problem
  • 🔒 Implementing Epsilon-Greedy in Julia
• 🔒 Markov (Reward) Process
• 🔒 Markov Decision Process

Computer Vision

• What is Computer Vision
• Noise and Artifacts
  • Salt and Pepper Noise

Deep Learning Theory

• Layer
  • Linear Layer
  • Convolutional Layer
  • Skip Connection
• Activation Functions
  • ReLU $\operatorname{ReLU}(x) = \max \left\{ 0, x \right\}$
  • Softplus $\zeta(x) = \ln (1 + e^{x})$
  • Sigmoid
  • Logistic $\sigma(x) = 1/(1+e^{-x})$
  • Softmax $\operatorname{softmax}(x)$
• What is an Artificial Neural Network^ANN
  • [Meanings and Differences of ANN (Artificial Neural Network), DNN (Deep Neural Network), FNN (Feedforward Neural Network)]
• Definition of Perceptron
  • Perceptron Convergence Theorem
• What is Deep Learning?
  • Backpropagation Algorithm
• Automatic Differentiation
  • Automatic Differentiation and Dual Numbers
  • Implementing Forward-Mode Automatic Differentiation with Dual Numbers in Julia
• Mathematical Foundations of Deep Learning, Proof of the Cybenko Theorem
  • What is a Discriminant Function?
  • What is a Sigmoidal Function?
• What is Continual Learning in Deep Learning
• Overfitting and Regularization
  • Dropout
  • Paper Review: Do We Need Zero Training Loss After Achieving Zero Training Error?
• Boltzmann Machine
  • Restricted Boltzmann Machine
  • Batch Learning Algorithm
  • Online Learning Algorithm
  • RBM for Classification
• Radial Basis Function
• [MLP (Multilayer Perceptron)]
• [CNN (Convolutional Neural Network)]
• PINN (Physics-Informed Neural Networks) Paper Review
• Autoencoder
• DeepONet (Deep Operator Networks) Paper Review
  • Imprementation in PyTorch
  • Imprementation in Julia
• U-Net Paper Review
  • Imprementation in Julia
• KAN(Kolmogorov-Arnold Neural Network) Paper Review

Deep Learning Frameworks

• Cheat Sheet: Equivalent Codes in Flux, PyTorch, TensorFlow

Python PyTorch

• How to Check Model/Tensor Device .get_device()
• Random Sampling from a Given Distribution torch.distributions.Distribution().sample()
• Creating and Using Custom Datasets with Numpy Arrays TensorDataset, DataLoader
• Saving and Loading Weights, Models, and Optimizers torch.save(model.state_dict())

Neural Networks

• Implementing a Multilayer Perceptron
• Defining Neural Networks with Lists and Loops nn.ModuleList
• Accessing Model Weights and Biases .weight.data, .bias.data
• Weight Initialization torch.nn.init

Tensors

• Modular Arithmetic fmod, remainder
• Handling Dimensions and Sizes .dim(), .ndim, .view(), .reshape(), .shape, .size()
• Creating Random Permutations and Shuffling Tensors torch.randperm, tensor[indices]
• Deep Copying Tensors .clone()
• Concatenating and Stacking Tensors torch.cat(), torch.stack()
• Padding Tensors torch.nn.functional.pad()
• Sorting Tensors torch.sort(), torch.argsort()

Troubleshooting

• Fixing ‘RuntimeError: grad can be implicitly created only for scalar outputs’
• Fixing ‘TypeError: can’t convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first’ with Lists
• Fixing ‘RuntimeError: Boolean value of Tensor with more than one value is ambiguous’
• Fixing ‘RuntimeError: Parent directory does not exist’ When Saving Models

Julia

• Julia’s Deep Learning Frameworks Flux.jl, Knet.jl, Lux.jl
• Using Machine Learning Datasets MLDatasets.jl

Flux

• Handling Hidden Layers
• Implementing MLP and Optimizing with Gradient Descent
• One-Hot Encoding onehot(), onebatch(), onecold()
• Implementing MLP for Nonlinear Function Approximation
• Implementing MLP and Training on MNIST
• Using GPU
• Setting Neural Network Training/Test Modes trainmode!, testmode!

References

• Christoper M. Bishop, Pattern Recognition annd Machine Learning (2006)
• Simon Haykin, Neural Networks and Learning Machines (3rd Edition, 2009)
• Trevor Hastie, The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd Edition, 2017)
• 오일석, 기계 학습(MACHINE LEARNING) (2017)
• Richard S. Sutton, Reinforcement Learning: An Introduction (2nd Edition, 2018)