Nonlinear Regression Models

Resources

Not covered: kernel smoothing, local weighting, moving averages, binning, loess (local estimation) etc.
Non-parametric regression -
- can factor in <math>y=f(x)+\mbox{other stuff}</math>
- confounding effects
- interactions
- can generalize to discrete and/or multivariate responses (logistic regression, etc.)
Example bases
- linear
- polynomial (Taylor series expansion)
  - why not?
  - it works... sort of
  - not good for smoothing: not "localized", not "parsimonious" ==> takes a lot of terms to get non-exactly polynomial
- See slide on general functions for tips on selected basis sets.
  - wavelet bases - smooth trends and spikes
    - can be "same" as wavelet transform, slowly
  - trigonometric (Fourier) - "frequency concept"
    - can be "same" as Fourier transform, slowly
  - Spline bases - general smoothing
    - We'll talk about these today. Good for general smoothing. General purpose, but do not preserve spikes.
Pick the basis for the eventual goal.

Highly controversial topic on spline fitting: fun reading [1]
Gauss and the "invention" of least squares: [2]
Big question: where to put the knots
- known inflection points
- quantiles
Only a few knot points can be fit with LS
If # parameters ~ # of data points -> fitting problems.
B-spline bases are equivalent, but the bases are much closer to orthogonal. Much more efficient and stable for solution than truncated polynomial terms. Slight change of bases, but fitted functions are the same.
In "R" --> bs (library "spline").
natural splines -- like b-splines, but with linear restriction near edges

cross-validation - leave out a data point, assess prediction value, optimizing tuning parameters to minimize prediction error
- computational concerns
- "generalized cross-validation" - way of approximating cross-validation
mixed model p-splines --
- fit generous number of knots
- place gaussian penalty on wiggly terms
- jointly estimate the coefficients and penalty
- ties in with mixed effects models.
- library spm - semipar: spm(y~(fx))
- there are good ways of estimating variance effects in mixed models : REML (less biased estimates than ML)
- package mgcv - gam
- NOTE: L1 penalty would be the same as a double-exponential distribution (penalty) "wiggle" terms
- Smoothing // estimation - L1 penalty tends not to help "smooth" as well. Select certain knots over -- why? better for model selection than for regression/smooth fitting.
- L2 penalty also known as ridge regression.
easily extends to additive models
can use standard data output. Can test whether coefficients are "0"