DLA Rules of Thumb

p = |feats|

First rule of thumb: there really are no hard and fast rules that always apply but these are best places to start.

DLA

  • Feat_occ_filter (--feat_occ_filter):

    • Rule of thumb: N/2 < p < N
  • Depends on:

    • Expected effect sizes
    • Sparsity of your observations
    • How many words do you have per person? (measure how well we are estimating word use rates)
    • "True" strength of relationship

Colloc_filter

  • See --feat_colloc_filter

  • When to apply?

    • good for DLA
    • usually not good for prediction (less accurate models)
  • generally a pmi threshold of 3 works for anything from 2grams to 4grams.

Prediction

  • Feat_occ_filter

    • See --feat_occ_filter
    • Rule of thumb: N < p < 2*N
    • (when you’re doing "magic sauce"”" feature selection or LASSO (L1) penalization)
  • Colloc filter: doesn’t usually help (sometimes hurts)

  • What usually works best

    • Regression (listed in order to what usually works best)

      • Feat_occ_filter => Univariate selection => PCA => L2 (ridge) regression (--feature_selection magic sauce ---model ridgecv)
      • LASSO L1 regression (with no separate feature selection)
      • ElasticNet L1L2 regression (presumably worse because there is one more hyper-parameter to set)
    • Choosing dimensions for PCA

      • If a lot of observations (>>10k): 10% of p
      • If few observations (< 10k) 50% of N
      • (see more complicated funtions in regressionPredictor.py featureSelectionString)
    • Classification

      • L1 linear-svm (--model linear-svc)
      • L1 logistic regression (--model lr)
      • Extremely randomized trees (--model etc)

Levels of analysis and group frequency threshold

  • See --group_freq_thresh

  • County level:

    • Ok to push the boundaries for p (use lots of features compared to observations), because you have well-estimated features
    • GFT: 20 to 50k range; (if really good data, use 50k)
  • User-level:

    • Above rules for p directly apply.
    • GFT: 1k (500 if N < 5k; 2k if N > 100k usually has absolutely no benefit)
  • Message-level:

    • Rules above apply except, normally best to use binary encoding of 1to3grams
    • GFT: 1 (but depends on task)