I was having a meaningful conversation with a great reviewer of my coming book. He's a fmr Barra researcher, now at a HF. It was about model development and research process. It helped clarify two different approaches. My daily thought thread (🗓️💭 🧵)(1/n)
First approach: serial. You estimate a model, explore it, think about it, compare performance to the previous one, critique, propose alternative, iterate until convergence. (2/n)
Second approach: parallel. You formulate a model family, estimate all models at once, account for family complexity, and select your model. (3/n)
The first approach is what everyone did until relatively recently. Think stepwise selection in linear regression. Like unprotected sex, you know you should not do, but you do it anyway, hoping for the best. (4/n)
The second approach (to keep the previous bad analogy), is like having sex with half of Europe at the same time while wearing 140M condoms. Morally even more questionable that option #1, really unusual, but if you have the technology to do it why not? (5/n)
I argued in my conversation that we should go with the parallel approach. Out of metaphor: I can estimate 10 yrs of a factor model on a 16-core machine and 256GB in a matter of minutes. Not even optimizing for GPUs. (6/n)
And schedulers allow me to parallelize across parameters with a couple of lines of code, across 10,000s (in some firms, up to 1M) of machines. I can estimate 1e6-1e8 models/day. Why wouldn't I do that? (7/n)
Mostly, because we don't really know how to (or are just used to) think about large-scale testing. But there are approaches. I have one in my book (ch.4); there are others. (8/n)
Gelman years ago asked the question "what would you do if you had *all* the data?". And I would add "and would you do if you had *all* the models?" I consider these to be pretty good questions. (9/n) n=9.
My personal recommendation, btw, is that you disconnect from carnal desires and think deeply about the questions I pose on X. (10/9)