I’m Richard Demsyn-Jones, and this is my technical blog on ML and related topics. I have spent a lot of time modeling human behavior, in domains including search engine ranking, spam, and credit risk. This is my place to put some ideas into words, and force myself to think them through in the process. Please let me know if I’m wrong about anything. Any feedback is welcome.
While my opinions are informed from my work experience, they are solely my own and do not represent my current or former employers.
Thanks for reading! See below for an explanation of the name and theme.
In machine learning research, a model is state of the art (SOTA) when it sets the high score. Researchers compete against well-known datasets, and typically win with innovative and novel complexities.
Yet if you pull back the curtain at successful tech companies and peak at their core models, you will more often spot established ML techniques than SOTA models. This can be true even at the same companies that have cutting-edge researchers publishing new techniques. In applied ML (applied to the world out there), especially in domains with human behavior, we need novel simplicities as much as novel complexities. SOTA in papers can diverge from SOTA in product, due to cherry picking, weak generalization, loss functions only approximating ultimate objectives, and the challenge of integration alongside other models or rules.
Many of our domains are moving underneath us, with feedback loops and changing behavior. Success occurs when models improve performance on their datasets, generalize to new data, align with the broader objective, and evolve seamlessly through extensions and as the system around them changes. In complex domains, each new element is best added in the simplest manner that minimizes the risk of catastrophic mistakes.
This doesn’t mean we only use simple models. Some subset of SOTA research methods will become SOTA applied methods through adaptation and diffusion. Occasionally that happens immediately. Paradoxically, systems can stay equally intelligible despite growing in intricacy. We absorb complexity by making complex ideas simple, by refining them into their most essential form and polishing them into reusable components. We build frameworks to abstract away some of the inherent complexity. Consider how transformers went from esoteric and novel in their first few years to being used as a basic building block in a matter of years, or consider (more generally) the widespread growth of neural networks as frameworks reached maturity.
Sometimes a complex idea comes to mind easily, as we envision the most complete approach that exhaustively covers the problem domain with elaborate systems or models. Then just as much ingenuity is needed for ruthlessly and effectively simplifying. There is an art to creating something robust out of complex precursors. Simplicity is state of the art.
