Home | Tyler Burleigh – Tyler Burleigh

About me

Hi! I’m an AI engineer with a unique background in human psychology and data science. I currently work at Khan Academy where I develop AI systems and user experiences using Large Language Models with a focus on rigorous evaluation.

My latest blog posts

Challenging SAMRE: Comparing multi-round debate-style LLM evaluation to a robust (and much simpler) baseline

In this post, I re-evaluate a method that was recently published in arXiv, critiquing their baseline model and then designing a new baseline model that implements standard best practices for comparison with the new method. I find that the new evaluation method proposed in the paper does not perform better than this robust baseline. This serves to highlight the importance of implementing best practices in baseline models for comparison with new methods, as well as being skeptical of claims in research papers that compare new methods to baseline.

Jan 12, 2025

Tyler Burleigh