On 21 May, 2025, our Editor Dr Paul Bilokon interviewed Dr Ernest P. Chan, an expert in machine learning and the application of quantitative models for asset management, about his latest book, Generative AI for Trading and Asset Management (co-authored with Hamlet Jesse Medina Ruiz, senior data scientist at Criteo). In this article we reproduce this conversation for the benefit of our readers.
PB: Ernie, you have been a pioneer not just of electronic trading and algorithmic trading, but also of trading education. If I am not mistaken, your first book, Quantitative Trading, was published in 2008. When I talk to people in finance, I find that people have learned such concepts as pairs trading, cointegration, and Kalman filtering from your books. There is the idea of alpha decay: once enough people know about a trading strategy, it eventually disappears. Are you worried about disclosing trading secrets? What’s in it for you?
EC: As with many researchers, I have found that I only understand a topic deeply if I have to explain it to someone. So that’s the primary benefit. Also, once I start the discussions, readers often share with me their knowledge and ideas, so I would say I am the net recipient of this knowledge exchange!
PB: You have just published another book in 2025 – Hands-On AI Trading with Python, QuantConnect, and AWS. Can you tell us more about that book?
I contributed one chapter in this book, which is a summary of our work in Predictnow.ai in applying AI for risk management and portfolio optimization. I also discussed in depth about the thankless but absolutely crucial step of financial features engineering. In general, the book is an excellent compilation of the expertise of several AI practitioners, and how these ideas can be implemented on QuantConnect.
PB: Our conversation, however, is about your latest book, which hasn’t yet been published, namely Generative AI for Trading and Asset Management. What’s the relationship between this work and Hands-On AI Trading with Python… as both seem to be related to AI?
EC: Actually, Generative AI for Trading and Asset Management has just been published – by the way, thank you for your warm praise on the book cover, Paul! The Hands-On AI book is about how to apply “discriminative AI” to finance, which is what most quants learned in school. The GenAI book is about the newer concept of “generative AI”, and how to apply it to finance. Contrary to popular understanding, GenAI is not just about text or images or chatbots. It is about high dimensional probability distributions that can be applicable to any data, especially time series data.
PB: So AI – is it hype? Is it real? Does it work in finance?
EC: AI is real, it is here, and it works. We better get on with the program!
PB: Some critics will say that financial data is too noisy for neural networks, and even on Kaggle it’s usually the ensemble models (such as boosting) that lead for such tabular datasets. Would you agree with these critics?
EC: There is some merit in the argument that tree models are better for financial data due to their inherent features selection ability. But as I wrote in a recent blog post on gatambook.substack.com, the attention mechanism in neural networks has proven quite effective as a sample-specific feature selection method, so I think this removes one weakness of NN. Furthermore, the power of deep NN is that one can pre-train a model on a vast amount of time series data, and then finetune on specific data we want to trade on. This reduces the overfitting issue.
PB: You mention no-code generative AI in your latest book – does this imply that there is no longer an absolute requirement to be a coder if a novice wants to become a trader?
EC: Yes, many chatbots can now turn ideas into codes quite easily. Of course, one still needs to be able to review, understand, and test the codes to ensure they are correct!
PB: Your latest book covers deep autoregressive models, deep latent variable models, and flow models in considerable depth. Can you briefly tell us why these models are important?
EC: Deep autoregressive models are a nonlinear extension of the simple linear autoregressive models that we have been taught in time series analysis courses. Deep latent variable models are a nonlinear extension of the simple linear latent variable models such as PCA.
Flow models extend the basic idea of transforming one probability distribution into another using a single invertible transformation, by composing multiple nonlinear and invertible transformations to model highly complex distributions.
PB: You have a chapter dedicated to sentiment analysis in trading. Is it possible to generate sustainable alpha using sentiment analysis?
EC: Yes, and everybody are already doing it.
PB: In Chapter 10 you talk about dedicated models, such as FinBERT. Are these models the future, or do you believe more in adapting the off-the-shelf stuff?
EC: FinBERT is off-the shelf. It is free and open source. Unless one has a multi-million-dollar budget for training your own LLMs, one must rely on pre-trained models one can use for free or cheaply.
PB: What’s your best advice for novices venturing into financial AI?
EC: Conceptual understanding of the markets is more important than learning a specific mathematical or computer science technique. After that, AI can certainly help improve the models, generate new ideas, or turn them into codes.
PB: Would you invite a Kaggle grandmaster to join your desk as a trader?
EC: Sure, if they are willing to dive deep into finance and gain domain-specific knowledge