top of page

One model to rule them all: How Gemini is set to change Racehorse Selection

Today the team at Google DeepMind launched a product that has the potential to change the game for racehorse selection: Gemini, the new multimodal Large Language AI Model which Google is now rolling out across their various products.



What separates Gemini from other AI models is that Gemini is built from the ground up for multimodality — reasoning seamlessly across text, images, video, audio, and code as input values.


If you've been following my previous posts, you know that we have explored various ways to analyze yearlings for selection by looking at pedigree data and computer vision models to assess physical conformation, movement and cardiovascular parameters.



To get this to work, we have had to 'stitch together' the outputs of various computer vision models as a dataset to then feed the output data from those models into a single tabular model that serves up a final prediction. So for each horse we get say the probability output of the biomechanics video model, the conformation image model, the cardiovascular video model and the DNA markers and then train another model to look at the probabilities from each of these models to generate a final prediction.


But what if there was a single model that could process all this information together in its raw format, instead of relying on separate models and concatenating them together like we have to date?


Enter Gemini: This groundbreaking model transcends the limitations of single-modal approaches. Unlike a standard video classification model that only analyzes visual data, Gemini can integrate information from multiple modalities at once and return a predictive output, so effectively for one horse we could have:


Video: Capturing the horse's biomechanical movement and cardiovascular parameters

Image: Capturing the horses conformation.

Other Data: Other derived data such as bone lengths or joint angles from the analysis of the Video (using DeepLabCut) or Image.

Pedigree/DNA: Providing genetic information and ancestral performance data.


By combining these diverse sources of data and weighting them properly as a single prediction, Gemini would be able to give us a more comprehensive picture of each horse's potential. The implementation of Gemini into our machine learninhg platform should bring a number of benefits.

  1. Improved Accuracy: Combining multiple data sources leads to more accurate predictions compared to single-modal models.

  2. Reduced Bias: Gemini incorporates different types of data, mitigating the potential bias inherent in any single modality (eg one of the smaller models outweighing another significantly that biases the tabular model)

  3. Improved Repeatability: Multiple measurements of the same horse taken at different times of the year should result in more highly correlated predictions within the horse.

  4. Increased Efficiency: Eliminates the need for multiple separate models, streamlining the analysis process.

  5. Deeper Insights: Provides a holistic view of each horse, allowing for better-informed selection and racehorse management decisions (e.g once all the data is in there and you know the results of previous horses, you could ask Gemini "is this horse more likely to want turf or dirt to race on and what would its preferred distance be?)

Gemini, or more specifically a model capable of handling raw data in a multi-modal fashion, has always been a goal of mine to integrate so I am excited by the potential of this announcement from Google. While it isn't going to be available for commerical use until mid-December and I will take some time for me to get this new model together and into production (it isn't clear if it can be fine tuned to domain specific data or not, I presume it can be) I hope to have it up and going early in the new year.


We will also have some news in Q1 of the new year on the Pedigree Analysis front in terms of having a machine learning model capable of generating a pedigree shortlist for a sale. This shortlist would be the cornerstone of the yearling selection pipeline as we would then gather all the information we can on the horses on that list which should result in a very highly precise final selection list. More on that front later.


Before I go (if the above is not enough!), I'd like to share some exciting news! I'll be heading to Australia and New Zealand in early 2024 for the yearling sales with a new venture in mind.


There is a bit to tie down but I plan to start a racing syndicate in Australia that will be specifically exploiting an opportunbity in the market we have identified and using the technology I have to purchase yearlings and race with leading trainers there. The industry is thriving in Australia and I will post a little more about my plans there after Christmas but if you're interested in getting involved in this exciting venture, please don't hesitate to get in touch.


481 views2 comments

Subscribe to our email list and get the latest post straight to your inbox

Thanks for submitting!

bottom of page