I tried feeding a 7s video of my bookshelf into Gemini Pro 1.5 to get back a JSON array of books... and it worked!
https://simonwillison.net/2024/Feb/21/gemini-pro-video/