As enterprises roll out production applications using AI model inferencing, they are finding that they are limited by the amount of memory that can be addressed by a GPU. This episode of Utilizing Tech features Steen Graham, founder of Metrum AI, discussing modern RAG and agentic AI applications with Ace Stryker and Stephen Foskett. Achieving the promise of AI requires access to data, and the memory required to deliver this is increasingly a focus of AI infrastructure providers. Technologies like DiskANN allow workloads to be offloaded to solid-state drives rather than system memory, and this surprisingly results in better performance. Another idea is to offload a large AI model to SSDs and deploy larger models on lower-cost GPUs, and this is showing a great deal of promise. Agentic AI in particular can be run in an asynchronous model, enabling them to take advantage of lower-spec hardware including older GPUs and accelerators, reduced RAM capacity and performance, and even all-CPU infrastructure. All of this suggests that AI can be run with less financial and power resources than generally assumed.
Apple Podcasts || Spotify || More Audio Links || YouTube Channel
Guest
Steen Graham is the Founder and CEO of Metrum AI. You can connect with Steen on LinkedIn and learn more about Metrum AI on their website.
Podcast Information
Stephen Foskett is the Organizer of the Tech Field Day Event Series and President of the Tech Field Day business unit, part of The Futurum Group. Connect with Stephen on LinkedIn and follow him on X/Twitter, on Mastodon, or on Bluesky. Learn more about Tech Field Day and information on upcoming events on their website.
Ace Stryker is the Director of Product Marketing at Solidigm. You can connect with Ace on LinkedIn and learn more about Solidigm and their AI efforts on their dedicated AI landing page or watch their AI Field Day presentations from the recent event.
Thank You for Listening to Utilizing Tech
Thank you for listening to Utilizing Tech Season 8: Utilizing AI at the Edge Presented by Solidigm. If you enjoyed this discussion, please subscribe in your favorite podcast application and consider leaving us a rating and a nice review on Apple Podcasts or Spotify. This podcast was brought to you by Solidigm and by Tech Field Day, part of The Futurum Group. For show notes and more episodes, head to our dedicated Utilizing Tech Website or find us on X/Twitter, on Bluesky, and Mastodon at Utilizing Tech.