By Stephanie Simone
</span>
<span itemprop=”>
Hina Gandhi,software engineering technical leader,Cisco, offered tips and techniques to pave the way for autonomous, efficient data pipelines that continuously adapt to changing workloads and infrastructure dynamics, during her Data Summit 2026 session, “Teaching Spark to Tune Itself: Reinforcement Learning for Smarter Optimization.”
The annualData Summitconference returned to Boston, May 6-7, 2026, with pre-conference workshops on May 5.
Existing optimization methods leave a pre-execution intelligence gap, she said.
Manual tuning requires deep domain expertise, is highly time consuming, and instantly breaks when the underlying data patterns shift.
Adaptive query execution kicks in after initial overhead and poor task scheduling has already occurred. The missing link is pre-execution intelligence, she explained.
Gandhi introduced an approach that empowers Spark to learn how to optimize itself—using reinforcement learning (RL), specifically Q-learning, to dynamically choose the most efficient partition strategies at runtime.
She explored how an RL agent can observe key performance signals—such as shuffle size, task duration, data skew, and executor utilization—and iteratively refine its partitioning decisions to minimize latency and resource cost.
The agent instantly perceives the state and selects an action based on memory. The agent applies the configuration and learns strictly from execution time.
The agent completes the continuous learning cycle by executing the job, measuring the exact performance, and updating its internal memory with a mathematical reward signal.
“Infrastructure must learn from experience rather than rely on static rules,” Gandhi said.
Many Data Summit 2026 presentations are available for review athttps://www.dbta.com/datasummit/2026/presentations.aspx.