The Sched app allows you to build your schedule but is not a substitute for your event registration. You must be registered for PyTorch Conference Europe 2026 to participate in the sessions. If you have not registered but would like to join us, please go to the event registration page to purchase a registration.
This schedule is automatically displayed in CEST (UTC/GMT +2). To see the schedule in your preferred timezone, please select from the drop-down menu to the right, above "Filter by Date."
Sign up or log in to add sessions to your schedule and sync them to your phone or calendar.
FlexAttention democratized attention research by letting researchers prototype custom attention variants in PyTorch without hand-written CUDA. Over 1,000 repos have adopted it, and dozens of papers cite it. But flexibility came at a cost: FlexAttention achieved only ~60% of FlashAttention-3's throughput on Hopper, and the gap widened dramatically on Blackwell GPUs.
We bridged this gap by integrating FlexAttention with FlashAttention-4, the new CuTeDSL-based implementation optimized for Blackwell's async pipelines and tensor memory. PyTorch's Inductor now generates CuTeDSL score/mask modifications directly, enabling JIT instantiation of FA4 for arbitrary attention variants.
Results: 1.2–3.2× speedups over the Triton backend on compute-bound workloads. On B200, patterns like ALiBi, document masking, and sliding window see up to 2.7× forward and 3× backward speedups. On Hopper, gains range from 1.3–2× across all sequence lengths.
This talk covers the technical integration: how Inductor lowers score mods to CuTeDSL, how FA4's warp-specialized kernel accommodates block-sparse iteration, and practical considerations for users adopting the Flash backend today.
I am currently a machine learning engineer working on core development of PyTorch. I received my Masters in Computer Science from the University of Illinois at Urbana-Champaign. I received a dual degree in Physics and Applied Mathematics from The Ohio State University. I also won... Read More →
Tuesday April 7, 2026 12:15 - 12:25 CEST Master Stage
