Proactive AI That Doesn’t Annoy Users: Building Context-Aware Notification Systems

The majority of current notification systems for AI fail, for either too much noise training users to ignore the notifications or too much conservatism missing important signals. Proactive AI at scale lessons learned from designing and operating such service on real world product. Authors started with quite optimistic assumptions that more intelligence would lead to more engagement after couple of iterations (and lots of usage), they learned that first version of the product was pretty noisy and unreliable. Most importantly, after couple of iterations, they learned that the issue wasn’t actually in the model, but in how you framed the decisions. In this talk I explain the thought process behind some design choices I made in shifting from an automatic system to a more deliberate one. In this session user preferences for things like consent and frequency limits as well as quiet hours are implemented. Then I’ll explain how the system determines if it has enough context to make a different decision. If the system doesn’t have enough information OR it has inconsistent information then the system does nothing –there is no fallback or force engage. My talk goes through how I used multiple user signals, user behavioral preferences, real world events, and timing to make a series of decisions leading up to a decision point. The hardest part was getting others to understand that suppression was a valid outcome and NOT a failure. We'll cover these core concepts as well as advanced topics such as how system handles duplicate events, user longitudinal data, and what we mean by engagement versus trust. This talk is for engineers who build AI powered product features, not just demo the features to users but also to systems, to others who are engineering similar features and are trying to understand trade offs.