We use a low expansion polyurethane foam for air sealing and insulating around the windows. It can also be used to fill out penetrations in your walls (around pipes etc). This one is called SIka Boom. You can get it online. You want to make sure your substrate is clean (no grease, dust, loose particles). For the Sika Boom AS adheres you don't need a primer. I would recommend dampening the substrate with water so the foam can cure properly (also prevents secondary foam expansion). Sika USA Sika C

This is a Crucial Step - Sealing up your exterior wall electrical boxes is one of those small steps that pays off big in cold climates. - Unsealed gaps let cold air in, driving up energy costs and causing condensation issues inside your walls. - Take a minute today to seal those boxes and keep your home warm, efficient, and protected all winter long. | MODERNMILT