Families of $A_\infty$ algebras and homotopy group actions