First and foremost, I think the ultimate goal for HA is "automation." So pulling out my phone to do everything is unacceptable, but so is for the most part using my TV to control my lights (or a box connected to my TV). I want my house to respond to me, my family, and our movements/actions. I don't want to have to tell it what to do. Perhaps a Nest style "learning" system would be ok as a "starter" for people, but ultimately the goal is to not interfere with it unless there is a special need to (in which case wall switches should be acceptable overrides). This is why I don't normally use "smart bulbs" except in lamps or where I can use a scene controller on a wall to control them. I don't always carry my phone or tablet and to have to pull one out to turn on a light is dumb in my opinion.