Yes and No.
Edit: While we will be wrestling with privacy for a while yet, and your point about the mobile phone camera is a good point, that will be a long running issue and really how expensive are some curtains?

However, in the cases of airliner airspace (drones are not supposed to go over 400 ft and this one was at 2300 ft in the flight path as well as being within the 5mile 'no fly without alerting FAA' zone so they had no good reason to be there-to this day they still do not know whose drone that was) or hanging out around the French Reactors where they are clearly in an area they are not supposed to be they can and should be brought down. The system is not designed to be scouring the whole countryside, but rather set up at defensive points. The intruding drone has to enter their region to trigger the response.
Plus while you are going on about destroying property, note the string in one motor forces the drone to land as it no longer has sufficient lift and while it may be damaged in a rough landing, many will survive the bump. This is not being shot down out of the sky with AA ammo.
I would rather people just be nice and respect one another and their space, but we have seen how well that idea works out.