The only serious harm I see from the recreational use of nitrous oxide is #FuckinNangs littering the streets. They're made of aluminum, so theoretically recyclable. But a single unpopped canister mixed in with the empties could be very dangerous when they crush them. So metal recyclers usually won't take them.
If Dr Shane Reti wants to regulate nitrous, ban those bloody little canisters. So you can only buy it in reusable tanks.