Well you technically don't need the SD card at all to operate the phone, some applications use it though. Nothing disastrous will happen if you wipe it though or move things around. I have used 3 different SD cards throughout the life of my phone, each I started with a clean slate (formatted the card). The apps will just make their folders again and go on as usual.
Whatever apps you were using that gave you those wallpaper folders will re-make them again if you run them again. So you may be kinda fighting a losing battle if you plan on running those apps more. But your phone won't explode if you do decide to tidy up.