Press icon, wait, drag down towards app drawer which changes into a trash can, when the icon under your finger turns red - drop it. N.B. This removes it from your home screen. To delete an app - go into Settings, find the applications submenu, and follow the instructions there.
All the screen is, is like a desktop and the shortcuts are just like icons or "doorways" to programs... you can delete the program but the door is still there and it leads to nowhere.
Once you press and hold, only then will the trashcan appear in the lower right.