Ugh...
Fragmentation of data occurs on any file system. SSD or HDD or hybrid, doesn't matter. EVEN YOUR COMPUTERS DDR RAM FRAGMENTS!
Defragmenting your SSD based memory WILL shorten its life span. HOWEVER, even with heavy use, you will never get to the end of your SD cards life-span, in your own life time. So, that being said, if your planning to give your children a hand me down USB stick, or SD card, then please do not defragment .. your children/grandchildren will be disappointed.
SSD memory has no moving parts, it stores data on "blocks" and "pages" as opposed to a round disk. Think of your HDD as a small record player. It uses a needle and head, if you will, to read/write data. Where as an SSD uses uniform blocks and pages. Huge difference, BUT ALL DATA WRITTEN AND READ, CAN FRAGMENT.
Think of fragmentation as writing with a pencil, to paper. When you erase a letter or a word, you can still see a "ghost image" of what you erased. This is fragmentation.
Just because SSD seek time is much lower than an HDD, doesn't make it pointless to speed it up. Use a defragmenting program such as O&O so that you can use different methods of defragmenting your memory to see which suits you best. You may find that defragmenting your card is useless to you.