Hey,
I thought about the bracket install also. The type they use to put a kickstand on 2-stroke bikes. Unfortunately, it would not look so great, hold very well, and would probably mar up your alum. swingarm.
I think you would be just fine to drill a couple holes in your swingarm. Just make sure you use very sharp tools. I would coat the drill/tap with grease to try to catch all the chips, instead of letting them fall into the swingarm. You could also install Heli-coils in the holes to ensure that the steel bolts will have steel threads to screw into and not soft aluminum ones.
Aluminum is a rather soft material, it can crack, especially around welds. But, it tends to bend and distort a good bit first. Just do it. Good luck.