Bizerk, you're going through some normal feelings. My husband broke himself 3 weeks ago (2 vertebrae) & I thought for half a second about selling the bikes & taking up knitting. I then thought about mum teaching me how to knit all them years ago & worked out it'll just make me grumpier than normal ;)
While he was in hospital (just overnight), I couldn't sleep & just started stressing out about what could've been. What was reality seemed irrelevant, it was more the "what if" that did it. I ride a road bike to work & on the Monday couldn't really face it. I called in sick as I'd had stuff all sleep & would've just worried about Henk all day anyway. The ride in on Tuesday I can't even remember, it wasn't a big deal. My first ride back on the dirtbike was freaky. I thought I'd got away with no repercussions & everything was cool, until I came around a corner & found the riders I'd sort of been chasing sitting around (I'd left ages after them & didn't really expect to see them & definitely not that close in - it was about 200m away from where Henk had broken his back). I immediately freaked out & asked what was wrong. Turned out it was just a bike problem.
I ride alone a lot & still do. Henk never had a problem with it but now sometimes does, the same goes with the other guys - in some ways they've become more protective. I don't go far & make sure someone knows where I am & carry a cellphone. The funny thing about the day Henk crashed was I was riding badly & I remember thinking that if I crashed, it'd take some time for the guys to work out I'd crashed. I look at the whole accident as a wake-up call - what we do can be/is dangerous & we have to be aware of the possible consequences & weigh it all up. Nothing may happen, but something could.
When Henk crashed, he'd stopped concentrating on what he was doing & got out of control by hitting a stump. In about 8-10 weeks, he'll be back on the bike & I think that's going to be the worst time for me. The ghosts will return but they will go away for a while.
You'll get through this time, as your brother will & in the meantime, keep your chin up, find someone to go riding with & decide then whether you want to continue with the sport or not. It's not all doom & gloom, there's a lot of fun inbetween but it's an individual choice none of us can make for you. Just don't rush into it. Let us know how the ride goes (if you decide to go) :)