I know what your talking about. I carry a Sig P220ST daily at work. I don't really worry about a AD with the long heavy first dbl action shot. If I carried it cocked in SA mode I would be worried.
I bought my M9A1 for IDPA shooting but have been debating carring it at work on occasion,maybe on the days I drive. The Kydex holster and Uncle Mike mag holder does not offer the best set-up for work.
I polished the fireing pin and other trigger parts and now have a GREAT clean light trigger,light enough where it does worry me. with little or no creep, and the short stroke it scares me a bit. I almost wish it did have a saftey. For IDPA its fine,but in a REAL situation I'm not sure I could get my finger OFF or away from the trigger.
I think the more I shoot it at IDPA events, and the more I feel confident with it, the more likly I will be to erase these fears.
My first IDPA shoot I droped it on the ground when I drew it from the holster.
At least I know it won't fire when droped!