Theres no such thing as cleaning a car and not taking the chance of instilling scratches to some degree.
The California Duster is a good tool, but just like everything else, it needs to be used correctly. It very hard for me to try and explain things over text like this, It'll be better if I could show you in person, but I'll do my best. Plus I'm on a good deal of drugs from my surgery yesterday, so I hope I make some type of sense
Most everyone just rests the duster with it's entire weight on the paint surface, then just wipe back and forth. The correct way to use it is to lightly lay the ends of the duster on the paint, only using about the first 1-2 inches of the fibers. Then every stroke or two, shake it out by turning is upside down and shake it by twisting it back and forth pretty aggressively. Then back on the paint for another swipe or two, then shake. The goal isn't a perfectly clean surface, but instead, you're just trying to get the larger dust/dirt off. Then you'll want to wipe it down with a quick detailer and a nice microfiber cloth.
Hope this makes sense, please let me know if I need to elaborate more on anything.