Rather than using your mouse to invoke Photoshop’s color/patter Fill dialog box, you can use the quick keyboard shortcut, which is Shift + Delete.