We have the same issues with the process colors on the Edge. The result is always grainy and only looks good from a distance. The spot colors look fantastic, but I have never made process colors look good. The Spectratone settings give you some other options by allowing you to layer spot colors, but if the fill is not 100%, you have the same problem as with the process colors.
There is a learning curve to working with the halftones and LPI settings as I explained in Post #5. You would do well to invest a few feet of materials and a couple of hours in testing the results of different settings. Otherwise you're trying to earn a return on your investment with one arm tied behind your back.
Listed below are settings that will get you started if you use Omega. The Gerbertone STC Photo halftone may not be available in older versions of Omega. It's main value is that it has LPI settings control that isn't available in other Gerbertone halftones.
My recommended default settings in Omega:
Vector Fills
Classical Dot Halftone
LPI = 53 for gradients
LPI = 70.7 for most solids
LPI = 90 for solids in mid to lighter shades
Raster Fills
Gerbertone STC Photo Halftone
LPI = 53 for gradients
LPI = 70.7 for most solids
LPI = 90 for solids in mid to lighter shades
Output Color Profile
Turn off automatic color management in GSPPlot and manually select the color profile named
Gerber EDGE II 300 DPI CMYK.