Brushed stainless will have microscopic "open grains" that will hold residue.
We do stone engraving and sometimes have issues with adhesive backed sandblasting stencils leaving a image on open-grained (not polished) stone and it takes months to dissipate outdoors with the UV. When that happens, we make a poultice from stone dust and solvent. When the solvent evaporates, capillary action pulls the contaminates up into the poultice and is then swept away.
In your case, it's going to be difficult to hang a poultice on a door. I would take a piece of news print or kraft paper and hold it against the wall and mist it with solvent from a spray bottle. As it saturates, it will naturally stick to the metal, and when it's completely dried it will fall to the floor. Do it again, but use an aqueous based cleaner instead of a solvent. The capillary action of the cellulose fibers may just do the trick in pulling out the contaminants.
If all else fails, use rubbing (buffing) compound or swirl mark remover and polish in the direction of the grain. I grew up in a body shop and I've seen the miracles that stuff can work.
JB
