I don't need a standard/simple answer, I need someone that has really looked into cleaning down.  30 years ago while working at a backpacking store in the Midwest I sent a warm Snow Lion down sleeping bag and a very warm down jacket (North Face I think) to a speciality shop in California that cleaned down things.  When they came back they both were very much less warm and less loft.

I bought a Gortex covered Marmot Gopher 5 degree bag that I have needed for warmth and used for over 30 years backpacking about 2 weeks a year.  I've never trusted washing it because of that previous experience.  I've washed off the surface area where my face is, but always have used with clothes and have been very care careful with it and have noticed no warmth difference in the 30 years.

Am I better off just leaving it alone or would it actually help to clean it now?

Maybe an REI person knows the answer.