I found the solution, here it is, I used UrlEncode
on the filename which helped me solve my problem.
Response.AddHeader("Content-Disposition", String.Format("attachment; filename={0}", HttpUtility.UrlEncode(docFileDTO.FileName)));
solved Japenese character in a filename changes to garbage if I download file from IE 11