There might be several reasons, why one might encrypt to a third party which is not recipient of the mail. This may be an occasional 'encrypt to self', there might be a companies directive, to encrypt with a general key in addition or it might be just a spleen of the mails sender... Mailclients like thunderbird give the option, to choose all the keys, a mail shall be encrypted with. While editing the gpg.conf file might be an option in some certain cases, in other cases that's not applicable.
I also run into this problem. Any chance that you allow to encrypt with more keys than just the recipients?
Perhaps one day someone will implement this in CM, with a clear and usable user interface. In the meantime, you can use encrypt-to or hidden-encrypt-to in your GPG configuration. https://www.gnupg.org/documentation/manuals/gnupg/GPG-Key-related-Options.html#GPG-Key-related-Options