To be fair, you chose that setting :D You could equally well use a different PGP key for pass, or have your agent not cache the secrets.