Having to double assign a ticket from one group to another if you are part of both groups

I have the following scenario:

An agent is in Group 1 and Group 2. A ticket comes in and is assigned to Group 1. The agent takes the ticket so it is now assigned to them in Group 1. They have worked the ticket as part of the Group 1 function and now need to move on to the next ticket. But, they must assign the ticket to Group 2 for a colleague to pick up and continue work. 

Upon assigning the ticket to Group 2 it automatically assigns the ticket to the agent in Group 2. If they then update the ticket a second time, in the same way and assign it to Group 2. It successfully assigns to Group 2. 

How do i avoid having to assign the ticket to Group 2 twice in order for it to be in the Group? Rather than in the Group and against the agent moving it.

This only happens if you are in both groups and are currently assigned the ticket. 

Any help much appreciated!