Dealing with monitoring alerts and notifications in Zendesk

@Olli - we deal with similar notifications from dozens of sources with their own systems' unique tracking IDs appended/prepended to subjects.  We also suffer from similar barrages of updates that cause duplicate tickets & de-threading as subject lines are modified, such as:

1. [Upcoming]: Planned maintenance ID#999
2. [In Progress]: Planned maintenance ID#999
3. [Resolved]: Planned maintenance ID#999
4. [Reopened]: Planned maintenance ID#999
5. [Resolved - Final]: Planned maintenance ID#999

To date, Zendesk native email processing doesn't have any function to handle this sort of process as intuitively as we'd both like (i.e. identifying the unique ID string from an external/partner system and auto-merging any emails based on it).  This causes a fair amount of de-threading (when our system or an external one's unique ID isn't included in a response), duplicate ticketing about the same request (when subject changes in emails), and lost messages (when automated systems send a lot of updates in a short time period, such as a system that pushes out emails once a day at a specified time).

The recommendations I've received from a Zendesk solution architect include:

Middleware/3rd party data reference

The cleanest solution forward would likely be leveraging a middleware + SMTP server to 1) keep things "as is" when it comes to your partners emailing in to an address, but 2) also maintaining a dependable table of ID relationships between the Zendesk IDs and the correct IDs coming in from other systems. 

At a high level, the idea would be to have this middleware (using a 3rd party API) store the Zendesk ticket ID along with the partner ID as a relationship. The SMTP service would then keep things as is on the front end for partners, but on the backend the emails would be hitting this middleware (via the SMTP server) to reference the proper ID(s), subsequently threading back to the correct Zendesk ticket and/or partner ticket. 

• As can be expected with implementing a technology like this, this would more than likely be a fair amount of work up front; however, this would likely provide the most flexible, long-term solution for this process that's currently in place. 
• Another plus of this option is that, so long as the IDs are maintained in this relationship table, you don't have to depend on a mutable Subject line and/or manually updating ticket fields. For example, a potential flow is that using this middleware, email headers could be used instead of a standard Subject line. 
• The emails from tickets would be coming from/going to a particular email address (the SMTP server/API) which would reference the table in question, then thread back to the correct ID on each respective end with the needed information

This flow also seems like it could work with regard to the third issue you sent over of "Partner's automated system emails us about an incident" and being able to thread notifications into these batch notification threads. 

As a side note, we also have partners that provide services such as this, such as Azuqua.

***The below functionality would depend heavily upon having the custom Carrier/Partner # field (or any needed custom field value) populated (either automatically or manually) in order to maintain this workflow
***I understand agents are not presented with these IDs all the time. The main point of the following options is that they are a bit more short-term/easily implemented. 

App Framework

Using our Apps Framework, you can update a custom field, then the ticket's interface (i.e. subject line) whenever the ticket is opened by an agent. The app would simply pull whatever information that is in the Subject line, run a REGEX match to validate the data, place this ID in a variable, then place this data (i.e. ID/number) in a custom field. This way you will at least have the needed ID(s) within the ticket.

Agents would of course need to ensure that the app is running and populating the correct data before setting updates on said ticket. 

Then, when tickets are subsequently opened by an agent (and assuming the Subject line was changed to remove the ID), the app could subsequently:

1. run a GET on this custom field value and store in a variable: https://developer.zendesk.com/apps/docs/support-api/ticket_sidebar#ticket.customfieldfieldname
2. using a function like .prepend() or concat(), add this ID back to the Subject line
3. agent then updates the ticket as needed which saves the ID back to the Subject line

Trigger+Target

Again assuming this custom field value exists for the needed ID, there is another option to automate the "correct the Subject line" action using a Trigger and Target. **Please note that this functionality isn't fully supported by Zendesk due to potential race condition issues.** 

In short, this flow will run a trigger that updates the same ticket that's already being updated, hence the potential for a race condition to occur. That said, I've used this solution in many other scenarios and for the most part it has been pretty successful.

*The trigger would look something like the below configuration:

Meet ALL conditions 
  Ticket status is less than Solved
  Ticket form is XYZ
Meet ANY conditions
  Ticket is created
  Ticket is updated
Actions
  Notify target: Update ticket with carrier and ticket ID

  BODY: {"ticket": {"subject": "{{ticket.ticket_field_###########}} + {{ticket.title}}"}}

*These conditions can be whatever your business rules dictate. The main point is that you're updating the ticket in question with a custom field value(s) while maintaining the rest of the original title. 

Then, the "Update ticket with carrier and ticket ID" target itself would look like the following:

Title: Update ticket with carrier and ticket ID
Url: https://SUBDOMAIN.zendesk.com/api/v2/tickets/{{ticket.id}}.json
Method: PUT
Content type: JSON
Basic authentication: enabled
Username: adminemail@example.com
Password: [applicable password]

You could also use API token authentication, in which case the username/password would look like:

Username: adminemail@example.com/token
Password: [your API token here]

In the end, when the trigger conditions are met, you would dictate that you want to prepend your custom field value(s) in the Subject line. 

Thanks for posting this, Matt! 

Did the solution architect share this with you in a ticket? It's great info.

Hi Jen!

Yes, this was a response to a solved ticket and related to an ongoing ticket/feature request I've had open for a considerable time now.  Using Zendesk for B2B relationships, particularly with larger businesses who utilize automated emails and/or run their own ticketing systems, leads to a variety of challenges such as this one. 

Related ones include: consistent bulk-managing of suspended tickets, whitelisting/validating/successfully receiving inbound emails from automated systems (especially if they send notifications in batch updates, such as processing payments at a certain daily time), lack of visibility if a spam threshold is reached & mail is outright rejected... there are lots of challenges since Zendesk was designed with direct human-to-human communications in mind.  :)

Thanks Matt! We are looking into turning this info into an article so others users can find it.

Thanks again for sharing it.