Troubleshooting

Introduction | Video

Tip 01 | Tip 02 | Tip 03 | Tip 04 | Tip 05 | Tip 06 | Tip 07 | Tip 08 | Tip 09 | Tip 10 | Tip 11

Introduction

Troubleshooting with Discovery Admin is easy and can be initiated with a few clicks!

1. To view previous Troubleshooting Records, click the { Discovery Admin } UI Action (visible on the Discovery > Status List View)

2. To initiate a new Troubleshooting, select one or more rows from the (Discovery Status) list below and click the { Discovery Admin } UI Action (visible on the Discovery > Status List View)

Video  (1 min :: 21 secs)

Time Stamps:

0:16 - Navigating to the Previous Troubleshooting

• To navigate to the previous Troubleshooting(s), start by navigating to the Discovery Status List View.

• From there, select the { Discovery Admin } UI Action, to navigate to the Troubleshooting List View.

0:34 - Initiating a new Troubleshooting

• To analyze ServiceNow Discovery Logs, navigate to the Discovery Status List View and select the relevant Records.

• Next, click on the { Discovery Admin } UI Action to initiate the analysis of the selected Records and navigate to the Troubleshooting Form.

Tip 01: Troubleshooting Duration

Discovery Admin can scan about 1 Million Logs a Day. This limitation is governed by ServiceNow limiting our ability to run queries and insert records in the background.

As a part of the analysis, for every IP Address, Discovery Admin does a lookup on several Tables including the CMDB and the Credential Affinity Table to aggregate and contextualize all the necessary data points.

This enables accurate analysis to determine the root cause and insights into who the remediation should be assigned to for resolution.

These lookups depend on the size of your CMDB and add time to the end-to-end processing of the Troubleshooting.

While 1 Million Logs per day is a large number for most customers, some customers generate more than 1 Million Logs a day, as a result of several MID Servers Clustered together to run ServiceNow Discovery.

In these scenarios, it's normal for Discovery Admin to take more than a day to completely analyze the ServiceNow Discovery Logs.

This is an acceptable compromise as the alternative to manually troubleshoot Millions of Logs accurately and reliably, would take several weeks (at best) requiring highly experienced ITOM Resources.

New in Discovery Admin v9.3:

Realtime duration is displayed on the Troubleshooting Form while Troubleshooting is in Progress.

The duration counter is temporarily stopped while Troubleshooting is Paused and continues (with the inclusion of the stopped time during Pause) after the Troubleshooting is Resumed.

The duration counter stops when the Troubleshooting is Canceled.

Additional attributes on the Troubleshooting Form that capture timestamps are:

• Discovery Started

• Discovery Ended

• Created

• Initiated (Analysis)

• Completed (Analysis)

• Updated

Audit History is also captured for Attributes: Status and Incident Information

This provides detailed insights into the history of the Troubleshooting from Initialization to Incident Generation.

New in Discovery Admin v9.5:

The information captured in Audit History is also displayed at the bottom of the Troubleshooting Form.

Tip 02: Discovery Admin SEEMS stuck on: Progress = Initializing...

This Status implies that Discovery Admin is waiting for another Troubleshooting to complete execution or for another Incident Generation to complete. You can verify this by navigating to the Troubleshooting List View.

This is by design to ensure Discovery Admin uses a limited and predictable number of background threads eliminating any impact on other background processing that may be scheduled on your ServiceNow Instance.

Tip 03: Discovery Admin IS still stuck on: Progress = Initializing...

This Status implies that the installation of Discovery Admin has gotten corrupted as a result of a missing Template or due to Cross Scope Issues, resulting in ServiceNow not allowing Discovery Admin to execute.

Please reach out to your Discovery Admin point-of-contact for resolution.

Tip 04: Troubleshooting Status = Canceled (v9.3)

Here are the reasons why the Troubleshooting Status = Canceled:

• Tip 05: The Troubleshooting was manually canceled via the { Cancel } UI Action

• Tip 06: The Troubleshooting was automatically canceled due to restricted access to the Tables required by Discovery Admin. The access restriction can be via ACLs or Application Cross Scope Access.

• Tip 07: The Troubleshooting was automatically canceled due to unresolved conflicts / skipped records after a Discovery Admin upgrade.

• Tip 08: The Troubleshooting was automatically canceled due to orphan records in the Upgrade History Log Table.

Tip 05: Canceling an Active Troubleshooting (v9.3)

Active Troubleshooting Analysis or Incident Generation can be canceled manually using the '{ Cancel }' UI Action on Troubleshooting Form.

The following information is displayed/updated on the Troubleshooting Form:

• Status is set to: Canceled 

• If canceled during Analyzing Logs, Incident Information is set to: Incident Generation is not available as this Troubleshooting is canceled

• If canceled during Incident Generation, Incident Information is set to: Incident Generation is canceled | Generated: ## of ## Incident(s)

• Note: No Error Message is displayed on the top of the Troubleshooting Form

Tip 06: Troubleshooting Canceled due to Restricted Access (v9.3)

The Troubleshooting gets canceled immediately after the Troubleshooting is Initiated, due to restricted access to the Tables in the Global Scope, required by Discovery Admin.

The access restriction is due to ACLs or Application Cross Scope Access.

For ACLs:

• Error Message set to:

  HINT: Troubleshooting Canceled due to Initialization Failed. Please reach out to your Discovery Admin point-of-contact.

• Status is set to: Canceled

• Progress is set to: Canceled > Initialization Failed

• Incident Information is set to: Incident Generation is not available as this Troubleshooting is canceled

For Application Cross Scope Access:

• Error Message set to:

  HINT: Troubleshooting Canceled. Discovery Admin Application Cross-Scope Access needed for Tables [table_name]. Please work with the ServiceNow Platform Admins to configure the Application Cross-Scope Access CORRESPONDING to Discovery Admin Property 1.3

• Status is set to: Canceled

• Progress is set to: Canceled > Application Cross-Scope

• Incident Information is set to: Incident Generation is not available as this Troubleshooting is canceled

Tip 07: Troubleshooting Canceled due to Skipped Records after Upgrade (v9.3)

The Troubleshooting gets canceled immediately after the Troubleshooting is Initiated, due to unresolved Skipped Records after the Discovery Admin Upgrade.

The following information displayed on the Troubleshooting Form:  

• Error Message set to:

  HINT: Troubleshooting Canceled due to Skipped Changes Not Reviewed. Please reach out to your Discovery Admin point-of-contact

• Status as set to: Canceled

• Progress is set to: Canceled > Skipped Changes Not Reviewed

• Incident Information is set to: Incident Generation is not available as this Troubleshooting is canceled

To Resolve the above Issue, please follow the Post Upgrade Steps.

Tip 08: Troubleshooting Canceled due to orphan records in Upgrade History Log (v9.3)

The Troubleshooting gets canceled immediately after the Troubleshooting is initiated, due to orphan records in the Upgrade History Log Table.

The following information displayed on the Troubleshooting Form:

• Error Message set to:

  HINT: Troubleshooting Canceled due to Skipped Changes Not Reviewed. Please reach out to your Discovery Admin point-of-contact

• Status as set to: Canceled

• Progress is set to: Canceled > Skipped Changes Not Reviewed

• Incident Information is set to: Incident Generation is not available as this Troubleshooting is canceled

Steps to Resolve the above issue: 

• First, make sure, this is NOT becuase of Skipped Changes Not Reviewed after a Discovery Admin Upgrade. If it is, please follow the steps in the previous Tip (above)

• Note: You will need 'admin' access to perform the steps below (Work with your ServiceNow Platform Team, as needed)

• If you still see this error after resolving Skipped Changes, navigate to:

  https://YOUR_INSTANCE.service-now.com/sys_upgrade_history_log_list.do?sysparm_query=resolution_status%3Dnot_reviewed%5Eupgrade_history.sys_idISEMPTY&sysparm_first_row=1&sysparm_view=

• Add column 'Created' to the List View

• Sort the List View in decending order (Created) to identify the creation date of the most recent record see the historical creation of the records.

• Typically these orphan records were created years ago. Additioanlly, they are of no significance as they are not associated to a corresponding 'Upgrade History' as the reference field is EMPTY.

• Update the 'Resolution' of all these records to: Reviewed

  
  

• Once the above update is done, navigate to:

  https://YOUR_INSTANCE.service-now.com/sys_upgrade_history_log_list.do?sysparm_query=resolution_status%3Dnot_reviewed%5Eupgrade_history.sys_idISEMPTY&sysparm_first_row=1&sysparm_view=

• If the above steps are executed successfully, there should now be no records corresponding to the link above.

• Discovery Admin will run successfully once the steps above are executed. 

EXPLANATION: Why are you seeing this error?

Before we run Discovery Admin, we check for the following filter to be empty to make sure that there aren’t any skipped records: 

https://YOUR_INSTANCE.service-now.com/sys_upgrade_history_log_list.do?sysparm_query=resolution_status%3Dnot_reviewed%5Eupgrade_history.sys_idISEMPTY&sysparm_first_row=1&sysparm_view=

However, in some cases, Customer Instances have orphan records from (often from many years ago), that were not reviewed. These orphan records (i.e. they are not associated with any App as the sys_id is empty) conflict with our filter above, resulting in Discovery Admin incorrectly thinking that there are skipped records.

Marking them as Reviewed (as referenced in the steps above) removes these records from our filter, enabling Discovery Admin to run, with no other impact to the environment.

Tip 09: Troubleshooting Summary (v9.3)

The Summary on the Troubleshooting Form has been updated to display the Discovery Status records in ascending numeric order.

The Discovery Status records are also analyzed in the same numeric order to ensure older Discovery Status records are analyzed before newer ones.

Tip 10: Deleting a Troubleshooting Record (v9.3)

Troubleshooting records should be Canceled or Paused before being deleted.

If an active Troubleshooting is deleted, the analysis is automatically canceled.

Deleting a Troubleshooting record deletes all the corresponding Log Analysis records and their corresponding Audit History.

Tip 11: Info Message 'Log Analysis count exceeds Discovery Log count' on Troubleshooting Form (v9.5)

This Info Message is displayed on the Troubleshooting Form for one of the following reasons:

• One or more Discovery Status Records were still ‘Active’ when Discovery Admin was troubleshooting the corresponding Discovery Status Records.

  
  

  NOTE: the Summary Field on the Troubleshooting Form provides insights into which Discovery Status Record was 'Active'.

  
  

  RESOLUTION: If this message is seen due to this reason, visit the Scheduled Troubleshooting Design to ensure that the corresponding Discovery Status Records complete before the Troubleshooting is initiated.

• The Troubleshooting was restarted due to maintenance activity on the ServiceNow Platform.

• ServiceNow has auto-purged Logs corresponding to the selected Discovery Status Records, while the Troubleshooing is taking place.