The OverOps Automated Root Cause provides a powerful mechanism to get to the root of errors and exceptions in production and staging environments. The page is divided into three panes, that provide information about the event to create a complete picture.
Automated Root Cause
The Analytics pane provides important details relating to the impact of this event in your application. It is divided into two areas: the Analytics pane and chart.
The Analytics pane provides the type of event, when it began and how many times its and out of how many calls. The Analytics chart shows you the volume of the event over the course of the selected timeframe (e.g. last hour,day, week,..). This enables you to filter the chart to display the volume of the event in specific applications or servers.
Click the JVM or Server labels to directly to see the volume of the event on specifically on that machine or application. Hover over the occurrences label to see the number of times this event has occurred and out of how many calls into the method containing it.
Click Open in Dashboard at the top-right of the screen to open this event in the main Dashboard. Click Go to snapshot on any point in the graph to jump and see the code and variable state at that moment in time.
The Analytics Pane (left) and the Chart (right) displaying the volume of the event
OverOps captures data when events, application errors (exceptions) and log (warnings and errors) occur according to a defined algorithm.
The OverOps snapshot contains valuable information about events in the monitored application, including:
- The date and time the snapshot was taken
- The server and application on which the event occurred
- The deployment on which the event was recorded
- The full Call Stack
- The Source Code view.
From version 4.13.1, when there are multiple web applications running on the same JVM, the snapshot includes the web application name in which the event was caught:
Although the information in these snapshots may be crucial to fix the problem causing the event, the number and frequency of these snapshots can impact performance. There are ways to reduce this impact.
The Collector's throttling mechanism enables mitigating the number of snapshots to reduce overall storage and optimize JVM performance. To enable throttling, see: Collector/Storage Optimization.
In certain cases it may be required to take a snapshot in addition to the snapshot algorithm.
To order a snapshot for the next time an event occurs:
- From the Dashboard Event List, click the and select Force Snapshot.
The snapshot is recorded the next time the event is seen in the application.
- From the ARC screen, in the left pane, click .
The time of the last snapshot is displayed and you can scroll between previous snapshots with the back arrow.
In order to understand and resolve errors, it is important to be able to trace their path through the code. In the Automated Root Cause screen, OverOps reveals the full call stack - from entry point to the method in which the event occurred - even across multiple machines.
The call stack for an event is displayed on the left hand side of the ARC screen. OverOps tracks the code and variables state for the event all the way back to its entry point into the code, where the parameters were passed into it. If the event involves calls across multiple machines, OverOps displays a unified call stack. Click on a method in the call stack to open it on the right hand side.
The Call Stack pane displays the chain of methods within the JVM leading up to the event. The first method in line is the last method on a non 3rd party code within your application. The icon in the method indicates that the variable state has been detected by the JVM Agent.
When an exception is caught and re-thrown once or more within the thread, Related Events drop-down displays the root cause (available only when such exceptions exist).
At the bottom of the stack the machine name and the JVM thread name in which this event occurred are displayed. By default, 3rd party code is hidden. To display them, from the bottom of the stack, turn Show 3rd party methods on. To copy the full stack to the clipboard, click COPY STACK.
Unified Call Stack
Call Stack pane
By default, the Source Code View shows a decompiled Java version of the bytecode executing within the JVM at the moment of event. Hover over any highlighted variable to display its value and jump to see its full contents within the variable grid. The row in which the event occurred is highlighted, as depicted below.
The full event message appears in the title of the code pane.
When configured, OverOps uses the original source code instead of decompiling it from the JVM.
Search the source code or the variable grid for any variable name or value using Search Variables. Click here to learn more about variable search.
Source code and Variable state pane
Recorded Variables displays the variable values and objects accessible from the method. Objects can be explored up to ten levels deep into the code. Click the button that appears when hovering over the object, to view its contents as a JSOB. The content can be copied to the clipboard.
The Recorded Variables pane contains all local variables and parameters (including "this" in non-static methods). The first method also contains thread-local variables defined for this thread as well as SLF4J and Log4J Mapped Diagnostics Context (MDC) values. These MDC objects are often too large, for the full set of data to be available in the log, the micro-agent, however, is able to capture and record the entire object.
In some use cases, such as asynchronous message passing, these MDC objects contain a key-value map of the recorded requests, initial servlet information, and much more. They can be seen in any OverOps snapshot, and provide better visibility to the source of the bad request. This provides helpful extended visibility feature since back tracing the source of a bad request in an asynchronous environment is a known challenge.
The choice of the collected variables most relevant within an allocated timeframe is determined by the Agent using an adaptive machine learning algorithm. The selection process involves which and how many variable to collect, the number of items to collect, the length of string to capture, etc.
Click here to learn more about object and variable state.
Variable grid displaying the variables state JSON representation of an object
The Log View displays the last 250 log statements leading up to the event. Since the log statements are collected directly from JVM memory, any DEBUG, TRACE or INFO statements are visible regardless of whether or not they were logged to file.
Click the button to switch between code and log view.
Click here to learn more about the Log View.
Log View pane
For each event and exception detected, OverOps displays a JVM view that displays the internal JVM state at the moment of the event, including memory usage (heap and non-heap), basic system information, CPU usage and more. This enables working with the OverOps code (“classic”), log and JVM data without leaving the application. Click here to learn more about the JVM View.
The Actions toolbar provides a set of capabilities to share, mark and search through the ARC screen:
- Send to Jira - Create a new Jira issue for an event linking it directly to the source, stack, state and statistics. Click here to learn more about Jira integration.
- Delete- Remove the event from the Dashboard Events List and Chart. The Agents no longer capture snapshots for this event. The event will appear under the "Archive" label in the Dashboard from which it can be resolved. Click here to learn more about deleting events.
- Resolve - Mark an event as fixed and remove it from the Dashboard Events List and Chart. Should this event occur after a new code deployment, it will return to the Event List and Chart marked 'Resurfaced', and you are notified by email. Click here to learn more about resolving errors.
- Label - Label events to classify them according to priority ('Critical' or 'Low'), responsibility ('John' or 'QA') and version ('V1 RC2'). Click here to learn more about creating and assigning labels.
- Add/Edit Note - Attach a note to an event and share it with your team members about which they are notified by email. Click here to learn more about sharing with team members.
Add Timer - Track predefined methods for latency. OverOps sends an alert when the code runs longer than expected. Click here to learn more about Timers.
Events are often connected to other events occurring in the code. OverOps automated root cause points to related events.
If an event has caused re-throws or has other events related to it, a Related Events dropdown menu appears in its Automated Root Cause screen. Click to list all the related events and rethrows, as well as their type (caught or uncaught exception, HTTP error, or log error). The currently displayed event is indicated in bold.
Related Events and re-throws in Automated Root Cause