Agent, Collector, Secret Keys relation

How agents, collectors, secret_keys interoperate in a deployment and how to choose how many instances of these entities to use?

Comments 1 comment
  • The data collected and analyzed in OverOps system use a data pipeline that has these 3 main actors that handle data in this order:

    • Agent: this is a "native agent" that runs inside the JVM. Every agent collect raw data from within the JVM and send it to a collector. The collector the agent connects to can be unique or picked from a pool of collectors eventually behind a Load Balancer. Agents has a "store & retry" buffer to store data if a collector is not available to receive it or if there is a communication problem
    • Collector: this is a native process (not a JAR neither a program running in a JVM). Each collector receives raw data from one or more agents, process it and then send it over to the backend allocating it to a SINGLE Secret Key. Collectors are stateless for their data exchange with the agents but they can only work for a SINGLE Secret Key during a single install. A pool of collectors can be used to increase the throughput and or achieve HA but all members of a pool can ONLY be configured to point ALL to a SINGLE Secret Key. Collectors cache data for a Secret Key so if a collector need to be reassigned to another key, it needs to be uninstalled and reinstalled using the new Secret Key. Collectors have a "store & retry" buffer that will manage data that need to be sent from the collector the the backed
    • Secret Key: this is made from a few fields separated by the character "#". The "Secret Key ID" is the first part of the Secret Key and it's in the form of SXXXXX where X are digits (i.e S12345 ). Every time we look at the default OverOps Dashboard UI, we see data from a single Secret Key ID. In the case a user has been granted access to a set of Secret Keys, (s)he can swap among them using a dropdown in the UI. Every collector in the system points and feed data to a SINGLE Secret Key during its installation lifespan.

    An OverOps account can use multiple Secret Keys to segregate data from different systems, reassign/repurpose/decommission monitored systems etc.

Please sign in to leave a comment.

Join conversation