DevOps Integrations

Contents

Introduction

Setup StatsD

--Linux

--OSX

--Windows

--Configuring Services

Alerting

 

Introduction

OverOps' agents natively support sending metrics via StatsD into graphing and alerting tools. StatsD is a protocol and open-source implementation for sending metrics between modern DevOps tools.  This enables you to visualize any combination of events within OverOps, and be alerted once they exceed a target threshold using your graphing and alerting tools of choice. 

 

To integrate with any of these tools, follow these 3 steps:

  1. Setup your StatsD node.

  2. Configure which services to integrate with.

  3. Configure which metrics to publish from OverOps.

 

Setup StatsD

StatsD is a simple JavaScript application running inside of Node.js. You can install a StatsD node on each OverOps monitored machine, or on a central machine with whom monitored nodes will communicate. The latter enables you to manage all your StatsD configurations through a centralized node, but is not mandatory. If your organization already has a StatsD server set up you can connect directly to it.

Linux

OverOps (v 3.16.1 and up) bundles all the StatsD Javascript extensions described above with the agent.

To use the OverOps provided StatsD daemon run:

1. Setup Node.js and StatsD by running: sudo /opt/takipi/statsd/etc/takipi-statsd-install

2. Configure the the services with which you want to integrate as described below.

3. Start StatsD by running – sudo /opt/takipi/statsd/etc/takipi-statsd-start

OSX

1. If you don’t have Node.js, install it using the .pkg installer.

2. Download StatsD, or clone from GitHub.

3. Download the Takipi config.json StatsD configuration file.

4. Configure the services to which to send data as described below.

5. Launch StatsD by running node <path-to-stats.js> <path-to-config.json>  (stats.js is located in your StatsD folder).

Windows

1. If you don’t have Node.js, install it using the .msi installer.

2. Download StatsD, or clone from GitHub.

3. Download the Takipi config.json StatsD configuration file.

4. Configure the services to which to send data as described below.

5. Launch StatsD by running node <path-to-stats.js> <path-to-config.json> (stats.js script is located in yourStatsD folder).

 

Configuring Services

To send metrics into any graphing or alerting backend, StatsD uses a concept of backends. An backend is a .js file which communicates with the target service (e.g Graphite, Nagios,..). The information required to connect with the remote service (i.e. address, API key, password..) is placed in the config.json file that is passed as a parameter into StatsD.

The config.json has pre-populated configuration samples for all the services listed below. Simply uncomment the sections relevant to the service(s) which you want to use, and fill in the authentication information. StatsD configuration data is private to your machine, and is not transmitted out of it.

If you’re using the prepackaged StatsD node (Linux only), all the relevant backends files have already been provided for you. If you’re using your own StatsD node, make sure you’ve installed the appropriate StatsD backend for any of the following services (click any tool to jump to its instructions):

Anodot, AppDynamics, CloudWatch, DataDog, Grafana, Graphite, HipChat, InfluxDB, Librato, Nagios, New Relic,OpenTSDB, PagerDuty, Sentry, Slack, Zabbix.

 

Anodot

Anodot is a service which captures data to detect anomalies and group them into one event which tells a single story.

Configuring the extension:

{
  backends: ["statsd-anodot-backend"] //identify this as an Anodot backend
  anodotApiToken: "API_KEY",   // The API key 
  anodotApiHost: “<api_host”>, // The API host
}

Learn more about additional Anodot options here.

Installing the backend:
If you’re using the OverOps supplied StatsD node, the StatsD backend has already been installed. Otherwise run:  npm install statsd-anodot-backend

 

AppDynamics

AppDynamics gives an integrated view of real-time application performance, user experiences, and infrastructure capacity.

Note: Make sure Machine agent API is on. For more information.

 

Configuring the extension:

{
  backends: ["statsd-appdynamics-backend"] //identify as AppDynamics backend
  appdynamics: 
  { 
    //URL to the Machine Agent listener
    base_url: 'http://localhost:8081/machineagent/metrics', 
    hide_statsd: false // enable to suppress the statsd.bad_lines_seen 
                       // and statsd.packets_received metrics
  },
}

Learn more about additional App Dynamics options here.

 

Installing the extension:

If you’re using the OverOps supplied StatsD node, the StatsD backend has already been installed. Otherwise run:npm install statsd-appdynamics-backend

 

AWS Cloudwatch

Amazon CloudWatch is a monitoring service for AWS cloud resources and the applications you run on AWS.
Configuring the extension:

{
  backends: [ "aws-cloudwatch-statsd-backend" ], // AWS CloudWatch backend
  cloudwatch: {
    region: "YOUR_REGION",              //Example: EU_WEST_1, US_EAST_1
    accessKeyId: 'YOUR_ACCESS_KEY_ID',  //See here: http://amzn.to/1sT9aw0
    secretAccessKey: 'YOUR_SECRET_ACCESS_KEY'  
  }
}

Learn more about additional AWS CloudWatch options here.

Installing the extension:

If you’re using the OverOps supplied StatsD node, the StatsD backend has already been installed. Otherwise run:npm install aws-cloudwatch-statsd-backend

 

Datadog

Datadog is a service which aggregates metrics and events across the full devops stack.

Configuring the extension:

{
  backends: ["statsd-datadog-backend"] // identify this as a Datadog backend
  datadogApiKey: "your_api_key", // Your API key. See here - 
                                 //app.datadoghq.com/account/settings#api
 
}

Learn more about additional Datadog options here.

Installing the extension:

If you’re using the OverOps supplied StatsD node, the StatsD backend has already been installed. Otherwise run:  npm install statsd-datadog-backend

 

Grafana

Grafana is an open source, feature rich metrics dashboard and graph editor for Graphite, InfluxDB & OpenTSDB.

OverOps supports sending data via StatsD using the InfluxDB, Graphite and OpenTSDB extensions.

 

Graphite

Graphite is a highly scalable real-time graphing system. Graphite processes, stores and visualizes numeric time-series data through a web interface.

Configuring the extension:

{
  backends: [ "./backends/graphite" ]   // identify this backend as Graphite
  graphitePort: 2003,                   // port of Graphite server 
  graphiteHost: "graphite.example.com", // hostname or IP of Graphite server
  deleteCounters: true,
  graphite: {  // Graphite tweaks for Takipi
    prefixCounter: "",
    prefixGauge: "",
    globalPrefix: "",
    legacyNamespace: false
  }
}

Learn more about additional Graphite options here.

Graphite is StatsD’s default backend, and as such no .js extension needs to be installed.

 

Hosted Graphite

Hosted Graphite is a monitoring service which visualizes data produced by your apps using Grafana dashboards.

Configuring the extension:

{
  backends: ['./backends/graphite'] // identify this backend as Hosted Graphite
  graphiteHost: "carbon.hostedgraphite.com",
  deleteCounters: true,
  graphite: {
    prefixCounter: "",
    prefixGauge: "",
    globalPrefix: "",
    legacyNamespace: false,
    globalPrefix: "your-api-key-here" // Your hosted graphite 
  }
}

Learn more about additional Hosted Graphite options here.

Installing the extension:

Hosted Graphite uses the original StatsD Graphite backend already included in all StatsD installations.

 

HipChat

HipChat lets you bring the office with you wherever you go.

To setup HipChat use the alerting integration.

 

InfluxDB

InfluxDB is designed to track data from tens of thousands of sensors all sampling at rates of once a second or more.

Configuring the extension:

{
  backends: ["statsd-influxdb-backend"] // identify this as an InfluxDB backend
  influxdb: {
    host: '127.0.0.1',   // InfluxDB host
    port: 8086,          // InfluxDB port
    version: 0.9,        // InfluxDB version
    database: 'dbname',  // InfluxDB database instance (required)
    flush: {
      enable: true       // Enable regular flush strategy
    },
  },
}

Learn more about additional InfluxDB options here.

Installing the extension:
If you’re using the OverOps supplied StatsD node, theStatsD backend has already been installed. Otherwise run:  npm install statsd-influxdb-backend

 

Librato

Librato is a real-time operations analytics service that accepts metrics from any source for aggregation and transformation, anomaly detection, alerting, visual analysis and storage.

Configuring the extension:

{
  backends: ["statsd-librato-backend"] // identify this as a Librato backend
  librato: {
    email:  "myemail@example.com", // the Librato account email
    token:  "API_KEY_TOKEN",       // api key from the Librato account page.
    source: "takipi"               // a logical identifier for this metrics group
  },
}

Learn more about additional Librato options here.

Installing the extension:

If you’re using the OverOps supplied StatsD node, theStatsD backend has already been installed. Otherwise run:npm install statsd-librato-backend

 

New Relic

A software analytics tool suite used by developers, ops, and software companies to understand how your applications are performing in development and production.

Configuring the extension:

{
  backends: ['statsd-newrelic-backend'],    //identify as NewRelic backend
  newRelicLicense: 'YOUR_LICENSE_KEY_HERE', //API Key - http://bit.ly/1MJdEBR
  newRelicApp: 'YOUR_APP_NAME_HERE'
}

Learn more about additional New Relic options here.

Installing the extension:

If you’re using the OverOps supplied StatsD node, theStatsD backend has already been installed. Otherwise run:npm install statsd-newrelic-backend

 

PagerDuty

PagerDuty provides alerting, on-call scheduling, escalation policies and incident tracking to increase uptime of your apps, servers, websites and databases.

To setup PagerDuty use the alerting integration.

 

Slack

Slack is real-time messaging, archiving and search for modern teams.

To setup Slack use the alerting integration.

 

Sentry

Sentry is an exception tracing service that gives you insight into the errors that affect your customers.

To setup Sentry use the alerting integration.

 

Zabbix

Zabbix is an enterprise-level software designed for real-time monitoring of millions of metrics collected from tens of thousands of servers, virtual machines and network devices.

Configuring the extension:

{
  backends: ["statsd-zabbix-backend"]     // Identify this as a Zabbix backend
  zabbixPort: 10051,                      // Zabbix port
  zabbixHost: "localhost",                // Zabbix host
  zabbixSender: "/usr/bin/zabbix_sender", // Zabbix sender
}

Learn more about additional Zabbix options here.

Installing the extension:

If you’re using the OverOps supplied StatsD node, theStatsD backend has already been installed. Otherwise run:npm install statsd-zabbix-backend

 

Alerting

This extension is used to fire an alert once a specific threshold is met into an alerting tool such as PagerDuty, Slack, HipChat or Sentry.

This extension defines a set of dispatchers which represent the alerting services (e.g PagerDuty), and metrics and events which define the conditions under which alerts are sent.

Events are fired once a metric whose name matches a target pattern is received. An example for that would be uncaught exceptions, or a specific error for which you want to be notified on every time it occurs.

Metrics define thresholds over or under which an alert is fired. An example of that would be over 1000 HTTP errors or IO exceptions during a reporting interval.

Configuring the extension:

{
  backends: ["statsd-alerting-backend"], //identify this as an alerting backend
  alerts: {
    dispatchers: {
      //
      //this defines a an integration into Slack
      //
      slackDispatcher: { 
        type: "slack",
        config: {
          host: "https://hooks.slack.com",
          token: "",                  //your Slack API key. 
                                      //See more here - http://bit.ly/1hwuDdF
          username: "statsd-alerts",  //the user is posting the alert
          channel: "#alerts"          //the Slack channel to post to
        }
      },
      //
      //this defines a an integration into PagerDuty
      //
      pagerdutyDispatcher: { 
        type: "pagerduty",
        config: {
          key: "" // PagerDuty API key. 
                  //See more here - http://bit.ly/1UBKu8Q
        }
      },
      //
      //this defines a an integration into Sentry
      //
      sentryDispatcher: { 
        type: "sentry",
        config: {
          dsn: ""
        }
      },
      //
      // this defines an integration with Hipchat
      //
      hipchatDispatcher: {
        type: "hipchat",
        config: {
          key: "", //Your API Key - www.hipchat.com/account/api
          room: "",
          color: "red"
        }
      }
    },
    events: [{
      name: "some.event.*", //any metric sent matching this  
                            //pattern will be fired as an alert
      dispatcher: "pagerdutyDispatcher" //the service to report to
    }],
    metrics: [{
      name: "some.event.counter", //any metric sent matching 
                                  //this pattern and whose value
                                  //matches the operator + value below 
                                  //will be sent to the target dispatcher 
      type: "counter_rates",
      gte: 0.2, //operator: gte, gt, lte, lt, eq, delta, delta_gte, 
                //delta_gt, delta_lte, delta_lt, delta_eq. 
                //delta = current sample - previous sample
      alert: "pagerdutyDispatcher" //the service (e.g, PagerDuty) 
                                   //to which this alert is sent.
    }]
  }
}

Installing the extension:

If you’re using the OverOps supplied StatsD node, the StatsD backend has already been installed. Otherwise run:  npm install statsd-alerting-backend

 

Nagios

[Coming Soon]

 

OpenTSDB

[Coming Soon]

Have more questions? Submit a request