Prometheus: Difference between revisions

From Bondix Wiki
(Initial page)
 
(Add more infos regarding status)
 
(4 intermediate revisions by the same user not shown)
Line 1: Line 1:
Prometheus Support must be enabled explicitly by editing /etc/bondixserver.json (/etc/saneserver.json). You can either use the default https port to also serve Prometheus metrics, or use a dedicated listener.


Once enabled, prometheus metrics will be accessible via https://<server-ip>/metrics
= Enabling Prometheus Support =
Prometheus support must be explicitly enabled by modifying the configuration file located at <code>/etc/bondixserver.json</code> or <code>/etc/saneserver.json</code>. You can either:


=== Serving Prometheus over default https port ===
* Serve Prometheus metrics over the default HTTPS port, or
In /etc/bondixserver.json, locate the following line:
* Configure a dedicated listener for Prometheus metrics.
  {"target": "server", "action": "add-https", "host": "0.0.0.0", "port": "443", "allowMonitor": true},
 
Here, we'll add the following properties:
Once enabled, Prometheus metrics will be accessible at:
<code>https://<server-ip>/metrics</code>
 
== Serving Prometheus Over the Default HTTPS Port ==
To enable Prometheus on the default HTTPS port, update your <code>bondixserver.json</code> configuration file.
 
=== Modify the Configuration ===
Locate the following line:
  <code>{"target": "server", "action": "add-https", "host": "0.0.0.0", "port": "443", "allowMonitor": true},</code>
Add the following properties:
{| class="wikitable"
{| class="wikitable"
|+
!Property
!Property
!Value
!Value
!Description
!Description
|-
|-
|allowPrometheus
|<code>allowPrometheus</code>
|true
|<code>true</code>
|enables Prometheus metric reports
|Enables Prometheus metric reporting
|-
|-
|prometheusUser
|<code>prometheusUser</code>
|<username>
|<code><username></code>
|Optional. If provided, the metrics are password protected
|(Optional) If set, metrics require authentication
|-
|-
|prometheusPassword
|<code>prometheusPassword</code>
|<password>
|<code><password></code>
|Optional, If provided, the metrics are password protected. Note that both username and password must be provided. '''Otherwise, metrics are not password protected!'''
|(Optional) Password for authentication. '''Both username and password must be provided, otherwise metrics remain unprotected.'''
|}
|}
Ideally, your modified configuration line would look something like this:
{"target": "server", "action": "add-https", "host": "0.0.0.0", "port": "443", "allowMonitor": true, "allowPrometheus": true, "prometheusUser": "prometheus", "prometheusPassword": "a-good-password"},


=== Serving Prometheus over dedicated port ===
=== Example Configuration ===
To add a dedicated prometheus listener, we'll have to add an additional configuration entry in the json file. Here we'll take some precautions to ensure that only Prometheus metrics are served - you can also choose whether to use https or http. Depending on your configuration you might limit access to the port by other means, making user based authentication optional. This is however completely in your responsbility.
Your modified configuration should look something like this:
  {"target": "server", "action": "add-http", "host": "127.0.0.1", "port": "181818", "allowMonitor": false, "allowTunnel": false, "allowPrometheus": true, "prometheusUser": "prometheus", "prometheusPassword": "a-good-password"},
<code>{"target": "server", "action": "add-https", "host": "0.0.0.0", "port": "443", "allowMonitor": true, "allowPrometheus": true, "prometheusUser": "prometheus", "prometheusPassword": "a-good-password"}</code>
Use action: "add-http" for http, or action: "add-https" for https. Select host and port according to your requirements.
 
== Serving Prometheus Over a Dedicated Port ==
To run Prometheus metrics on a dedicated port, add a new configuration entry to <code>bondixserver.json</code>. This ensures only Prometheus metrics are served. You can choose between HTTP and HTTPS.
 
=== Example Configuration ===
The following configuration allows Prometheus metrics on a dedicated port while restricting other services:
  <code>{"target": "server", "action": "add-http", "host": "127.0.0.1", "port": "181818", "allowMonitor": false, "allowTunnel": false, "allowPrometheus": true, "prometheusUser": "prometheus", "prometheusPassword": "a-good-password"}</code>
 
=== Configuration Notes: ===
 
* Use <code>"action": "add-http"</code> for HTTP or <code>"action": "add-https"</code> for HTTPS.
* Adjust the <code>host</code> and <code>port</code> as per your requirements.
* If other security measures restrict access to the port, authentication (<code>prometheusUser</code> and <code>prometheusPassword</code>) may be optional.


=== Adding bondix server to Prometheus ===
== Adding Bondix Server to Prometheus ==
Here is an example snippet to for scraping Bondix servers using https and basic_auth
To configure Prometheus to scrape metrics from the Bondix server, add the following snippet to your Prometheus configuration file:
  - job_name: 'endpoints'
  - job_name: 'endpoints'
    static_configs:
    static_configs:
      - targets: ['your_endpoint_server.com']
      - targets: ['your_endpoint_server.com']
     scheme: https
     scheme: https
    basic_auth:                                                                                                                                                username: 'prometheus'
    basic_auth:
      password: 'a-good-password'
      username: 'prometheus'
   tls_config:
      password: 'a-good-password'
      insecure_skip_verify: true
    tls_config:
    enable_http2: false
      insecure_skip_verify: true
    enable_http2: false
This configuration enables Prometheus to securely scrape metrics over HTTPS with basic authentication.
----
 
= Available Metrics =
Here is an overview of the available Prometheus metrics, grouped by category.
----
 
== 1. Tunnel Metrics ==
These metrics provide information about tunnel status, traffic, and processing.
 
* '''<code>bondix_server_connected_tunnels</code>''' (gauge) – Total number of connected tunnels per environment. 
''Labels: <code>environment_interface</code>, <code>environment_name</code>''
* '''<code>bondix_server_tunnel_status</code>''' (gauge) – Tunnel status. Possible Values:
** 0 = Disconnected
** 1 = Connected
** 2 = No channel connected, waiting for reconnect
 
* '''<code>bondix_server_tunnel_process_time</code>''' (gauge) – Time taken to process a tunnel (nanoseconds).
* '''<code>bondix_server_tunnel_environment_index</code>''' (gauge) – Environment index associated with a tunnel.
* '''<code>bondix_server_tunnel_proxy_connections</code>''' (gauge) – Current number of proxy connections.
* '''<code>bondix_server_tunnel_qos_rule_count</code>''' (gauge) – Number of active QoS (Quality of Service) rules.
* '''Traffic Metrics:'''
** '''<code>bondix_server_tunnel_total_rx_bytes</code>''' – Total bytes received.
** '''<code>bondix_server_tunnel_total_tx_bytes</code>''' – Total bytes transmitted.
** '''<code>bondix_server_tunnel_uptime</code>''' – Tunnel uptime (seconds).
''Labels: <code>tunnel</code>''
 
----
 
== 2. Environment Metrics ==
These metrics provide details about packet cache usage and processing time for environments.
 
* '''<code>bondix_server_environment_packet_cache_size</code>''' (gauge) – Packet cache size.
* '''<code>bondix_server_environment_packet_cache_use</code>''' (gauge) – Number of packet cache entries in use.
* '''<code>bondix_server_environment_process_time</code>''' (gauge) – Time taken to process all tunnels in an environment (nanoseconds).
''Labels: <code>environment_name</code>''
 
----
 
== 3. Tunnel Channel Metrics ==
These metrics track channel performance, connectivity, and traffic.
 
=== Channel Connection and Status ===
 
* '''<code>bondix_server_tunnel_channel_connected_count</code>''' (gauge) – Number of connected channels per tunnel.
* '''<code>bondix_server_tunnel_channel_count</code>''' (gauge) – Total number of channels per tunnel.
* '''<code>bondix_server_tunnel_channel_status</code>''' (gauge) – Status of a tunnel channel. Possible values:
** 0 = Disconnected
** 1 = Control Connect (TCP Handshake)
** 2 = Data/UDP Handshake I
** 3 = Data/UDP Handshake II
** 4 = Connected
** 5 = Connected, Stalled
** 6 = Connected, Standby
''Labels: <code>channel</code>, <code>tunnel</code>''
 
=== Latency and Performance ===
 
* '''<code>bondix_server_tunnel_channel_latency</code>''' (gauge) – Current latency (milliseconds) of a tunnel channel.
* '''<code>bondix_server_tunnel_channel_idle_latency</code>''' (gauge) – Idle latency of a tunnel channel.
* '''<code>bondix_server_tunnel_channel_latency_ms</code>''' (histogram) – Distribution of channel latency (milliseconds).
** <code>_bucket</code>: Observations within different latency ranges.
** <code>_sum</code>: Total latency sum.
** <code>_count</code>: Number of observations.
''Labels: <code>channel</code>, <code>tunnel</code>''
* '''<code>bondix_server_tunnel_channel_loss</code>''' (gauge) – Packet loss percentage.  ''Labels: <code>tunnel</code>''
=== Traffic Metrics ===
 
* '''Received Data:'''
** '''<code>bondix_server_tunnel_channel_total_rx_bytes</code>''' – Total received bytes.
** '''<code>bondix_server_tunnel_channel_total_rx_packets</code>''' – Total received packets.
** '''<code>bondix_server_tunnel_channel_total_rx_lost_packets</code>''' – Total lost packets on the receiving side.
* '''Transmitted Data:'''
** '''<code>bondix_server_tunnel_channel_total_tx_bytes</code>''' – Total transmitted bytes.
** '''<code>bondix_server_tunnel_channel_total_tx_packets</code>''' – Total transmitted packets.
** '''<code>bondix_server_tunnel_channel_total_tx_lost_packets</code>''' – Total lost packets on the transmitting side. 
''Labels: <code>channel</code>, <code>tunnel</code>''
 
=== Other ===
 
* '''<code>bondix_server_tunnel_channel_uptime</code>''' (gauge) – Tunnel channel uptime (seconds).
''Labels: <code>channel</code>, <code>tunnel</code>''

Latest revision as of 15:38, 11 March 2025

Enabling Prometheus Support

Prometheus support must be explicitly enabled by modifying the configuration file located at /etc/bondixserver.json or /etc/saneserver.json. You can either:

  • Serve Prometheus metrics over the default HTTPS port, or
  • Configure a dedicated listener for Prometheus metrics.

Once enabled, Prometheus metrics will be accessible at:

https://<server-ip>/metrics

Serving Prometheus Over the Default HTTPS Port

To enable Prometheus on the default HTTPS port, update your bondixserver.json configuration file.

Modify the Configuration

Locate the following line:

{"target": "server", "action": "add-https", "host": "0.0.0.0", "port": "443", "allowMonitor": true},

Add the following properties:

Property Value Description
allowPrometheus true Enables Prometheus metric reporting
prometheusUser <username> (Optional) If set, metrics require authentication
prometheusPassword <password> (Optional) Password for authentication. Both username and password must be provided, otherwise metrics remain unprotected.

Example Configuration

Your modified configuration should look something like this:

{"target": "server", "action": "add-https", "host": "0.0.0.0", "port": "443", "allowMonitor": true, "allowPrometheus": true, "prometheusUser": "prometheus", "prometheusPassword": "a-good-password"}

Serving Prometheus Over a Dedicated Port

To run Prometheus metrics on a dedicated port, add a new configuration entry to bondixserver.json. This ensures only Prometheus metrics are served. You can choose between HTTP and HTTPS.

Example Configuration

The following configuration allows Prometheus metrics on a dedicated port while restricting other services:

{"target": "server", "action": "add-http", "host": "127.0.0.1", "port": "181818", "allowMonitor": false, "allowTunnel": false, "allowPrometheus": true, "prometheusUser": "prometheus", "prometheusPassword": "a-good-password"}

Configuration Notes:

  • Use "action": "add-http" for HTTP or "action": "add-https" for HTTPS.
  • Adjust the host and port as per your requirements.
  • If other security measures restrict access to the port, authentication (prometheusUser and prometheusPassword) may be optional.

Adding Bondix Server to Prometheus

To configure Prometheus to scrape metrics from the Bondix server, add the following snippet to your Prometheus configuration file:

  - job_name: 'endpoints'
    static_configs:
      - targets: ['your_endpoint_server.com']
    scheme: https
    basic_auth:
      username: 'prometheus'
      password: 'a-good-password'
    tls_config:
      insecure_skip_verify: true
    enable_http2: false

This configuration enables Prometheus to securely scrape metrics over HTTPS with basic authentication.


Available Metrics

Here is an overview of the available Prometheus metrics, grouped by category.


1. Tunnel Metrics

These metrics provide information about tunnel status, traffic, and processing.

  • bondix_server_connected_tunnels (gauge) – Total number of connected tunnels per environment.

Labels: environment_interface, environment_name

  • bondix_server_tunnel_status (gauge) – Tunnel status. Possible Values:
    • 0 = Disconnected
    • 1 = Connected
    • 2 = No channel connected, waiting for reconnect
  • bondix_server_tunnel_process_time (gauge) – Time taken to process a tunnel (nanoseconds).
  • bondix_server_tunnel_environment_index (gauge) – Environment index associated with a tunnel.
  • bondix_server_tunnel_proxy_connections (gauge) – Current number of proxy connections.
  • bondix_server_tunnel_qos_rule_count (gauge) – Number of active QoS (Quality of Service) rules.
  • Traffic Metrics:
    • bondix_server_tunnel_total_rx_bytes – Total bytes received.
    • bondix_server_tunnel_total_tx_bytes – Total bytes transmitted.
    • bondix_server_tunnel_uptime – Tunnel uptime (seconds).

Labels: tunnel


2. Environment Metrics

These metrics provide details about packet cache usage and processing time for environments.

  • bondix_server_environment_packet_cache_size (gauge) – Packet cache size.
  • bondix_server_environment_packet_cache_use (gauge) – Number of packet cache entries in use.
  • bondix_server_environment_process_time (gauge) – Time taken to process all tunnels in an environment (nanoseconds).

Labels: environment_name


3. Tunnel Channel Metrics

These metrics track channel performance, connectivity, and traffic.

Channel Connection and Status

  • bondix_server_tunnel_channel_connected_count (gauge) – Number of connected channels per tunnel.
  • bondix_server_tunnel_channel_count (gauge) – Total number of channels per tunnel.
  • bondix_server_tunnel_channel_status (gauge) – Status of a tunnel channel. Possible values:
    • 0 = Disconnected
    • 1 = Control Connect (TCP Handshake)
    • 2 = Data/UDP Handshake I
    • 3 = Data/UDP Handshake II
    • 4 = Connected
    • 5 = Connected, Stalled
    • 6 = Connected, Standby

Labels: channel, tunnel

Latency and Performance

  • bondix_server_tunnel_channel_latency (gauge) – Current latency (milliseconds) of a tunnel channel.
  • bondix_server_tunnel_channel_idle_latency (gauge) – Idle latency of a tunnel channel.
  • bondix_server_tunnel_channel_latency_ms (histogram) – Distribution of channel latency (milliseconds).
    • _bucket: Observations within different latency ranges.
    • _sum: Total latency sum.
    • _count: Number of observations.

Labels: channel, tunnel

  • bondix_server_tunnel_channel_loss (gauge) – Packet loss percentage. Labels: tunnel

Traffic Metrics

  • Received Data:
    • bondix_server_tunnel_channel_total_rx_bytes – Total received bytes.
    • bondix_server_tunnel_channel_total_rx_packets – Total received packets.
    • bondix_server_tunnel_channel_total_rx_lost_packets – Total lost packets on the receiving side.
  • Transmitted Data:
    • bondix_server_tunnel_channel_total_tx_bytes – Total transmitted bytes.
    • bondix_server_tunnel_channel_total_tx_packets – Total transmitted packets.
    • bondix_server_tunnel_channel_total_tx_lost_packets – Total lost packets on the transmitting side.

Labels: channel, tunnel

Other

  • bondix_server_tunnel_channel_uptime (gauge) – Tunnel channel uptime (seconds).

Labels: channel, tunnel