Telemetry Configuration Guide

introduction introduction telemetry is a technology that enables high speed remote data collection from network devices it facilitates rapid fault isolation and efficient, intelligent network operations monitored devices proactively and in real time push their own metrics (such as cpu, memory, and traffic statistics), logs, and events to a collector or analytics platform this model replaces traditional pull based monitoring mechanisms, such as snmp polling grpc (google remote procedure calls) is a high performance, general purpose open source rpc framework released by google in 2015 it is built on the http/2 protocol and supports interface definition using protocol buffers, with data transmitted in binary encoding it also provides cross platform capabilities both communication parties perform secondary development based on this framework grpc handles transport, security, and encoding, allowing developers to focus on application layer interface definition and implementation telemetry reports encoded data to the collector over the grpc protocol for reception and storage the default data encoding format is json ietf, which is a yang json encoding strictly defined in ietf rfc 7951 basic concepts basic concepts subscription modes subscription modes telemetry supports two subscription modes dial in and dial out the difference lies in the roles of the device and the collector dial in mode in dial in mode, the device acts as the server and the collector acts as the client the collector actively establishes a connection to the device and listens for data to perform collection the device side configuration is relatively simple this mode is suitable for short term state monitoring dial out mode in dial out mode, the device acts as the client and the collector acts as the server the device proactively initiates a connection to the collector and pushes telemetry data this mode is more suitable for long term monitoring in large scale networks reporting modes reporting modes in dial in mode, the device supports two data reporting modes stream and poll in stream mode, subscriptions support on change reports data only when a change occurs sampled reports data periodically at a specified interval in poll mode, data is retrieved through explicit polling requests initiated by the collector in dial out mode, the device supports periodic and stream reporting modes in periodic mode, the device sends data at regular intervals in stream mode, data transmission is triggered by changes different reporting modes support different collection targets and yang paths collection targets collection targets collection targets are classified into db and non db types in dial in mode, the supported db targets include appl db appl db asic db asic db counters db counters db loglevel db loglevel db config db config db pfc wd db pfc wd db flex counter db flex counter db state db state db for non db targets others others (others is a system defined target type for non database data) in dial out mode, the supported db targets include counters db counters db state db state db for non db targets others others collection paths collection paths for db type data , users can create custom events for data collection the collection path supports the following formats \<table> \<table> / \<key> \<table> / \<key> / \<field> in addition, the system supports a set of predefined collection paths, also referred to as virtual paths , as listed below table 1 db type events table 1 db type events event name target virtual path description total ingress/egress pfc packets counters db counters/ethernet /pfc cnt total ingress packet loss counters db counters/ethernet /ingress loss cnt collects packet drops caused by buffer overflow not supported on n v2 models total egress packet loss counters db counters/ethernet /queue egress loss cnt queue transmitted bytes counters db counters/ethernet /queue transmitted bytes queue dropped bytes counters db counters/ethernet /queue dropped bytes queue transmitted packets counters db counters/ethernet /queue transmitted packets queue dropped packets counters db counters/ethernet /queue dropped packets ipv4 traffic statistics counters db counters/ethernet /ip status not supported on n v2 models ipv6 traffic statistics counters db counters/ethernet /ipv6 status traffic statistics summary counters db counters/ethernet /pkt status includes packet drops, error packets, and bum traffic statistics note the asterisk ( ) indicates a wildcard for example counters/ethernet /pfc cnt collects pfc statistics for all ethernet interfaces counters/ethernet0/pfc cnt collects pfc statistics for ethernet0 only if non db type data is collected, the following fixed paths are supported table 2 non db type events table 2 non db type events event name path target path logs of configuration operations others cmd/history cpu utilization others platform/cpu memory statistics others proc/meminfo i/o statistics others proc/diskstats system load average others proc/loadavg virtual memory statistics others proc/vmstat cpu and system level statistics others proc/stat default telemetry configuration default telemetry configuration the default telemetry configuration is shown in the table below table 3 default telemetry configuration table 3 default telemetry configuration parameter default value telemetry feature enabled telemetry destination collector not configured telemetry subscription not configured telemetry data reporting interval 5000 ms data encoding format json ietf configuring telemetry in dial in mode configuring telemetry in dial in mode the grpc service is enabled by default and listens on all interfaces no manual configuration is required ensure that layer 3 connectivity between the device and the collector is reachable note the grpc service can be disabled using the command feature telemetry state disable configuring telemetry in dial out mode configuring telemetry in dial out mode configuring the telemetry client configuring the telemetry client table 4 telemetry client configuration table 4 telemetry client configuration operation command description enter global configuration mode configure terminal enter telemetry client configuration mode telemetry client create a telemetry destination group destination group destination group name configure the collector ip address and port { ipv4 address a b c d | ipv6 address a b } port port number a b c d / a b collector address should be layer 3 reachable from the device port number collector listening port only one collector can be configured per destination group configure multiple destination groups if multiple collectors are required configuring telemetry subscriptions configuring telemetry subscriptions table 5 telemetry subscription configuration table 5 telemetry subscription configuration operation command description enter global configuration mode configure terminal enter telemetry client configuration mode telemetry client create a telemetry subscription and enter its configuration mode subscription subscription name bind a destination group to the subscription destination group destination group name the subscription should be bound to a destination group before communication can be established each subscription supports only one destination group configure the collection target path target \[ counters db | state db | others ] each subscription supports only one collection target configure collection paths paths path name refer to telemetry configuration guide for the path format multiple paths can be configured per subscription configure reporting mode report type \[ periodic | stream ] (optional) configure reporting interval when periodic mode is used report interval interval unit milliseconds range 100 to 1,800,000 default 5000 ms display and maintenance display and maintenance table 6 telemetry display and maintenance table 6 telemetry display and maintenance operation command description display telemetry client configuration show telemetry client summary display telemetry destination group configuration show telemetry client destination group \[ destination group name ] display telemetry subscription configuration and status show telemetry client subscription \[ all | subscription name ] typical configuration examples typical configuration examples configuring dial out telemetry to subscribe to packet loss statistics configuring dial out telemetry to subscribe to packet loss statistics network requirement as the network scale increases, users need to optimize and troubleshoot the network based on device statistics for example, when packet loss occurs on the device, the device reports the data to the collector this enables timely traffic monitoring and performance tuning network topology configuration steps step 1 configure the interface ip address sonic# configure terminal sonic(config)# interface ethernet 0/0 sonic(config if 0/0)# ip address 10 0 0 2/24 step 2 configure the collector the collector uses opentelemetry collector the sample configuration is shown below receivers otlp protocols grpc endpoint "0 0 0 0 8081" tls cert file "d /programs/otelcol/server crt" key file "d /programs/otelcol/server key" exporters file path "d /programs/otelcol/exporters json" logging service pipelines traces receivers \[otlp] exporters \[file, logging] metrics receivers \[otlp] exporters \[file, logging] logs receivers \[otlp] exporters \[file, logging] ensure that layer 3 connectivity between the device and the collector is established step 3 configure the telemetry client sonic# configure terminal sonic(config)# telemetry client sonic(telemetry client)# destination group test sonic(telemetry destination group test)# ipv4 address 10 0 0 1 port 8081 sonic(telemetry destination group test)# exit sonic(telemetry client)# subscription test sonic(telemetry subscription test)# destination group test sonic(telemetry subscription test)# path target counters db sonic(telemetry subscription test)# paths counters/ethernet0/queue egress loss cnt sonic(telemetry subscription test)# report interval 10000 sonic(telemetry subscription test)# report type periodic sonic(telemetry subscription test)# exit verification verify that the opentelemetry collector receives the reported data logging output and the exported file should contain the reported packet loss statistics \# logging 2024 11 01t11 44 25 978+0800 info tracesexporter {"kind" "exporter", "data type" "traces", "name" "logging", "resource spans" 1, "spans" 1} 2024 11 01t11 44 35 979+0800 info tracesexporter {"kind" "exporter", "data type" "traces", "name" "logging", "resource spans" 1, "spans" 1} \# exporters json &#x9;{ &#x9; "resourcespans" \[ &#x9; { &#x9; "resource" { &#x9; "attributes" \[ &#x9; { &#x9; "key" "service name", &#x9; "value" { &#x9; "stringvalue" "grpc dialout" &#x9; } &#x9; } &#x9; ] &#x9; }, &#x9; "scopespans" \[ &#x9; { &#x9; "scope" { &#x9; "name" "telemetry dialout/client" &#x9; }, &#x9; "spans" \[ &#x9; { &#x9; "traceid" "dc78f3671e112263b54dd5fa1a6fb45d", &#x9; "spanid" "6818243589fca8ec", &#x9; "parentspanid" "", &#x9; "name" "dialout report", &#x9; "kind" 1, &#x9; "starttimeunixnano" "1730432661299663511", &#x9; "endtimeunixnano" "1730432661299733369", &#x9; "attributes" \[ &#x9; { &#x9; "key" "sonic data", &#x9; "value" { &#x9; "stringvalue" "update <\n timestamp 1730432661296098535\n prefix <\n target \\"counters db\\"\n >\n update <\n path <\n elem <\n name \\"counters\\"\n >\n elem <\n name \\"ethernet0\\"\n >\n elem <\n name \\"queue egress loss cnt\\"\n >\n >\n val <\n json ietf val \\"{\\\\\\"ethernet0 0\\\\\\" {\\\\\\"queue dropped bytes\\\\\\" \\\\\\"0\\\\\\",\\\\\\"queue dropped packets\\\\\\" \\\\\\"0\\\\\\",\\\\\\"type\\\\\\" \\\\\\"unicast\\\\\\"},\\\\\\"ethernet0 1\\\\\\" {\\\\\\"queue dropped bytes\\\\\\" \\\\\\"0\\\\\\",\\\\\\"queue dropped packets\\\\\\" \\\\\\"0\\\\\\",\\\\\\"type\\\\\\" \\\\\\"unicast\\\\\\"},\\\\\\"ethernet0 2\\\\\\" {\\\\\\"queue dropped bytes\\\\\\" \\\\\\"0\\\\\\",\\\\\\"queue dropped packets\\\\\\" \\\\\\"0\\\\\\",\\\\\\"type\\\\\\" \\\\\\"unicast\\\\\\"},\\\\\\"ethernet0 3\\\\\\" {\\\\\\"queue dropped bytes\\\\\\" \\\\\\"0\\\\\\",\\\\\\"queue dropped packets\\\\\\" \\\\\\"0\\\\\\",\\\\\\"type\\\\\\" \\\\\\"unicast\\\\\\"},\\\\\\"ethernet0 4\\\\\\" {\\\\\\"queue dropped bytes\\\\\\" \\\\\\"0\\\\\\",\\\\\\"queue dropped packets\\\\\\" \\\\\\"0\\\\\\",\\\\\\"type\\\\\\" \\\\\\"unicast\\\\\\"},\\\\\\"ethernet0 5\\\\\\" {\\\\\\"queue dropped bytes\\\\\\" \\\\\\"0\\\\\\",\\\\\\"queue dropped packets\\\\\\" \\\\\\"0\\\\\\",\\\\\\"type\\\\\\" \\\\\\"unicast\\\\\\"},\\\\\\"ethernet0 6\\\\\\" {\\\\\\"queue dropped bytes\\\\\\" \\\\\\"0\\\\\\",\\\\\\"queue dropped packets\\\\\\" \\\\\\"0\\\\\\",\\\\\\"type\\\\\\" \\\\\\"unicast\\\\\\"},\\\\\\"ethernet0 7\\\\\\" {\\\\\\"queue dropped bytes\\\\\\" \\\\\\"0\\\\\\",\\\\\\"queue dropped packets\\\\\\" \\\\\\"0\\\\\\",\\\\\\"type\\\\\\" \\\\\\"unicast\\\\\\"}}\\"\n >\n >\n>\n" &#x9; } &#x9; } &#x9; ], &#x9; "events" \[ &#x9; { &#x9; "timeunixnano" "1730432661299668184", &#x9; "name" "telemetry dialout" &#x9; } &#x9; ], &#x9; "status" {} &#x9; } &#x9; ] &#x9; } &#x9; ], &#x9; "schemaurl" "https //opentelemetry io/schemas/v1 4 0" &#x9; } &#x9; ] &#x9;} configuring dial out telemetry to subscribe to ptp status configuring dial out telemetry to subscribe to ptp status network requirement to meet visualization and o\&m requirements for high precision time synchronization, the user plans to deploy a time synchronization network based on ptp (precision time protocol) telemetry is introduced to monitor and centrally analyze ptp runtime status requirements changes in the ptp gm identity should be reported in real time other ptp information should be collected every 10 seconds network topology configuration roadmap configure ptp (omitted) ensure layer 3 connectivity between the device and the collector (omitted) configure opentelemetry collector on the collector (omitted) configure telemetry subscriptions on the device for gm identity and other ptp data separately configuration steps switch a is used as an example other devices are configured similarly step 1 configure the telemetry destination group switcha# configure terminal switcha(config)# telemetry client switcha(telemetry client)# destination group ptp switcha(telemetry destination group ptp)# ipv4 address 10 250 0 247 port 4317 switcha(telemetry destination group ptp)# exit step 2 configure a subscription for gm identity (real time reporting) switcha(telemetry client)# subscription ptp gmi switcha(telemetry subscription ptp gmi)# destination group ptp switcha(telemetry subscription ptp gmi)# path target state db switcha(telemetry subscription ptp gmi)# paths ptp domain/127/config/grandmaster clock identity switcha(telemetry subscription ptp gmi)# report type stream switcha(telemetry subscription ptp gmi)# exit step 3 configure a subscription for other ptp information (periodic reporting) switcha(telemetry client)# subscription ptp switcha(telemetry subscription ptp)# destination group ptp switcha(telemetry subscription ptp)# path target state db switcha(telemetry subscription ptp)# paths ptp domain/127/config switcha(telemetry subscription ptp)# report type periodic switcha(telemetry subscription ptp)# report interval 10000 verification verify the configuration the destination group is configured correctly switcha# show telemetry client destination group + + + + \| name | ipv4 destinations | ipv6 destinations | +========+=====================+=====================+ \| ptp | 10 250 0 247 4317 | | + + + + the subscription state shows sub state success switcha# show telemetry client subscription + + + + \| name | destination group | path target | +============+=====================+===============+ \| ptp | ptp | state db | + + + + \| ptp gmi | ptp | state db | + + + + switcha# show telemetry client subscription ptp \============================== name ptp destination group ptp path target state db report type periodic report interval 10000 ms paths ptp domain/127/config sub state success switcha# show telemetry client subscription ptp gmi \============================== name ptp gmi destination group ptp path target state db report type stream paths ptp domain/127/config/grandmaster clock identity sub state success verify the opentelemetry collector receives real time updates when the ptp gm identity changes 2026 02 28t17 22 13 725+0800 info traces {"resource" {"service instance id" "85227404 4820 42b6 b5c9 f712ee26258e", "service name" "otelcol", "service version" "0 141 0"}, "otelcol component id" "debug", "otelcol component kind" "exporter", "otelcol signal" "traces", "resource spans" 1, "spans" 1} 2026 02 28t17 22 13 725+0800 info resourcespans #0 resource schemaurl https //opentelemetry io/schemas/v1 4 0 resource attributes > service name str(grpc dialout) scopespans #0 scopespans schemaurl instrumentationscope telemetry dialout/client span #0 trace id 08d1d1bc54224c1da437c08bdc63619c parent id id 1fa55528dbdffa3a name dialout report kind internal start time 2026 02 28 09 21 42 084190055 +0000 utc end time 2026 02 28 09 21 42 084201147 +0000 utc status code unset status message attributes > sonic data str(update < timestamp 1772270502084010719 prefix < target "state db" \> update < path < elem < name "ptp domain" \> elem < name "127" \> elem < name "config" \> elem < name "grandmaster clock identity" \> \> val < string val "001000 fffe 000172" \> \> \> ) events spanevent #0 > name telemetry dialout > timestamp 2026 02 28 09 21 42 08419638 +0000 utc > droppedattributescount 0 {"resource" {"service instance id" "85227404 4820 42b6 b5c9 f712ee26258e", "service name" "otelcol", "service version" "0 141 0"}, "otelcol component id" "debug", "otelcol component kind" "exporter", "otelcol signal" "traces"} periodic reports of other ptp information every 10 seconds 2026 02 28t17 20 36 266+0800 info traces {"resource" {"service instance id" "85227404 4820 42b6 b5c9 f712ee26258e", "service name" "otelcol", "service version" "0 141 0"}, "otelcol component id" "debug", "otelcol component kind" "exporter", "otelcol signal" "traces", "resource spans" 1, "spans" 1} 2026 02 28t17 20 36 266+0800 info resourcespans #0 resource schemaurl https //opentelemetry io/schemas/v1 4 0 resource attributes > service name str(grpc dialout) scopespans #0 scopespans schemaurl instrumentationscope telemetry dialout/client span #0 trace id bc7e948d7e79cf6eff2b066fe9b7ef4a parent id id 4c8898c6bd4541a7 name dialout report kind internal start time 2026 02 28 09 20 03 68683352 +0000 utc end time 2026 02 28 09 20 03 686940305 +0000 utc status code unset status message attributes > sonic data str(update < timestamp 1772270403686610270 prefix < target "state db" \> update < path < elem < name "ptp domain" \> elem < name "127" \> elem < name "config" \> \> val < json ietf val "{\\"clock accuracy\\" \\"fe\\",\\"clock class\\" \\"248\\",\\"clock priority1\\" \\"125\\",\\"clock priority2\\" \\"128\\",\\"clock step\\" \\"two step\\",\\"clock type\\" \\"bc\\",\\"domain\\" \\"127\\",\\"grandmaster clock accuracy\\" \\"fe\\",\\"grandmaster clock class\\" \\"248\\",\\"grandmaster clock identity\\" \\"001000 fffe 000173\\",\\"grandmaster clock priority1\\" \\"10\\",\\"grandmaster clock priority2\\" \\"128\\",\\"local clock identity\\" \\"000000 fffe 109109\\",\\"local ts\\" \\"257052135973854\\",\\"master clock identity\\" \\"000000 0000 000000\\",\\"max steps removed\\" \\"255\\",\\"offset to master\\" \\"1\\",\\"parent port num\\" \\"1\\",\\"path delay\\" \\"250\\",\\"servo state\\" \\"locked\\"}" \> \> \> ) events spanevent #0 > name telemetry dialout > timestamp 2026 02 28 09 20 03 686840264 +0000 utc > droppedattributescount 0 {"resource" {"service instance id" "85227404 4820 42b6 b5c9 f712ee26258e", "service name" "otelcol", "service version" "0 141 0"}, "otelcol component id" "debug", "otelcol component kind" "exporter", "otelcol signal" "traces"}