can't get data from device, but snmpwalk does. #850
Comments
|
For questions/help/support please use our community channels. There are more people available to potentially respond to your request and the whole community can benefit from the answers provided. |
|
debug log from snmp_exporter |
|
Here the "fmt" section of the snmp.yml |
|
This does seem like there may be an actual walk issue with SNMPv1 here. But it does seem like this is a device bug we may need to workaround in the upstream gosnmp library. |
|
Have you tried reducing |
|
I wonder if we should change the exporter default to 10 to match net-snmp. |
|
Tried max_repetitions: 10 or 1 and didn't help. |
|
@jsfrerot Can you try replacing |
|
same result. tried with max_repetitions: 10 or 1 again. |
|
It's difficult to debug flaky snmp engines without access to the device, but you could also try experimenting a bit with appending a (cf. http://www.net-snmp.org/wiki/index.php/FAQ:Applications_09) |
|
Could you NB: Be very careful in where and how you share the resulting files, and use a temporary community string, not your production one! |
|
I have included both tcpdumps in the original post and I think I understand you want it to be in a file. Is there any particular tcpdump flags you want to have ? |
|
@RichiH Here are the pcap files. Let me know if you need anything else. |
|
The key difference in the snmp_exporter capture is that it is trying to do a The As a last resort, list the objects that you want individually in your generator.yml, e.g.: modules:
fmt:
version: 1
walk:
- vCardState
- vDeviceType
- ...=> generate => # WARNING: This file was auto-generated using snmp_exporter generator, manual changes will be lost.
fmt:
get:
- 1.3.6.1.4.1.40989.10.16.1.2.1.0
- 1.3.6.1.4.1.40989.10.16.1.2.2.0
metrics:
- name: vCardState
oid: 1.3.6.1.4.1.40989.10.16.1.2.1
type: gauge
help: Description. - 1.3.6.1.4.1.40989.10.16.1.2.1
enum_values:
0: "off"
1: "on"
- name: vDeviceType
oid: 1.3.6.1.4.1.40989.10.16.1.2.2
type: OctetString
help: Description. - 1.3.6.1.4.1.40989.10.16.1.2.2
version: 1Note how it produces a list of OIDs to get, with fmt:
walk:
- 1.3.6.1.4.1.40989.10 |
|
@jsfrerot Out of curiosity, can you try an snmpwalk of 1.3.6.1.4.1.40989 without the |
|
@dswarbrick you're right, snmpwalk doesn't work without the -Cc option: |
|
It's nearly 10 years since I wrote my last MIB, but I noticed that the first subidentifier after the "MG-SOFT Corporation" private enterprise number is From what I can tell, the product is not actually made by MG-SOFT, which appears to just be a software house that specialises in SNMP MIB creation / browsing tools. The actual hardware is made by FS.COM, and they simply haven't filled in their company information or private enterprise number (assuming they have one). Reading the SNMP section in their FMT quickstart guide really makes me wonder if they know what they are doing. They have different MIBs depending on which slot a module is plugged into. This really seems like they don't understand SNMP indexes. They even have spelling mistakes in the object names in the OAP-C1-OEO MIB, e.g. @jsfrerot Can you try Failing that, you'll need a more verbose generator.yml which will walk the walk-able parts of the MIB, and get the remaining scalar objects. modules:
fmt:
version: 1
walk:
- vCardState
- vDeviceType
- vDeviceDescription
- vSoftwareVerion
- vHardwareVerion
- vSerialNumber
- vFactoryDate
- vSFPA1
- vSFPA2
- vSFPB1
- vSFPB2
- vSFPC1
- vSFPC2
- vSFPD1
- vSFPD2You might also want to implement some metric relabelling config in Prometheus to fix the misspelled metric names. PS: This might make it a little clearer to see the MIB represented hierarchically. |
|
@RichiH here is the output as requested so it doesn't work. and here is what I get: I've tried to use what you proposed: but the result is more limited: |
Ooof. That is disappointing. I was pretty sure that it would successfully walk the vSFPxx branches since their leaf OIDs are contiguous. The SNMP agent in that hardware must be really brain dead if you have to explicitly GET each object. Be thankful that the MIB is not that large. |
|
It looks like you're going to have to override the types for several of the metrics: This is yet another sign of a poorly-written MIB, that isn't specifying the correct encoding type. It looks like the label values are being handled as |
|
@dswarbrick very helpful insights. I was going to propose a PR to support this device. Not sure if it's a good idea anymore, but still I would like to be able to get the data from them. |
|
@jsfrerot My personal opinion is that snmp_exporter actually shouldn't ship with any vendor-specific config, since the amount of SNMP-capable hardware out there is virtually limitless, and as you've found out first hand, some of it can be pretty dubious. I recognise that for a lot of people, SNMP MIBs are akin to black magic, and they're often hard to source, especially for EOL hardware. The MIBs themselves often fall under some copyright or incompatible license, which is why they are not packaged by Debian for example, as they would violate Debian's packaging policies. Indeed, the Debian prometheus-snmp-exporter package (which I co-maintain) strips out the bundled snmp.yml. From my own experience operating snmp_exporter in a large service provider environment, with Juniper, Mellanox, Dell, and Supermicro hardware, there rarely is a case of "one size fits all", and we used a completely from-scratch config that only walked the OIDs that we were specifically interested in. The out-of-the-box config may have worked, but would have resulted in huge amounts of unwanted metrics on some of the large Juniper switch virtual chassis (as well as having a negative and quite noticeable impact on the CPUs of some network hardware). |
|
@dswarbrick I setup https://github.com/prometheus-community/snmp as a place to start a project to create a whole bunch of snmp_exporter generator.yml examples for various devices. But I never got around to deciding exactly how to layout such a project. For example, policies like "should we inline vendor MIBs with dubious copyright issues". |
and others
Host operating system: output of
uname -aLinux yul-blackbox-001 5.4.0-137-generic #154-Ubuntu SMP Thu Jan 5 17:03:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
snmp_exporter version: output of
snmp_exporter -versionsnmp_exporter, version 0.21.0 (branch: HEAD, revision: 0d8c352)
build user: root@51dfd4b1f59b
build date: 20221122-15:14:18
go version: go1.19.3
platform: linux/amd64
What device/snmpwalk OID are you using?
FMT04-CH1U, 1U Managed Chassis https://www.fs.com/products/39214.html?attribute=54780&id=870796
OID: 1.3.6.1.4.1.40989
If this is a new device, please link to the MIB(s).
https://bestmonitoringtools.com/mibdb/mibs/OAP-C1-OEO.mib
There is C1 through C16, it's basically 1.3.6.1.4.1.40989.10.16.1 through 1.3.6.1.4.1.40989.10.16.16 on seperate files
What did you do that produced an error?
I used the generator with the following diff
Than ran "make clean", "make mibs" and "make generate"
uploaded the snmp.yml to /etc/snmp-exporter.yaml on the snmp_exporter node and started it with the following params:
/usr/local/bin/snmp_exporter --config.file=/etc/snmp-exporter.yaml
What did you expect to see?
a snmpwalk like this show the expected result:
and here is the tcpdump for troubleshooting
What did you see instead?
The snmp_exporter shows only this:
here is the snmp tcpdump if it can help:
The text was updated successfully, but these errors were encountered: