Supermicro X9 compatibility

[petersulyok]

Based on this forum topic we can assume that Super Micro X9 motherboards are behaving differently in setting up fan's level with IPMI. The current working hypothesis is that smfc is not working properly on a X9 motherboards.

This issue was created to collect all testing and correction effort related this issue.
Thanks to @matthuska for raising my attention on this issue.

[petersulyok]

A new branch created, called x9compat, with a new IPMI script called set_ipmi_fan_level_x9.sh.
A basic test to check if the fan mode and fan level settings are working correctly based on the forum topic:

1. Set the threshold for your fans

   • edit and execute set_ipmi_threshold.sh script properly
   • check the values with ipmitool sensor command

2. set IMPI mode to FULL

   • execute set_ipmi_fan_mode.sh 1 command
   • can check the result on the remote web interface

3. setup an arbitrary fan level (1-100%) - be careful with the lower limit

   • execute set_ipmi_fan_level_x9.sh 0 50 command
   • check the fan level with ipmitool sdr command
   • execute set_ipmi_fan_level_x9.sh 1 100 command
   • check the fan level with ipmitool sdr command

[matthuska]

Motherboard model: X9SCi-LN4F. As you pointed out this has a different BMC than the one smfc was originally developed to work with, the "Nuvoton WPCM450 BMC".

Test results:

Set the threshold for your fans

* edit and execute `set_ipmi_threshold.sh` script properly

* check the values with `ipmitool sensor` command

On this motherboard, the fans are called "FAN 1" instead of "FAN1" (note the space). Once that was changed, and the actual thresholds modified to suit my fans, this script worked.

Note that if the nct6775 module is loaded (see below, currently needed for fan level setting), you also need to modify the fanx_min settings to match or be lower than the thresholds you have set via ipmitool, for example: echo 100 > /sys/devices/platform/nct6775.2608/hwmon/hwmon6/fan1_min. If you don't do this you will have similar behavior to when you haven't set the thresholds properly via ipmitool.

set IMPI mode to FULL

* execute `set_ipmi_fan_mode.sh 1` command

* can check the result on the remote web interface

This worked without any modifications. I would suggest modifying the script to output the current fan mode if no mode is specified on the command line. For my motherboard the command to do that is: ipmitool raw 0x30 0x45 0x00

setup an arbitrary fan level (1-100%) - be careful with the lower limit

* execute `set_ipmi_fan_level_x9.sh 0 50` command

* check the fan level with `ipmitool sdr` command

* execute `set_ipmi_fan_level_x9.sh 1 100` command

* check the fan level with `ipmitool sdr` command

ipmitool-based raw commands for fan level control don't work so far. I tried several commands that were suggested on various forums with no luck (I'll document these properly in a following comment). Currently, I can control the fan levels by loading the nct6775 module and using something like the following command: echo (fan level 0-255) > /sys/devices/platform/nct6775.2608/hwmon/hwmon6/pwm1 (pwm1 is for FAN A and pwm2 is for FAN 1-4 I think, need to double check this).

I'll incorporate these changes, test them for a while to see how they work for me, and then do a PR against the x9compat branch.

[petersulyok]

I found this script for Supermicro X9 boards using an Nuvoton WPCM450 BMC. I think it would makes sense to test the ipmi commands in it (especially adjusting fan levels).

[matthuska]

No luck. The script uses the following command:

ipmitool raw 0x30 0x91 0x5A 0x3 0x10 0x35

trying that on my system gives an error:

Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x91 rsp=0x82): Unknown (0x82)

[petersulyok]

Please make sure that nct6775 kernel module is not loaded (it might interfere with IPMI or can lock some resources).
Then apply the 50% fan level in the zone where your fans are installed:

# Peripheral zone - FANA, FANB
ipmitool raw 0x30 0x91 0x5A 0x3 0x10 127

# CPU zone - FAN1, FAN2
ipmitool raw 0x30 0x91 0x5A 0x3 0x11 127

I would be also interested to see the output of the following commands:

ipmitool sdr
impitool sensors

My last idea: you may also check Supermicro's own IPMI tool, called IPMICFG, can be downloaded from here (I assume you need the Linux 64-bit version). It has a slightly different syntax but it surely supports Nuvoton BMC while ipmitool might have issues with that.

Thanks for your help!

[matthuska]

First, the output of the commands you asked about:

# ipmitool sdr
System Temp      | 0 degrees C       | ok
Peripheral Temp  | 30 degrees C      | ok
CPU Temp         | 0x08              | ok
FAN 1            | no reading        | ns
FAN 2            | 675 RPM           | ok
FAN 3            | 675 RPM           | ok
FAN 4            | 750 RPM           | ok
FAN A            | 975 RPM           | ok
Vcore            | 0.66 Volts        | ok
3.3VCC           | 3.34 Volts        | ok
12V              | 12.24 Volts       | ok
VDIMM            | 1.51 Volts        | ok
5VCC             | 5.06 Volts        | ok
-12V             | -12.29 Volts      | ok
VBAT             | 3.09 Volts        | ok
VSB              | 3.34 Volts        | ok
AVCC             | 3.36 Volts        | ok
Chassis Intru    | 0x01              | ok
# ipmitool sensor
System Temp      | 0.000      | degrees C  | ok    | -9.000    | -7.000    | -5.000    | 80.000    | 85.000    | 90.000    
Peripheral Temp  | 30.000     | degrees C  | ok    | -9.000    | -7.000    | -5.000    | 80.000    | 85.000    | 90.000    
CPU Temp         | 0x8        | discrete   | 0x0700| na        | na        | na        | na        | na        | na        
FAN 1            | na         |            | na    | na        | na        | na        | na        | na        | na        
FAN 2            | 675.000    | RPM        | ok    | 0.000     | 75.000    | 150.000   | 1575.000  | 1725.000  | 1800.000  
FAN 3            | 675.000    | RPM        | ok    | 0.000     | 75.000    | 150.000   | 1575.000  | 1725.000  | 1800.000  
FAN 4            | 750.000    | RPM        | ok    | 0.000     | 75.000    | 150.000   | 1575.000  | 1725.000  | 1800.000  
FAN A            | 975.000    | RPM        | ok    | 0.000     | 75.000    | 150.000   | 2100.000  | 2175.000  | 2325.000  
Vcore            | 0.688      | Volts      | ok    | 0.480     | 0.512     | 0.544     | 1.488     | 1.520     | 1.552     
3.3VCC           | 3.344      | Volts      | ok    | 2.816     | 2.880     | 2.944     | 3.584     | 3.648     | 3.712     
12V              | 12.243     | Volts      | ok    | 10.494    | 10.600    | 10.706    | 13.091    | 13.197    | 13.303    
VDIMM            | 1.512      | Volts      | ok    | 1.152     | 1.216     | 1.280     | 1.760     | 1.776     | 1.792     
5VCC             | 5.056      | Volts      | ok    | 4.096     | 4.320     | 4.576     | 5.344     | 5.600     | 5.632     
-12V             | -12.292    | Volts      | ok    | -13.844   | -13.650   | -13.456   | -10.934   | -10.740   | -10.546   
VBAT             | 3.088      | Volts      | ok    | 2.816     | 2.880     | 2.944     | 3.584     | 3.648     | 3.712     
VSB              | 3.344      | Volts      | ok    | 2.816     | 2.880     | 2.944     | 3.584     | 3.648     | 3.712     
AVCC             | 3.360      | Volts      | ok    | 2.816     | 2.880     | 2.944     | 3.584     | 3.648     | 3.712     
Chassis Intru    | 0x1        | discrete   | 0x0100| na        | na        | na        | na        | na        | na   

[matthuska]

I removed the nct6775 module and tried the two ipmitool commands you provided, but again they both return an error:

# modprobe -r nct6775
# ipmitool raw 0x30 0x91 0x5A 0x3 0x11 127
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x91 rsp=0x82): Unknown (0x82)
# ipmitool raw 0x30 0x91 0x5A 0x3 0x10 127
Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x91 rsp=0x82): Unknown (0x82)

[matthuska]

And finally with ipmicfg:

# ./ipmicfg -summary
Summary
-------------------------------------------
IPv4 Address           : <redacted>
BMC MAC Address        : <redacted>
Firmware Revision      : 3.38
Firmware Build Time    : 01/05/2015
System LAN1 MAC Address: <redacted>
System LAN2 MAC Address: <redacted>
# ./ipmicfg -raw 0x30 0x91 0x5A 0x3 0x11 127
IPMI command not completed normally, Completion Code=82h
# ./ipmicfg -raw 0x30 0x91 0x5A 0x3 0x10 127
IPMI command not completed normally, Completion Code=82h

So still no luck setting fan duty cycles using ipmitool or ipmicfg. I tried looking up the code (82h) with no luck.

It looks like our best best at this point is to use ipmitool for setting the fan mode and thresholds, as you're doing already, and rely on the nct6776 module and /sys/ interface for setting fan duty cycles. What do you think? I've been using that for the last few weeks (with my own old script, I still have to adapt smfc) and it's been flawless.

[petersulyok]

Thanks for the clarification.

My current understanding is that your motherboard has limited IPMI functionality (can monitor sensors but cannot change them). I found similar forum feedbacks here on X9 motherboards.

Related to the fan mode, please check if the fan mode is configured correctly (I have a bad feeling that cannot be interpreted here either on this HW). For example, I see this output for my X11 motherboard:

  #./IPMICFG-Linux.x86_64 -fan
  Current Fan Speed Mode is [ Full Mode ]
  
  Supported Fan modes:
  0:Standard
  1:Full
  2:Optimal
  4:Heavy IO

I tend to accept your solution proposal, and it requires additional configuration items, codes and unit tests where I'm happy to contribute.

[matthuska]

Setting fan modes works perfectly. I typically have the fan in Full Mode and manually control the duty cycle using the /sys/ interface:

# ./ipmicfg -fan
Current Fan Speed Mode is [ Full Mode ]

Supported Fan modes:
0:Standard
1:Full
2:Optimal
# ./ipmicfg -sdr | grep FAN
       | (205) FAN 1              |          N/A |       N/A |        N/A |
    OK | (272) FAN 2              |      675 RPM |    75 RPM |   1725 RPM |
    OK | (339) FAN 3              |      675 RPM |    75 RPM |   1725 RPM |
    OK | (406) FAN 4              |      750 RPM |    75 RPM |   1725 RPM |
    OK | (473) FAN A              |      975 RPM |    75 RPM |   2175 RPM |

And setting to other modes has the intended effect:

# ./ipmicfg -fan 0
Done.
# ./ipmicfg -fan
Current Fan Speed Mode is [ Standard Mode ]

Supported Fan modes:
0:Standard
1:Full
2:Optimal
# ./ipmicfg -sdr | grep FAN
       | (205) FAN 1              |          N/A |       N/A |        N/A |
    OK | (272) FAN 2              |      300 RPM |    75 RPM |   1725 RPM |
    OK | (339) FAN 3              |      300 RPM |    75 RPM |   1725 RPM |
    OK | (406) FAN 4              |      375 RPM |    75 RPM |   1725 RPM |
    OK | (473) FAN A              |      975 RPM |    75 RPM |   2175 RPM |
# ./ipmicfg -fan 1
Done.
# ./ipmicfg -fan
Current Fan Speed Mode is [ Full Mode ]

Supported Fan modes:
0:Standard
1:Full
2:Optimal
# ./ipmicfg -sdr | grep FAN
       | (205) FAN 1              |          N/A |       N/A |        N/A |
    OK | (272) FAN 2              |     1350 RPM |    75 RPM |   1725 RPM |
    OK | (339) FAN 3              |     1275 RPM |    75 RPM |   1725 RPM |
    OK | (406) FAN 4              |     1425 RPM |    75 RPM |   1725 RPM |
    OK | (473) FAN A              |     1575 RPM |    75 RPM |   2175 RPM |
# ./ipmicfg -fan 2
Done.
# ./ipmicfg -fan
Current Fan Speed Mode is [ Optimal Mode ]

Supported Fan modes:
0:Standard
1:Full
2:Optimal
# ./ipmicfg -sdr | grep FAN
       | (205) FAN 1              |          N/A |       N/A |        N/A |
    OK | (272) FAN 2              |      300 RPM |    75 RPM |   1725 RPM |
    OK | (339) FAN 3              |      300 RPM |    75 RPM |   1725 RPM |
    OK | (406) FAN 4              |      375 RPM |    75 RPM |   1725 RPM |
    OK | (473) FAN A              |      975 RPM |    75 RPM |   2175 RPM |

The commands work equally well when using ipmitool:

# ipmitool raw 0x30 0x45 0x00
 01
# ipmitool raw 0x30 0x45 0x01 0x00

# ipmitool raw 0x30 0x45 0x00
 00
# ipmitool raw 0x30 0x45 0x01 0x02

# ipmitool raw 0x30 0x45 0x00
 02

I'd suggest sticking with the ipmitool interface for fan mode setting, as you're already using since it's easier to install in most cases.

[petersulyok]

I have he following high-level solution proposal in my mind:

1. Configuration changes, we have to introduce new configuration parameters in smfc.conf:

    [Ipmi]
    motherboard_type=[ X9_WPCM450 | X9 | X10X11 ]
    x9_module = nct6775
    x9_cpu_zone_fans = pwm3, pwm4, pwm5
    x9_hd_zone_fans = pwm1, pwm2

Note:

• I would separate a generic X9 case with the IPMI raw command (ipmitool raw 0x30 0x91 0x5A 0x3 0x1x xxx)

2. Changes in Ipmi() class:

• in case of X9_WPCM450 motherboards, we have to find/build path strings for the fan control files in HWMON. This will be the most complex step.
• we have to implement different versions of the Ipmi.set_fan_level() function (for all motherboard types) and use the adequate one (through a function variable).

Documentations and unit tests should be also updated. I'm happy to do the implementation if you can test that on your HW, or happy to receive any input from you.

[matthuska]

Sounds like a very reasonable approach. I don't have much free time at the moment but I'd be happy to comment on your implementation if you have time to put something together, or if you're also busy I can do it when I have a bit more time.