monorail task confused by Front I/O board? about hubris

[Aaron-Hartwig]

I looked at this with @mkeeter this morning. Matt mentioned that this looked like it could be power related due to some prior debugging experience. Some attempts to talk to the PHY over the SMI manually yielded notably strange failures:

aaron@jeeves ~ $ pfexec humility -t sidecar hiffy -t ecp5_front_io -c Fpga.user_design_write_reg -adevice_index=1,addr=24,op=Write,value=2
humility: attached to 0483:374e:002900184741500520383733 via ST-Link V3
Fpga.user_design_write_reg() => ()
aaron@jeeves ~ $ pfexec humility -t sidecar hiffy -t ecp5_front_io -c Fpga.user_design_write_reg -adevice_index=1,addr=25,op=Write,value=2
humility: attached to 0483:374e:002900184741500520383733 via ST-Link V3
Fpga.user_design_write_reg() => ()
aaron@jeeves ~ $ pfexec humility -t sidecar hiffy -t ecp5_front_io -c Fpga.user_design_read -adevice_index=1,addr=19 --num=2
humility: attached to 0483:374e:002900184741500520383733 via ST-Link V3
Fpga.user_design_read() => ()
Data: [0, 0]

Rather than yielding the data expected from the device, we read back 0x0000. That is strange given that a non-responsive device expected to speak MDIO would result in a bus which floats high for the whole read, or 0xFFFF. suspicion of power intensifies

Due to a rather invasive power change (see MCN AE04 for details), we cannot trust the FPGA with respect to what it knows about PHY power. The SMI interface between the FPGA and PHY has its signals pulled up to V2P5_PHY_A2:

This was useful to us during remote debugging because those pins go to the FPGA, so we could read their status. I made an adjustment to the QSFP FPGA image to keep the MDIO bidir pin in tri-state and added a register to our SPI-accessible interface, allowing us to see if it floated up or not, indicating that V2P5_PHY_A2 was present. While we already had MDINT_L in that register interface, it is ambiguous if that signal would be low due to a power failure or the PHY driving it low.
The FPGA changes I made:

aaron@Aaron-Lenovo:~/Oxide/git/quartz/build$ gs
# On branch: main...origin/main  |  [*] => $e*
#
➤ Changes not staged for commit
#
#       modified: [1] ../hdl/MDIO/MDIO.bsv
#       modified: [2] ../hdl/boards/sidecar/qsfp_x32/VSC8562/VSC8562.bsv
#       modified: [3] ../hdl/boards/sidecar/qsfp_x32/VSC8562/vsc8562.rdl
#
aaron@Aaron-Lenovo:~/Oxide/git/quartz/build$ gd 1 2 3
diff --git a/hdl/MDIO/MDIO.bsv b/hdl/MDIO/MDIO.bsv
index d8a9809..5ddcd16 100644
--- a/hdl/MDIO/MDIO.bsv
+++ b/hdl/MDIO/MDIO.bsv
@@ -70,6 +70,7 @@ interface MDIO;
     interface Pins pins;
     interface Put#(Command) command;
     interface Get#(Bit#(16)) read_data;
+    interface Reg#(Bit#(1)) mdio_in_r;
 endinterface

 module mkMDIO #(Parameters parameters) (MDIO);
@@ -92,8 +93,8 @@ module mkMDIO #(Parameters parameters) (MDIO);
     // pin registers
     Reg#(Bit#(1)) mdc           <- mkReg(1);
     Reg#(Bit#(1)) mdint         <- mkReg(0);
-    Reg#(Bit#(1)) mdio_out      <- mkReg(1);
-    Reg#(Bit#(1)) mdio_out_en   <- mkReg(1);
+    Reg#(Bit#(1)) mdio_out      <- mkReg(0);
+    Reg#(Bit#(1)) mdio_out_en   <- mkReg(0);
     Reg#(Bit#(1)) mdio_in       <- mkReg(0);

     // Edge detection for MDC
@@ -139,8 +140,8 @@ module mkMDIO #(Parameters parameters) (MDIO);

     (* fire_when_enabled *)
     rule do_idle_bus(state == Idle);
-        mdio_out_en     <= 1;
-        mdio_out        <= 1;
+        mdio_out_en     <= 0;
+        mdio_out        <= 0;
     endrule

     // A future improvement to make would be to dynamically control preamble
@@ -249,6 +250,8 @@ module mkMDIO #(Parameters parameters) (MDIO);
     endinterface

     interface Get read_data = toGet(read_data_q);
+
+    interface mdio_in_r = mdio_in;
 endmodule

 endpackage: MDIO
diff --git a/hdl/boards/sidecar/qsfp_x32/VSC8562/VSC8562.bsv b/hdl/boards/sidecar/qsfp_x32/VSC8562/VSC8562.bsv
index d90cbf7..89ae6a9 100644
--- a/hdl/boards/sidecar/qsfp_x32/VSC8562/VSC8562.bsv
+++ b/hdl/boards/sidecar/qsfp_x32/VSC8562/VSC8562.bsv
@@ -247,7 +247,8 @@ module mkVSC8562 #(Parameters parameters) (VSC8562);
         interface ReadOnly phy_smi_status   = valueToReadOnly(
             PhySmiStatus{
                 busy: pack(smi_busy),
-                mdint: mdint
+                mdint: mdint,
+                mdio: smi.mdio_in_r
             });
         interface Reg phy_smi_rdata_h       = smi_rdata_h;
         interface Reg phy_smi_rdata_l       = smi_rdata_l;
diff --git a/hdl/boards/sidecar/qsfp_x32/VSC8562/vsc8562.rdl b/hdl/boards/sidecar/qsfp_x32/VSC8562/vsc8562.rdl
index 0130041..f460ef4 100644
--- a/hdl/boards/sidecar/qsfp_x32/VSC8562/vsc8562.rdl
+++ b/hdl/boards/sidecar/qsfp_x32/VSC8562/vsc8562.rdl
@@ -67,6 +67,10 @@ addrmap vsc8562 {
         name = "Status bits for the SMI interface to the VSC8562 (valid on FPGA1 only)";
         default sw = r;

+        field {
+            desc = "status of MDIO";
+        } MDIO[2:2] = 0;
+
         field {
             desc = "1 for active interrupt (inversion of miim_phy_to_fpga_mdint_l pin)";
         } MDINT[1:1] = 0;
@@ -140,4 +144,4 @@ addrmap vsc8562 {
             desc = "Read = 0, Write = 1";
         } RW[0:0] = 0;
     } PHY_SMI_CTRL;
-};
\ No newline at end of file
+};

With suspicion mounting that V2P5_PHY_A2 is not functioning, I flashed with my modified FPGA and read out the status register where my useful bits were held.

aaron@jeeves ~ $ pfexec humility -t sidecar hiffy -t ecp5_front_io -c Fpga.user_design_read_reg -a device_index=0x1,addr=0x12
humility: attached to 0483:374e:002900184741500520383733 via ST-Link V3
Fpga.user_design_read_reg() => 0x2

To decode this:
Bit 2 is MDIO, which is currently 0, meaning it has not floated high when we released the bus.
Bit 1 is MDINT, which is current 1. This is an inversion of MDINT_L, so we know it is low as well (and likely not because of the PHY).
Bit 0 is a don't care here.

While none of this is definitive, we strongly expect the rework done to the PHY power supplies (as mentioned above) to be broken in some way.

[Aaron-Hartwig]

This was inspected by @refugeesus and the add-on board we kludged in to supply 5V bias for the LT3072 dev kit had it's output disabled via a pushbutton helpfully placed there. Toggling the button fixed things 🙃