open-power / capi2-bsp

CAPI 2.0 Board Support Package

CAPI2-BSP (CAPI version 2.0 board support)

What is CAPI 2.0?

The majority of computational accelerator systems use PCI Express (PCIe) to connect to the main processor units via the I/O subsystem. The IBM Coherent Accelerator Processor Interface (CAPI) provides an alternative that removes the complexity and overhead of the I/O subsystem to enable higher system performance.

While the first version of CAPI was designed for acceleration in a POWER8 system, its successor CAPI 2.0 is the CAPI version for acceleration through the PCIe link on POWER9 systems.

For more information about CAPI, see https://developer.ibm.com/linuxonpower/capi

Board support and PSL

The Power Service Layer (PSL) is the component for FPGAs that provides access to CAPI.

This repository contains the necessary resources for creating a Xilinx Vivado IP container that integrates the PSL component into an FPGA card specific infrastructure for CAPI 2.0.

Currently, support for the following cards is implemented with Xilinx Vivado 2019.2 (unless otherwise noted):

• AD9V3 (AlphaData 9V3)
• AD9H3 (AlphaData 9H3)
• AD9H7 (AlphaData 9H7)
• N250SP (Nallatech 250S+ tested up to 2018.2)
• RCXVUP (ReflexCES XpressVUP-LP9PT last test was on 2017.4)
• FX609 (Flyslice FX609QL)
• S241 (Semptian NSA241)
• U50 (Xilinx U50 still in development)
• U200 (Xilinx U200 still in development)

The build process for the CAPI 2.0 board support depends on POWER9 PSL (PSL9) sources that can be obtained as zip archive from the IBM Portal for OpenPOWER

https://www.ibm.com/systems/power/openpower

From the menu, select "CAPI"->"Coherent Accelerator Processor Interface (CAPI)" or directly click the "CAPI" icon to go to the CAPI section and download the required zip archive contained in "PSL IP Source Files for POWER9 CAPI".
Alternatively, the following link will provide direct access:

https://www-355.ibm.com/systems/power/openpower/posting.xhtml?postingId=1BED44BCA884D845852582B70076A89A

Please copy the downloaded zip file to the subdirectory psl.

Creating container via make process

With the archived PSL9 IP core available in the subdirectory psl the CAPI board support IP container for a supported card with name <CARD NAME> can simply be generated by calling

make <CARD NAME>

This will automatically create the required PSL9 IP for the card's FPGA chip which will be included in the resulting CAPI board support IP container

<CARD NAME>/build/ip/capi_bsp_wrap.xcix

Adding new cards

In order to add a new card <NEW CARD> a new subdirectory has to be created that contains

• a Makefile for setting up the card specific environment variables (see AD9V3/Makefile for an example)
• a subdirectory src keeping the capi board support sources
• a subdirectory tcl containing at least the following files
  • add_ip.tcl to add card specific IP to the card_board_support project
  • add_src.tcl to add card specific files not contained in <NEW CARD>/src
  • create_ip.tcl to create the required card specific IP
  • optionally a script patch_ip.tcl to apply additional patches
• a subdirectory xdc containing card specific constraint files

For examples, please refer to AD9V3 or N250SP.

After creation of the directories and files as described above, the new card can be enabled by just adding the card to the variable CARDS in the top level Makefile.

PSL9 IP

When creating a CAPI 2.0 board support IP container, the required PSL9 IP will be generated automatically from the archived PSL9 IP core (see Board support and PSL for information on how to obtain that archived IP core). It is not even necessary to take care of that in the card specific create_ip.tcl.

The build scripts for PSL9 IP generation are contained in the subdirectory psl. In that directory the build process also expects to find the archived PSL9 IP core

• ibm.com_CAPI_PSL9_WRAP_$(PSL_VERSION).zip

where PSL_VERSION is an environment variable used by the build process to identify the PSL version. If that variable is not defined, the default value 2.00 will be assumed, currently.

While it is not necessary to explicitly build the PSL9 IP for a card, calling

make -C <CARD_NAME> psl

will build the PSL9 IP for the FPGA part that belongs to the card with name <CARD_NAME>.

Contributing

This is an open-source project. We greatly appreciate your contributions and collaboration. Before contributing to this project, please read and agree to the rules in

• CONTRIBUTING.md

To simplify the sign-off, you may want to create a ".gitconfig" file in you home by executing:

$ git config --global user.name "John Doe"
$ git config --global user.email [email protected]

Then, for every commit, use git commit -s to add the "Signed-off by ..." message.

By default the git repository is read-only. Users can fork the snap repository, make the changes there and issue a pull request. Even members with write access to this repository can't commit directly into the protected master branch. To contribute changes, please create a branch, make the changes there and issue a pull request.

Pull requests to merge into the master branch must be reviewed before they will be merged.

capi2-bsp's Issues

Extraction of zip file is read from SNAP_ROOT dir instead of PSL9_IP_CORE

In snap/snap_env.sh, user specifies a path to ibm.com_CAPI_PSL9_WRAP_2.00.zip file and assign it to PSL9_IP_CORE variable.
In psl/Makefile, the zip file is searched in current snap path regardless of the PSL9_IP_CORE path!
Thus it is impossible to specify a path to a zip file which is not in the current snap path.

Vivado 2018.3 known issue prevents from using this release for building BSP

From the capi-bsp directory (latest release), when executing for example:

make clean
make AD9V3

we get the following error if and only if we use vivado 2018.3

.../...
Generating capi_bsp IP
Applying patches
Creating capi_bsp IP container
ERROR: [filemgmt 56-196] The vivado project is inside the IP directory. You can not convert this IP to a core container.
make[1]: *** [/afs/vlsilab.boeblingen.ibm.com/proj/fpga/framework/mesnet/snap_test_hlsmk/hardware/capi2-bsp/AD9V3/.create_ip_done] Error 1

Error is located at this line
which is known and referenced by Xilinx here

Adding the following line in common/tcl/create_capi_bsp.tcl should implement this workaround
set_property coreContainer.enable 1 [current_project]
The problem is to find where to implement that

Reset of N250SP on P9 leads to lost PCIe link

On the N250S+ in a Zaius P9 system, the card reset via sudo sh -c "echo 1 > /sys/class/cxl/card0/reset"doesn't work most times. This means the card can't be updated reliably without power cycling the system, and automated Jenkins testing is not possible.
This may also affect other CAPI 2.0 cards on P9, so the fix must be done for all supported cards.

The issue was found and fixed in another N250SP design:
Changes required for CAPI2-BSP/N250SP/src/capi_bsp.vhdl: It is missing some fixes related to reset that might be causing lab issues.

1. The MCM that generates the psl_clk and icap_clk should have its reset input tied off:
   -- MMCM to generate PSL clock (100...250MHz)
   pll0: flashgtp_clk_wiz
   PORT MAP (
   clk_in1 => pcihip0_psl_clk, -- Driven by PCIHIP
   clk_out1 => psl_clk, -- Goes to PSL logic
   clk_out2 => psl_clk_div2, -- Goes to PSL logic
   clk_out3 => icap_clk, -- Goes to SEM, multiboot
   clk_out3_ce => icap_clk_ce, -- gate off while unstable to prevent SEM errors
   -- reset => pcihip0_psl_rst, -- Driven by PCIHIP
   -- reset was pcihip0_psl_rst. this killed the clock to icap before a reconfig could complete
   reset => '0',
   locked => clk_wiz_2_locked
   );

2. The reset inputs to PSL9_WRAP_0 need to change as well:

   -- PSL_RST and PCIHIP_PSL_RST must both be asserted if one is asserted
   -- If only 1 is asserted, async fifo gets into invalid state
   PSL_RST         => pcihip0_psl_rst,  -- was psl_reset_sig,
   PSL_CLK         => psl_clk,
   PCIHIP_PSL_RST  => pcihip0_psl_rst,
   PCIHIP_PSL_CLK  => pcihip0_psl_clk
   

=> take over latest PSL changes into capi2-bsp, in addition to the two changes listed above