opencomputeproject / project-zipline

Defines a lossless compressed data format that is independent of CPU type, operating system, file system, and character set, and is suitable for compression using the XP10 algorithm.

Project Zipline

Introduction

Project Zipline is a program to accelerate innovation in lossless compression. With this release Microsoft is making available:

• A new compression format called XP10 which is tailored for modern cloud datasets.
• All specifications for the entire pipeline.
• All RTL to support implementation for the entire pipeline.
• A testbench environment specifically developed for the VCS simulator.

Project Zipline is open-sourced under the MIT License, see the LICENSE file.

Specifications

The following Project Zipline specifications have been included under the “specs” directory in this repository:

• Project_Zipline_Compression_Specification
• Project_Zipline_Huffman_Encoder_Micro_Architecture_Specification
• Project_Zipline_Comp_lz77_Micro_Architecture_Specification
• Project_Zipline_Crypto_Engine_Micro_Architecture_Specification
• Project_Zipline_Decompression_Top_Micro_Architecture_Specification
• Project_Zipline_Keyblob_Micro_Architecture_Specification
• Project_Zipline_KME_Micro_Architecture_Specification
• Project_Zipline_Prefix_Attach_Micro_Architecture_Specification
• Project_Zipline_Prefix_Micro_Architecture_Specification
• Project_Zipline_SSB_Micro_Architecture_Specification
• Project_Zipline_Top_Micro_Architecture_Specification

Simulation Notes

Edit the file ./zipline.setup for VCS, SYNTH, and VERDI environment settings.

Then source the file as follows:

source zipline.setup

Go to either the CCE_64, CDD_64, or KME run directory:

cd dv/CCE_64/run

cd dv/CDD_64/run

cd dv/KME/run

To build the simv executable:

make build_simv

Example for simulating a test:

make run_simv TESTNAME=xp10

Example for simulating a test with waves (.vpd):

make run_simv TESTNAME=xp10 WAVES=1

Example for simulating a test with Verdi waves (.fsdb):

make run_simv TESTNAME=xp10 VERDI_WAVES=1

All of the tests for the CCE_64, CDD_64, and KME can be run via a regress script in the associated "run" directory:

dv/CCE_64/run/regress

dv/CDD_64/run/regress

dv/KME/run/regress

All test files are located in the associated engine directory "tests". Each directory also contains a README file with a description of each test.

dv/CCE_64/tests

dv/CDD_64/tests

dv/KME/tests

A description of the programmable registers in the CCE/CDD/KME can be found in the register_doc directory. Please read register_doc/README for more details.

KME Notes

The KME RTL (rtl/cr_kme) has been modified to remove the following modules:

• AES engine in the random GUID generator
• SHA engines within the KDF function
• AES engine within the Key Decryption logic

With these reductions, the RTL only supports Key Types 1-6 without KDF and Key Type 0 without encryption/authentication

Additionally, the KME output has been reduced to support a single engine.

All modifications are identified with "KME_MODIFICATION_NOTE" comments in the RTL code.

Synthesis Notes

Note: For synthesis, please edit syn/Makefile to choose an LSF or dedicated machine with at least 16G of memory. Edit the contents of this line, before "dc_shell":

bsub -Is -q irv-cpx-M16 -R "rusage [mem=16000]" dc_shell -f syn_example.tcl | tee ./LOGS/syn_eample.tcl.date '+%m%d.%H:%M'.log

To run synthesis for CDD

cd syn/cr_cddip

make syn

To run synthesis for CCE

cd syn/cr_cceip_64

make syn

To run synthesis for KME

cd syn/cr_kme

make syn

project-zipline's Issues

Undefined System Task Call $fsdbDumpfile

Hi,
I am running the make build_simv and getting the following error

Error-[VERDI_VCS_MM] Possible VERDI_HOME and VCS_HOME mismatch
  The Verdi tab and pli.a cannot be added to the 'vcs' compile command because
  the file
  '/global/apps/verdi_2019.06/verdi/Verdi_P-2019.06/share/PLI/VCS/LINUX64/pli.a'
  could not be found.
  Please check to make sure VERDI_HOME and VCS_HOME point to installations
  that are either both 32-bits or both 64-bits.

                         Chronologic VCS (TM)
         Version P-2019.06_Full64 -- Thu Dec 12 17:03:58 2019
               Copyright (c) 1991-2019 by Synopsys Inc.
                         ALL RIGHTS RESERVED


Error-[UST] Undefined System Task Call
./zipline_tb.v, 169
  Undefined System Task call to '$fsdbDumpfile'.

how can I resolve this issue?

simulation hangs using Cadence toolflow

I'm trying to run project zipline using Cadence simulator. It compiles the RTL and tb correctly, but then runs and hangs after this step: APB_INFO: @time:573 vector --> w 0x00000374 0x0007a120
It doesn't even use any simulation time, like it's waiting for something.

This is my command to compile, elaborate, and run
build_simv: kme.vlist
xrun -work LIB_TOP -sv -assert -f kme.vlist -l vlogan.log +TESTNAME=kme_key_type_0 +SEED=${SEED}

Are there any command line options I should add ?

Thanks

is there any plan to convert the code to Opencl 2.0?

hi, great software

I would like to know if is there any plan to convert the code to Opencl 2.0?

Build instructions

Hello author, I'm really appreciate that you can open source this part of the code, because I am studying data compression, but in your source code according to the establishment of your instructions for simulation encountered some problems.So I want to know exactly how you compiled it successfully in this section- from RTL to simv .Thank you very much.

Implementing the design using Vivado

Hi,
I found this project very interesting. I wanted to run it on a Zynq board.
I tried to make this project in Xilinx Vivado, however, it seems it's not that straight forward.
Can you please give me some information about the way that I make the Vivado project?

Missing macros?

I have a bunch of macros which don't seem to be defined anywhere.
Any idea where those are defined?

 1 [HDL 9-3952] use of undefined macro 'LOG_VEC'

  2 [HDL 9-3952] use of undefined macro 'BIT_VEC'

  3 [HDL 9-3952] use of undefined macro 'BIT_EC'

  4 [HDL 9-3952] use of undefined macro 'LOG_EC'

  5 [HDL 9-3952] use of undefined macro 'CR_ISF_C_AUX_CMD_EV_MATCH_AL_0_COMP_T_DECL'
  
6 [HDL 9-3952] use of undefined macro 'CR_ISF_C_AUX_CMD_EV_MATCH_AL_0_CRYPTO_T_DECL'

  7 [HDL 9-3952] use of undefined macro 'CR_ISF_C_AUX_CMD_EV_MASK_AL_0_COMP_T_DECL'

  8 [HDL 9-3952] use of undefined macro 'CR_ISF_C_AUX_CMD_EV_MASK_AL_0_CRYPTO_T_DECL'

  9 [HDL 9-3952] use of undefined macro 'CR_ISF_C_AUX_CMD_EV_MATCH_AL_1_COMP_T_DECL'

 10 [HDL 9-3952] use of undefined macro 'CR_ISF_C_AUX_CMD_EV_MATCH_AL_1_CRYPTO_T_DECL'

 11 [HDL 9-3952] use of undefined macro 'CR_ISF_C_AUX_CMD_EV_MASK_AL_1_COMP_T_DECL'

 12 [HDL 9-3952] use of undefined macro 'CR_ISF_C_AUX_CMD_EV_MASK_AL_1_CRYPTO_T_DECL'

 13 [HDL 9-3952] use of undefined macro 'CR_ISF_C_AUX_CMD_EV_MATCH_AL_2_COMP_T_DECL'

 14 [HDL 9-3952] use of undefined macro 'CR_ISF_C_AUX_CMD_EV_MATCH_AL_2_CRYPTO_T_DECL'

 15 [HDL 9-3952] use of undefined macro 'CR_ISF_C_AUX_CMD_EV_MASK_AL_2_COMP_T_DECL'

 16 [HDL 9-3952] use of undefined macro 'CR_ISF_C_AUX_CMD_EV_MASK_AL_2_CRYPTO_T_DECL'

 17 [HDL 9-3952] use of undefined macro 'CR_ISF_C_AUX_CMD_EV_MATCH_AL_3_COMP_T_DECL'

 18 [HDL 9-3952] use of undefined macro 'CR_ISF_C_AUX_CMD_EV_MATCH_AL_3_CRYPTO_T_DECL'

 19 [HDL 9-3952] use of undefined macro 'CR_ISF_C_AUX_CMD_EV_MASK_AL_3_COMP_T_DECL'

 20 [HDL 9-3952] use of undefined macro 'CR_ISF_C_AUX_CMD_EV_MASK_AL_3_CRYPTO_T_DECL'

 21 [HDL 9-3952] use of undefined macro 'CR_ISF_AUX_CMD_EV_MATCH_AL_0_COMP_CONFIG'

 22 [HDL 9-3952] use of undefined macro 'CR_ISF_AUX_CMD_EV_MATCH_AL_0_CRYPTO_CONFIG'

 23 [HDL 9-3952] use of undefined macro 'CR_ISF_AUX_CMD_EV_MASK_AL_0_COMP_CONFIG'

 24 [HDL 9-3952] use of undefined macro 'CR_ISF_AUX_CMD_EV_MASK_AL_0_CRYPTO_CONFIG'

 25 [HDL 9-3952] use of undefined macro 'CR_ISF_AUX_CMD_EV_MATCH_AL_1_COMP_CONFIG'

 26 [HDL 9-3952] use of undefined macro 'CR_ISF_AUX_CMD_EV_MATCH_AL_1_CRYPTO_CONFIG'

 27 [HDL 9-3952] use of undefined macro 'CR_ISF_AUX_CMD_EV_MASK_AL_1_COMP_CONFIG'

 28 [HDL 9-3952] use of undefined macro 'CR_ISF_AUX_CMD_EV_MASK_AL_1_CRYPTO_CONFIG'

 29 [HDL 9-3952] use of undefined macro 'CR_ISF_AUX_CMD_EV_MATCH_AL_2_COMP_CONFIG'

 30 [HDL 9-3952] use of undefined macro 'CR_ISF_AUX_CMD_EV_MATCH_AL_2_CRYPTO_CONFIG'

 31 [HDL 9-3952] use of undefined macro 'CR_ISF_AUX_CMD_EV_MASK_AL_2_COMP_CONFIG'

 32 [HDL 9-3952] use of undefined macro 'CR_ISF_AUX_CMD_EV_MASK_AL_2_CRYPTO_CONFIG'

 33 [HDL 9-3952] use of undefined macro 'CR_ISF_AUX_CMD_EV_MATCH_AL_3_COMP_CONFIG'

 34 [HDL 9-3952] use of undefined macro 'CR_ISF_AUX_CMD_EV_MATCH_AL_3_CRYPTO_CONFIG'

 35 [HDL 9-3952] use of undefined macro 'CR_ISF_AUX_CMD_EV_MASK_AL_3_COMP_CONFIG'

 36 [HDL 9-3952] use of undefined macro 'CR_ISF_AUX_CMD_EV_MASK_AL_3_CRYPTO_CONFIG'

 37 [HDL 9-3952] use of undefined macro 'CR_KME_C_KDF_DRBG_STATE_ALUE_T_DECL'

 38 [HDL 9-3952] use of undefined macro 'CR_KME_KDF_DRBG_SEED_0_STATE_ALUE_31_0'

 39 [HDL 9-3952] use of undefined macro 'CR_KME_KDF_DRBG_SEED_0_STATE_ALUE_63_32'

 40 [HDL 9-3952] use of undefined macro 'CR_KME_KDF_DRBG_SEED_0_STATE_ALUE_95_64'

 41 [HDL 9-3952] use of undefined macro 'CR_KME_KDF_DRBG_SEED_0_STATE_ALUE_127_96'

 42 [HDL 9-3952] use of undefined macro 'CR_KME_KDF_DRBG_SEED_1_STATE_ALUE_31_0'

 43 [HDL 9-3952] use of undefined macro 'CR_KME_KDF_DRBG_SEED_1_STATE_ALUE_63_32'

 44 [HDL 9-3952] use of undefined macro 'CR_KME_KDF_DRBG_SEED_1_STATE_ALUE_95_64'

 45 [HDL 9-3952] use of undefined macro 'CR_KME_KDF_DRBG_SEED_1_STATE_ALUE_127_96'

 46 [HDL 9-3952] use of undefined macro 'CR_KME_CCEIP_ALIDATE_KOP_FIFO_OVERRIDE'

 47 [HDL 9-3952] use of undefined macro 'TRUNC_NUM_3'

 48 [HDL 9-3952] use of undefined macro 'N_PHD_WORDS'

 49 [HDL 9-3952] use of undefined macro 'N_PFD_WORDS'

 50 [HDL 9-3952] use of undefined macro 'BIT_EC_BASE'

Decompression RTL ?

Hi,
I just bumped into this and from the FAQ, it seems there is no decompression RTL being released? I haven't dive into RTL yet but it seemes there is a decompression spec. Basically I need to come up with my own decompression rtl logic?

During OCP Summit'19, only the Huffman encoder associated with compression was released.
As Microsoft announced during the opening keynote, we will be open sourcing the entire
compression and decompression pipelines. This will include hooks for data authentication
and encryption as well. We will also release a test bench to allow developers to exercise
the pipeline implementation. Please see HISTORY.md or individual component release history.

Regards
Yao

mis-match between the doc and data in test

I am not sure if my understanding is correct.
I have checked the data in the gzip.inbound, and the head data are
0x400000000a000400 SoT 0xff
0x0000000000000000 EoT 0xff
0x800000000a000601 SoT 0xff
0x000000000580f000 0xff
0x3a00000000000450 EoT 0xff
0x800000000a000d0f SoT 0xff

i can see from the waveform they are mapping to RQE, CMD,AUX FRMD part but each field i can hardly aligned as 3.1.2.9 comp_ctrl (Offset:0x10[31:0], Size: 32) have field algorithm, but i cannot find the data above 0x3a00000000000450 can mapped to gzip algo ..
also in the spec the define Length of the TLV in units of 4 bytes. This length is computed over the entirety of the TLV Type, Length and data fields are the same description for each header part in the Top_Micro_Architeture spec .
Are there any updated spec and is there plan for this IP roadmap?

thanks

Vivado Implementation Issues

Hi @michaelgmcintyre ,
While trying to implement this project in Vivado, we found some syntax errors and some ports which were wrongly declared, like in file:

• https://github.com/opencomputeproject/Project-Zipline/blob/master/rtl/cr_prefix/cr_prefix_fe_ctlr.v

Can you please clarify whether these are bugs or any mistakes that we are overlooking from our side?

Doubts regarding to KME test-cases

Hello, In CCEIP and CDDIP test-cases are somewhat satisfying according to specifications and we can see or make some conclusion from the waveforms. But in KME, as we know , it is mentioned in the RTL or verilog code that, whoever use this code they have to add their own AES ans SHA definitions. So, it is absent in this verilog code, then in this Key management engine what are this given test-cases verifying?
Because it is not generating any key of AES and SHA , then in test-cases what are we verifying? Even in input means in kme_ib_tdata signal TLV format is not properly given , means input MEGA TLV is also not correct.
So, I couldn't understand what these test-cases are doing? Can you please put some light on it?

C reference implementation of X10 codec

Is there a C/C++ reference implementation of X10 compressor and decompressor to benchmark some workloads.

Parameter to change compression level (window size)

Is the RTL parameterized to facilitate the support for smaller search window sizes?

Also, the symbol_queue in the RTL (cr_huf_comp_sq.v) defaults to a depth of 20,480. Is it safe to use a smaller queue? What are the factors I should consider when changing the size of this queue?

Understanding the Top level parameters

Hi,

I came across this article and was interested in having a look at the RTL code. In the top level file, I have seen a parameter name FPGA_MODE which can be 0 or 1.
Can you please give me a brief idea of what this parameter does?

Requesting CCEIP CDDIP Memory Maps in machine readable file format.

The register mapping details as provided here are in html & pdf formats only. Can you provide the register maps in any machine readable format , like json5, ip-xact or xml, if possible?

@michaelgmcintyre I made a new github account, I used my private one (Meradanis) earlier.

Hi Bastian
I see that you have worked on Makefile. I have a question regarding this issue:

vlib work
** Warning: (vlib-34) Library already exists at "work".
vdel -all
vlib work
vmap work work
QuestaSim-64 vmap 2019.2_1 Lib Mapping Utility 2019.05 May 13 2019
vmap work work
Modifying modelsim.ini
vlog zipline.vlist
QuestaSim-64 vlog 2019.2_1 Compiler 2019.05 May 13 2019
Start time: 07:42:24 on Sep 27,2020
vlog zipline.vlist
** Error: (vlog-13069) zipline.vlist(2): near "#": syntax error, unexpected '#', expecting class.
End time: 07:42:24 on Sep 27,2020, Elapsed time: 0:00:00

It would be great is you can help me with this.

Size of Zipline.

Hi, I'm sawool.

I compiled zipline design. But I suprised because of sizes of zipline.

It has almost 38,000,000 gate counts.

I think that it is something wrong.

Do you know size after compile ?

I cant find information about sizes of zipline.

Request for Performant XP10 SW Library

Looking for updates regarding the availability of a performant XP10 SW Library that was referenced in the OCP Zipline presentation back in 2019.

Issues related to the verification and use of zipline IP

Hello,
There are something i want to know about zipline. First, does CCE_IP or CDD_IP have encryption and decryption module, I can not find them? Second, does Zipline open source have C model or other reference model, i think it's hard to verify the ip with only rtl and spec. Last one, can you open source of some verification platform like uvm, i think the testbench in the project can not completely verify the IP. Thank you.