yaoleiqi / dscnet

Pytorch Implement of Dynamic Snake Convolution (ICCV2023)

Hello friends, I'm Yaolei! 👋

Welcome to my GitHub. I am currently a Ph.D. student in the Department of Computer Science at Southeast University, Nanjing. I am currently advised by Prof. Guanyu Yang throughout both my master's and doctoral studies. Prior to this, I completed my bachelor's degree at Southeast University as well.

  

My research lies at the intersection of Deep Learning and Artificial Intelligence in Medicine -- with a special focus on dealing with the tough task in tubular structure analysis that are knowledge-driven and data-efficient. My research interests include medical imaging, weakly-supervised learning, network design, federated learning, foundation model, and computer-assisted diagnosis, etc. If you have any questions on my projects, please feel free to send me an email.

dscnet's Issues

Can I use 3-channel RGB image instead of only channel G?

Thank you for sharing your code!
However, you mentioned you chose channel G (which contains the most vescular infomation) as input image, is your model also efficient and powerful enough when I used raw 3-channel RGB image as input? I tried to put the 3-channel image from DRIVE dataset, but the final dice indice seemed not as high as your result in you paper.

TCLoss实现细节问题

您好，非常感谢您的开源！但是我关于TCloss的实现有一些困惑。
目前找到一个采用Persistence Homology计算loss方法的源码中（https://github.com/marrlab/SHAPR_torch/tree/master ） 是使用torch_topology.nn.CubicalComplex这个类实现的，不知您是否使用这个方法实现。但是我目前理解这个类的input是每个pixel/voxel对应的一个概率（比如最后一层是sigmoid）。在分类任务中，ground truth是一个整形的tensor，对应的persistence diagram（即您文章中的DGM(L))应该如何获取？
期待您的回答！

关于蛇形卷积代码中维度的问题

您好，在'S3_DSConv_pro.py'中的128行
batch_size, _, width, height = offset.shape
pytorch中tensor变量的维度一般是Batch size、Channel、Height、Width的顺序，请问这里width和height为什么反过来了？

请问关于更换其他数据集有什么要求吗，直接使用常规的语义分割数据集格式可以吗?谢谢

单线插值可以吗

我看论文和代码中，都是分x和y维度进行的，算偏移坐标的时候始终有一个维度没有加上offset，是否意味着直接使用单线性插值就可以，在算offset的卷积输出channel是不是可以不用乘2？

3*3卷积核是否不能偏移

在您的工作中，我尝试调用DSConv时参数使用3*3卷积核，但是我发现它并没有执行可变形卷积的代码（if if_offset内的内容）

关于特征融合和TCLoss

您好，通过阅读您的文章，我对其中关于特征融合和TCLoss的部分非常感兴趣，但在代码中没有找到这两部分，请问这两部分可以开源供我们学习吗。感激不尽。
期待您的回复。

some questions about dataset process

Congratulations! Really nice work!
I have some questions about the ROAD dataset.
"All data enhancements are performed simultaneously during the training process. The random crop size of 256 × 256 is used as a fixed size to feed into the model training."
First, in the training process, the size of images is 256 × 256, the method of resizing is nearest-neighbor interpolation or linear interpolation ?
Second, what about the test process, is it also 256×256?
Third, your final result is a range, such as 78.21±5.77, what's the meaning of it and how do you get this result?

Thanks very much!

OOM！

Thanks for sharing! When I apply DSC3D block in my own net and dataset. I meet the OOM problem, the training patch_size is 128x128x128, the batchsize is 2 and the input_channels is 32. How can i sovle this problem？ Or there are any tricks to reduce the GPU Memroy? BTW, my GPU is GTX 3090 with 24GB memory.

batch_size不等于1时，程序报错

Traceback (most recent call last):
File "d:\projects\medit_seg\DSCNet_3D_opensource\Code\Kipa\DSCNet\S0_Main.py", line 229, in
Process(args)
File "d:\projects\medit_seg\DSCNet_3D_opensource\Code\Kipa\DSCNet\S0_Main.py", line 63, in Process
Train(args)
File "d:\projects\medit_seg\DSCNet_3D_opensource\Code\Kipa\DSCNet\S3_Train_Process.py", line 503, in Train
Train_net(net, args)
File "d:\projects\medit_seg\DSCNet_3D_opensource\Code\Kipa\DSCNet\S3_Train_Process.py", line 129, in Train_net
loss = train_epoch(
File "d:\projects\medit_seg\DSCNet_3D_opensource\Code\Kipa\DSCNet\S3_Train_Process.py", line 50, in train_epoch
output = model(image)
File "C:\Users\Administrator.conda\envs\py39\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "d:\projects\medit_seg\DSCNet_3D_opensource\Code\Kipa\DSCNet\S3_DSCNet.py", line 104, in forward
x_0x_0 = self.conv0z(x)
File "C:\Users\Administrator.conda\envs\py39\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "d:\projects\medit_seg\DSCNet_3D_opensource\Code\Kipa\DSCNet\S3_DSConv.py", line 50, in forward
deformed_feature = dcn.deform_conv(f, offset, self.if_offset)
File "d:\projects\medit_seg\DSCNet_3D_opensource\Code\Kipa\DSCNet\S3_DSConv.py", line 415, in deform_conv
z, y, x = self._coordinate_map_3D(offset, if_offset)
File "d:\projects\medit_seg\DSCNet_3D_opensource\Code\Kipa\DSCNet\S3_DSConv.py", line 279, in _coordinate_map_3D
z_new = z_new.reshape([self.num_batch, self.num_points, 1, 1, self.depth, self.width, self.height])
RuntimeError: shape '[2, 3, 1, 1, 128, 96, 96]' is invalid for input of size 3538944

关于roi_shape的一些问题

请问roi_shape参数的作用是什么

想问一下作者，DSConv的可视化是怎么做的

DL的小白，请教一下代码的复现

我直接在运行的python S0_Main，但是为什么pycharm一直报找不到目录，先是root目录的***，
OSError: [WinError 123] 文件名、目录名或卷标语法不正确。: '**'
我将改成了相对路径，然后下面训练的图片路径也报错了FileNotFoundError: [WinError 3] 系统找不到指定的路径。: './DSCNet_2D_opensource/Data/DRIVE/Image_Tr/'
但我看了下图片确实在这个目录下，这个需要怎么做

KiPA数据集的请教

KiPA数据集的30个测试图片的标签从哪里获取呀？

TCLoss

文章提到的TCLoss的源码请问还没有上传吗？

寻求帮助：将DSConv插入YOLOv5

@YaoleiQi 您好，十分感谢你们的研究工作！我想把DSConv插入YOLOv5，但遇到了一些问题，想寻求一下帮助。

（1）将DSConv的脚本复制到YOLOv5的common.py中，如下：
复制的地址：
https://github.com/YaoleiQi/DSCNet/blob/main/DSCNet_2D_opensource/Code/DRIVE/DSCNet/S3_DSConv.py
加入DSConv后的common.py：
comman-dsconv.txt

（2）在YOLOv5的yolo.py中加入DSConv

        if m in (Conv, GhostConv,DSConv, Bottleneck):
            c1, c2 = ch[f], args[0]
            if c2 != no:  # if not output
                c2 = make_divisible(c2 * gw, 8)

            args = [c1, c2, *args[1:]]

（3）在yolov5.yaml加入DSConv

# Parameters
nc: 11  # number of classes
depth_multiple: 1.0  # model depth multiple
width_multiple: 1.0  # layer channel multiple
anchors:
  - [10,13, 16,30, 33,23]  # P3/8
  - [30,61, 62,45, 59,119]  # P4/16
  - [116,90, 156,198, 373,326]  # P5/32

# YOLOv5 v6.0 backbone
backbone:
  # [from, number, module, args]
  [[-1, 1, Conv, [64, 6, 2, 2]],  # 0-P1/2
   [-1, 1, Conv, [128, 3, 2]],  # 1-P2/4
   [-1, 3, C3, [128]],
   [-1, 1, Conv, [256, 3, 2]],  # 3-P3/8
   [-1, 6, C3, [256]],
   [-1, 1, Conv, [512, 3, 2]],  # 5-P4/16
   [-1, 9, C3, [512]],
   [-1, 1, Conv, [1024, 3, 2]],  # 7-P5/32
   [-1, 3, C3, [1024]],
   [-1, 1, SPPF, [1024, 5]],  # 9
  ]

# YOLOv5 v6.0 head
head:
  [[-1, 1, DSConv, [512, 1]],
   [-1, 1, nn.Upsample, [None, 2, 'nearest']],
   [[-1, 6], 1, Concat, [1]],  # cat backbone P4
   [-1, 3, C3, [512, False]],  # 13

   [-1, 1, DSConv, [256, 1]],
   [-1, 1, nn.Upsample, [None, 2, 'nearest']],
   [[-1, 4], 1, Concat, [1]],  # cat backbone P3
   [-1, 3, C3, [256, False]],  # 17 (P3/8-small)

   [-1, 1, Conv, [256, 3, 2]],
   [[-1, 14], 1, Concat, [1]],  # cat head P4
   [-1, 3, C3, [512, False]],  # 20 (P4/16-medium)

   [-1, 1, Conv, [512, 3, 2]],
   [[-1, 10], 1, Concat, [1]],  # cat head P5
   [-1, 3, C3, [1024, False]],  # 23 (P5/32-large)

   [[17, 20, 23], 1, Detect, [nc, anchors]],  # Detect(P3, P4, P5)
  ]

（4）训练：python train.py --cfg yolov5l-dsconv.yaml --device 1
但遇到了如下问题：

  File "F:\User\yolov5-master\models\comman-dsconv.py", line 475, in forward
    x = self.dsc_conv_x(deformed_feature)
  File "F:\ProgramData\Anaconda3\envs\yolov5-master\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "F:\ProgramData\Anaconda3\envs\yolov5\lib\site-packages\torch\nn\modules\conv.py", line 399, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "F:\ProgramData\Anaconda3\envs\yolov5\lib\site-packages\torch\nn\modules\conv.py", line 395, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same

想请教一下该如何解决呢？

Unable to successfully embed U-Net structure --- RuntimeError: Given groups=1, weight of size [6, 1, 3, 3], expected input[4, 5, 6, 7] to have 1 channels, but got 5 channels instead

When I try to use a structure similar to U-Net (for example, the feature map output by conv0 convolution, and then passed through DSConv again), the following error occurs:
Code :
if name == 'main':
device = torch.device("cpu")
A = np.random.rand(4, 5, 6, 7)
A = A.astype(dtype=np.float32)
A = torch.from_numpy(A)
conv0 = DSConv(1,10,3, 1,0,True,device=device)

A = A.to(device)
conv0 = conv0.to(device)
out = conv0(A)

_, inch, _, _ = out.shape
print(in_ch)
conv1 = DSConv(in_ch, 32,3,1,1,True,device=device)

conv1 = conv1.to(device)  
out1 = conv1(out)
print(out1.shape)

ERROR :
Traceback (most recent call last):
File "",, line 314, in
out = conv0(A)
File "", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "",, line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/snake_pro.py", line 76, in forward
offset = self.offset_conv(input)
File "/ules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "nn/modules/conv.py", line 460, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/odules/conv.py", line 456, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Given groups=1, weight of size [6, 1, 3, 3], expected input[4, 5, 6, 7] to have 1 channels, but got 5 channels instead

x_center和x_grid代表什么意思

您好，请问您代码中的x_center和x_grid代表什么意思？

3d data path

hi,

thank you for your great work. I wonder where is the 3d data for the 3d segmentation.
thx again.
look forward to your reply

best,

关于卷积核的一些疑问

“S3_DSConv.py”中介绍的蛇形卷积结构，是否没有对卷积核进行形变，而是利用offset对输入图像进行形变？再利用(n,1)(1,n)卷积核对形变后的图像进行卷积？

请问3D的工作可以上传一下以供参考吗

作者您好，很感谢您出色的工作，但是我只找到了2D相关的代码，请问您可否上传一下3D的代码以便学习，非常感谢！

3D Kipa数据训练显存占用问题和TypeError: only integer tensors of a single element can be converted to an index索引问题

DSCNet/DSCNet_3D_opensource/Code/Kipa/DSCNet/S0_Main.py

Lines 180 to 183 in 0181adf

	Reference: --ROI_shape: (128, 96, 96) 3090's memory occupancy is about 16653 MiB
	"""
	parser.add_argument("--ROI_shape", default=(128, 96, 96), type=int, help="roi size")
	parser.add_argument("--batch_size", default=1, type=int, help="batch size")

注释中说当 ROI_shape: (128, 96, 96) 的时候显存占用了 16653 MiB ，然鹅我 24G 的显卡一直OOM。

ROI 改成 (96,96,96) 依旧跑不起来。

而且 batch_size 已经是 1 了，依旧OOM。

最后换了显卡，跑起来发现竟然用了 37186MiB 显存！！！

dice result in paper

The dice in your paper is the foreground(retina) dice, or the mean value of both the foreground(retina) dice and the background dice?

关于snake conv 划分成x y两个方向的思考

非常漂亮的工作，很有启发
我有2个问题
（1）为什么要拆分成 x y两个维度，分别求offset；为什么不像dcn一样，直接求的xy上面的offset？有相关消融实验验证，或者某种结论支撑这样的操作吗？
（2）offset 分成x和y，如何做两者结合的可视话呢？demo上面的可视化？

thank you for your time！

Question about code - 'base' 'flatten' 'reshape'

首先十分感谢您所开源的这份优秀工作，我在阅读代码时遇到了一些不懂的地方，因此想请教您几个问题。

1. 在2D S3_DSConv.py 289行到298行出现的'base'：
   我理解的base用于对应batch中的不同图像，但是在代码中加入base后紧接着又减去了base，我认为这会导致后续的value_a0 value_a1 value_c0 value_c1仅对应batch中第0个图像，感觉会影响双线性插值的结果。

2. 在2D S3_DSConv.py 270-272行代码的逻辑'：
   在这两行代码中，首先将输入特征flatten，接着又reshape到原维度，我不理解这样做的原因。

3. 在2D S3_DSConv.py 225-227行代码计算offset时：
   循环似乎没有处理到最外层的卷积核offset。

First of all thank you very much for this excellent work that you have open sourced, I am reading through the code and have come across something I don't understand, so I want to ask you a few questions.

1. In 2D S3_DSConv.py lines 289 to 298 'base' appears:
   I understand that 'base' is used to correspond to the different images in single batch, but by adding base and then subtracting it immediately after in the code, I think this causes the subsequent value_a0 value_a1 value_c0 value_c1 to correspond only to the 0th image in a batch, which I feel affects the results of the bilinear interpolation.

2. Logic in 2D S3_DSConv.py lines 270-272 of code':
   In these two lines of code, the input features are first flatten and then reshape to the original dimension, I don't understand the reason for this.

3. In 2D S3_DSConv.py lines 225-227 the code calculates the offset:
   The loop doesn't seem to process to the outermost convolution kernel offset.

期待您的回复。

Look forward to see the code！

关于特征融合部分一些困惑

作者您好！我是一名初学者，在阅读代码的过程中没有找到关于特征融合的部分，请问这部分以后会开源吗？
以及在阅读paper过程中对这随机丢弃过程有一些困惑：
（1）我的理解是在Figure 4中是表示随机丢弃x或y方向特征中的一个，公式中表示随机丢弃一组由同一个offset得到的x和y方向的特征，图和公式我理解得不一致，应该是如何丢弃的呢？
（2）特征融合是进行多次DSConv然后得到多个offset，进而得到多个特征来进行融合的吗？offset每次是否都不一样
由于我刚接触图像分割，可能上述问题有很幼稚的地方，望见谅。期待您的回复！

Can I replace the 3*3 convolution with DSConv?

You've done such a great job that you've attracted a large number of enthusiasts like me to your work!
I would like to know what to do if I use DSConv to replace the 3*3 convolution.
`class DSConv(nn.Layer):

def __init__(self, in_ch, out_ch, kernel_size, extend_scope, morph,
             if_offset, device):`

I don't know what to do with the parameters morph, if_offset, and device.
Very much looking forward to your reply!

当DCN的kernel size 为3时，和普通的Deformable Conv是一样的吗？

您好，我尝试了您提供的3D DSCNet，里面DCN_Conv使用的kernel size都是3。
当kernel size为3时，DCN_Conv forward时不会累加前面的offset。

center = int(self.num_points // 2) # self.num_points = kernel_size
z_offset1_new[center] = 0
y_offset1_new[center] = 0
for index in range(1, center): # when the kernel size (num_points) = 3, DSConv is equal to Deformable Conv?
    z_offset1_new[center + index] = z_offset1_new[center + index - 1] + z_offset1[center + index]
    z_offset1_new[center - index] = z_offset1_new[center - index + 1] + z_offset1[center - index]
    y_offset1_new[center + index] = y_offset1_new[center + index - 1] + y_offset1[center + index]
    y_offset1_new[center - index] = y_offset1_new[center - index + 1] + y_offset1[center - index]

所以kernel size 为3时和普通DCNet是一样的吗？

Multi-visual Feature Fusion

Could you please let me know if the code you have open-sourced utilizes the multi-visual feature fusion strategy mentioned in the paper?

在paddlepaddle中使用DSConv

您好，首先非常感谢您做出的贡献！
我将DSConv和DSC类单独拿了出来，想在paddlepaddle中使用DSConv来替换普通3*3卷积，但是在初始化中的参数：
`class DSConv(nn.Layer):

def __init__(self, in_ch, out_ch, kernel_size, extend_scope, morph,
             if_offset, device):`

但是device等参数，我不知道该怎么用。
非常期待您能为我解惑。
感谢！

当蛇形卷积插入自己model中遇到的问题

当我将蛇形卷积替换普通卷积的时候遇到了下面的错误：
Traceback (most recent call last):
File "E:\工作点2\train.py", line 332, in
main(args, config)
File "E:\工作点2\train.py", line 107, in main
loss_epoch_train, psnr_epoch_train, ssim_epoch_train = train(train_loader, device, net, criterion, optimizer, logger)
File "E:\工作点2\train.py", line 211, in train
loss_total.backward()
File "D:\Anaconda3\envs\pytorch_gpu\lib\site-packages\torch_tensor.py", line 488, in backward
self, gradient, retain_graph, create_graph, inputs=inputs
File "D:\Anaconda3\envs\pytorch_gpu\lib\site-packages\torch\autograd_init_.py", line 199, in backward
allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1280, 64, 5, 16]], which is output 0 of ReluBackward0, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

想问一下，这个需要怎么修改呢？

蛇形卷积适用情景

请问一下，这样的卷积核是不是更适用在线状图的特征上？比如血管，裂纹，道路等。对于其它不是由线状特征决定的情况，比如人脸，物品分类什么的，它可能会有更多误检的可能？

'if_offset' and 'device'？

1. Traceback (most recent call last):
   File "E:\yolov5-7.0\models\yolo.py", line 386, in
   model = Model(opt.cfg).to(device)
   File "E:\yolov5-7.0\models\yolo.py", line 189, in init
   self.model, self.save = parse_model(deepcopy(self.yaml), ch=[ch]) # model, savelist
   File "E:\yolov5-7.0\models\yolo.py", line 358, in parse_model
   m_ = nn.Sequential(*(m(*args) for _ in range(n))) if n > 1 else m(*args) # module
   File "E:\yolov5-7.0\models\common.py", line 1861, in init
   self.conv = DSConv(c1, c2, k, s, autopad(k, p, d))
   TypeError: init() missing 2 required positional arguments: 'if_offset' and 'device'

2. 添加后

3. File "E:\yolov5-7.0\models\common.py", line 1861, in init
   self.conv = DSConv(c1, c2, k, s, autopad(k, p, d), if_offset, device)
   NameError: name 'if_offset' is not defined

tensor的变化过程的ppt

您好，关于您代码中提到的tensor变化的ppt方便开源一下吗？我想知道程序中tensor的变化的过程对应的图示。

Can you share more training details?

Thanks a lot for sharing your code!
Your conference paper shows that the dice coefficient of the baseline models and your proposed model both reached approximately 0.8. However, when I conduct the experiment, my model doesn't perform as well as yours. Would you like to share more training details? For example, did you use any pretrain weight models?

想知道计算cldice的代码在哪里

想知道论文中的图是用什么软件画的，谢谢！

Why is your 2D prediction image not a binary image?

About Visualization Issues

DSCNet is a very impressive work, and I'm very interested in the visualization of fig8 in the supplementary materials of the paper. Could you please share the visualization code if possible?

DSConv_pro

DSConv_pro是dsconv的升级版吗？升级了那些内容？

想问一下betti number error的代码可以分享一下吗

3D version of DSConv

Thanks for sharing your great work! However I don't really understand why channel of 3d offset is "3 * 2 * kernel_size" instead of "3 * kernel_size"? I have noticed that in function _coordinate_map_3D(), offset is first divided into two groups and for two different flatten direction(morph=0,1 e.g.) involving the same offset direction(z e.g.), two offsets are used independently which seems to be the reason of having a dimension of "3 * 2 * kernel_size", but still I don't get the motivation of this. Why do they need to be distinguished?
By the way, I'm looking forward to pro version of 3d DSConv!

请问TCLoss代码什么时候上传

没有在code里找到

3D版本的DSCconv_pro问题

您好，对比了您在2D版本中提供的DSCconv_pro与之前的DSCconv的计算消耗，发现有很大改善，想问一下有没有3D版本的改写可以提供呢？

wish a pretrained model weight

3D复现二分类任务问题

尊敬的作者和各位朋友，在三维项目中，已开源的代码中进行的是一项三分类任务，我想在二分类任务上进行尝试。
但经过我的一些修改，代码虽然正常运行，训练过程的loss也持续下降至0.005左右，但预测中输出结果始终为全0图，也就是说每一个像素点预测均为背景。经过过程debug，发现predict全图中标签为0概率均在0.9以上，标签为1概率0.0几，是否有类似的情况发生？

我在思考是否是我的修改出了问题。修改如下：
修改为二分类：
（1）num_classes=2
（2）S3_DataLoader中，# Only focus on vessels ...以下的部分修改为label_trans = to_categorical(label_trans[0], 2)
（3）S3_Train_Process中，修改了Dice，但应该不是dice部分的问题，因为它不影响预测过程
请问是否还有其他地方需要修改？十分感谢！

中文翻译版乱码

您好，中文翻译版本打开时乱码，感谢！

关于ROADS数据集上的训练与评估

作者您好，请问是否有在ROADS数据集上相关的训练以及评估代码，目前只开源了DRIVE数据集相关训练。如果能够提供ROADS数据集上的相关工作代码，不胜感激！

How to get 729 points in Figure 7

The title of Figure 7 in the article says that you superimpose 3 layers with a total of 729 points (red) on each image to show the convolution kernel’s position and shape for a given point (yellow). I want to know why it's 729 points . I see that the horizontal DSC convolution and vertical DSC convolution are done 7 times each in S3_DSCNet.py, and the default size of kernel_size is 9. Can you explain how the 729 points are obtained?

在3D数据集上定义DSCNet的参数

您好！请问您方便分享一下在3d数据集上定义DSCNet时的参数吗？（如下图）作为参考我想在其他的数据集上也尝试一下，十分感谢！