uzh-rpg / rpg_event_representation_learning Goto Github PK
View Code? Open in Web Editor NEWRepo for learning event representations
License: MIT License
Repo for learning event representations
License: MIT License
Hi,
Thanks for sharing your code. I was studying your paper and trying to write the code of optical flow estimation part, but I failed. Is it convenient for you to share the code of your optical flow estimation part? Looking forward to your reply, and I will appreciate your reply. Thanks a lot!
Hi!
I am really appreciated that your great work including the code and dataset are all released. According to the paper and the released code, I have two little confusion about the implementation.
Hi,
Thanks for so much effort you did. Recently, I'm trying to migrate the released code to the optical flow prediction task, however bench of CUDA errors occurred and hard to solve. I am wondering if it is possible to share the code of optical flow prediction? I appreciate that and look forward to hearing from you.
Best wishes,
Tony
Hi,
Since the training and test just show the accuracy, they did not provide the learned representations of events.
Is there any way to see the learned representations of events?
Another question is about the code:
idx_before_bins = x
+ W * y
+ 0
+ W * H * C * p
+ W * H * C * 2 * b
Here, what's the intention of doing this?
Thanks
Hi!Thanks for your sharing. There was a problem when I was doing the optical flow estimation experiment, that is, an out-of-range occurred at the code :
vox.put_(idx.long(), values, accumulate=True)
I checked this issue and found that it was a problem with the sample I generated from MVSEC dataset. I aggregated the events of the left and right event cameras, and extracted the events every 20ms as a sample. Is this what your article describes? How is the correct sample generated?
Looking forward to your reply! Thanks!
Hi,
Thanks for sharing your reference implementation. I was studying your code and paper and come up with several questions I could not answer when refering back to the paper. I was wondering if I can get your feedback on them. The questions are as follows:
What is the trilinear kernel initialization, and how should one modify it (retrain it) when using another dataset?
What is C
in the voxel dimensions and why is its value set to 9?
It seems, based on line 114 in models.py
that this is the number of bins, which is referred to as B
in the paper. However, line 94 of the same file has the variable B
, but it is not obvious to me how the calculations performed on it represent the bin size.
How the C
value should be changed for event record lengths greated than 100ms? I have a dataset of about 1s record lengths. If C
is the number of bins, it might makes sense that a greater C
is necessary for larger record lengths, right?
What is the purpose of the crop_and_resize_to_resolution
method? Given that it is the input to the classifier, it seems that its only purpose is to satisfy the requirement that the classifier's input should at least be 224x224. Is this correct?
How would you suggest to process an RGB event camera input with the learnable representation network you propose? A way that occurs to me would be to create 3 networks, one for each channel, and combine their outputs in another hidden layer whose weights should be learnt. This is of course not efficient, and does not leverage the trilinear kernel and information structure.
I know this are several questions and hope this is not a burden, I will appreciate your hints and time.
Thanks!
Dear author,
I am interested in your paper and I do not understand what is trilinear kernel
is.
I found the implantation from models.py
where there is one step to init
the kernel.
def trilinear_kernel(self, ts, num_channels):
gt_values = torch.zeros_like(ts)
gt_values[ts > 0] = (1 - (num_channels-1) * ts)[ts > 0]
gt_values[ts < 0] = ((num_channels-1) * ts + 1)[ts < 0]
gt_values[ts < -1.0 / (num_channels-1)] = 0
gt_values[ts > 1.0 / (num_channels-1)] = 0
return gt_values
However, I do not figure out what the meaning of the above code. It should be the gt
values of one trilinear kernel
.
Then, I write one demo to test it. As follows:
import torch
def trilinear_kernel(ts, num_channels):
gt_values = torch.zeros_like(ts)
gt_values[ts > 0] = (1 - (num_channels - 1) * ts)[ts > 0]
gt_values[ts < 0] = ((num_channels - 1) * ts + 1)[ts < 0]
gt_values[ts < -1.0 / (num_channels - 1)] = 0
gt_values[ts > 1.0 / (num_channels - 1)] = 0
return gt_values
ts = torch.rand((1, 2000))
ts.uniform_(-1, 1)
print(trilinear_kernel(ts,9))
The problem is the result is 0
no matter how much times I run.
I am trying to obtain a one-to-one correspondence between the Event data from N-Caltech and the Image data from Caltech. However, it appears that you have renamed files in the validation and testing sets of N-Caltech. To maintain consistency with your data split, could you share the details of your split, such as in a JSON file format? Thank you!
Hi,
Great work! Thanks for sharing your code. Due to the data format is not a standard mini-batch tensor format as BatchID x Channel x Width x Height, I am wondering whether the model can be trained on multi-GPU? If can, how should I implement it?
Looking forward to your reply, any help would be appreciated. Thanks a lot!
Best wishes,
Siqi Li
Hello!
I was testing a trained model, and found that the testing accuracy reported by the code is lower for bigger batch sizes.
For instance:
python testing.py --test $TEST_PATH--checkpoint log/model_best_caltech101.pth --batch_size 12 --num_workers 12
reports:
While:
python testing.py --test $TEST_PATH--checkpoint log/model_best_caltech101.pth --batch_size 48 --num_workers 12
reports:
Any idea why this might be happening?
Edit:
It gets weirder, as running again the evaluation reports different accuracies (all tested with --batch_size 48).
Run 1:
Hi,
I trained the model for 100 epochs with the default parameters provided in your github code.
I observe that my training loss reduces as expected.
However, the validation loss increases with epoch. The validation accuracy is also close to 0.26, much lower than 0.817. I used N-Caltech 101 dataset as mentioned in this repo.
Is there something I am doing wrong ? Please let me know.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.