got-10k / toolkit Goto Github PK

groundTruth's four numbers refer to tracked object center point coordinates, weight and height?

Thank you for your useful toolkit. It saved me lots of time and effort. Just a quick note: I checked the VOT results of different trackers from the VOT websites, their initial bounding boxes are different if you use the current code. For persistence results, I suggest the following change:

In the vot.py , the function corner2rect should return the following:
np.array([x1, y1, x2-x1, y2-y1]).T instead of np.array([cx - w / 2, cy - h / 2, w, h]).T

groundTruth's four numbers refer to tracked object center point coordinates, width and height?

I have been troubled by the randomness of ImageNetVID, and finally found the reason. In some versions of python, e.g. python 3.5, the cache files in JSON format will be unorderly loaded. This won't happen when we use python 2.7 or python 3.6. This greatly prevented me from reproducing my experiments, since the random order of the training data between runs will lead to different gradients in early epochs when training SiamFC. I suggest we may cache the dataset in a more stable way, e.g. using numpy or cpickle, and using OrderedDict or else.

Details:

import json
seq_dict = json.load(open('imagenet_vid_train/seq_dict.json', 'r'))
seq_names = [n for n in seq_dict]
print(seq_names[0])

gives ILSVRC2015_train_00000000.0 twice when using python 3.6 (completely in order),
gives ILSVRC2015_train_00646001.0 twice when using python 2.7 (not in order but repeatable),
but gives ILSVRC2015_train_00053009.1 and ILSVRC2015_train_00047000.2 when using python 3.5.

Could you explain the values in absence.label, cover.label, and cut_by_image.label

Hi, I encountered some wired errors.
1st, when I unzip the files from BaiduYun, I only got 335 archives that doesn't match the numbers in list.txt file.
Then I create a new list file with my 335 folders and continue my training, the error assert len(img_files) == len(anno) in got10k.py, line 66 came out.
Can anyone help me?Thanks.

Thanks for your beautiful work. Testing on VOT now is quite easy and we no longer need to deal with those bugs in code of vot.
Just open this issue to ask, do you have any plans to add EAO calculation in the short future? Also the robustness score may also need to be adjusted.

I have tried to use the results generated by your code, only ploting EAO ranking using official version. However, lots of bugs appeared.

你好，我想请问一下用这个接口的VOT18的评测得到的结果，在baseline 的EAO为0.24+，可是实时的上面才0.20或者0.19，看了一下帧率，40多帧每秒，是实时的算法，结果应该差不太多啊，baseline和实时的EAO得分。谢谢。

我在leaderboard上提交了got10k测试集上的跟踪结果，然而过了20分钟，依然没有任何反馈，包括评估结果或者提交错误。

Hi, thanks for your great datasets.

In OTB, Fast Motion is defined as the motion of the ground truth is larger than tm pixels (tm=20).
In GOT-10k, you provide the motion_adverb in meta_info.ini. So how do you define whether the target is fast or slow? Is it same with OTB？

Thanks！

I've provided email address([email protected]), however, I've never received the download links. I wanna download test data only.

Hi, when I run the tracking method, I find that some frames in the folders of GOT-10k_Train_001636 and GOT-10k_Train_001638 are incorrect, for example, the 43 ~ 53 frames in 001636, the 25 ~ 33 frames in 0001638. Could you please provide the correct frames of these sequences?

Hello, i installed GOT-10k dataset and the toolkit and save it to the my drive. I run : pip install -r requirements.txt and move got10k folder to my current direcory. But i tried to loop over GOT-10k dataset, i get an error like : Dataset not found or corrupted. I checked my root_dir (data/GOT-10k) and it is correct. What can cause this error ?
The script i try to run :

from PIL import Image
from got10k.datasets import GOT10k
from got10k.utils.viz import show_frame

dataset = GOT10k(root_dir='data/GOT-10k', subset='train')
img_file, anno = dataset[10]

for s, (img_files, anno) in enumerate(dataset):
seq_name = dataset.seq_names[s]
print('Sequence:', seq_name)

for f, img_file in enumerate(img_files):
    image = Image.open(img_file)
    show_frame(image, anno[f, :])

Python version at the Colab i use is 3.6.5
I would be grateful if you help me about that.

Hi, I checked the bounding boxes on each frame, and find some of them are outside of the image region. For example, in the video GOT-10k_Train_000399, at frame 66, the image size is (w=1920, h=1080), but the bounding box is [570. 477. 831. 607.], so that ymax = 477+607 = 1084 > 1080.

Also for the case that object moves outside of image, the bounding boxes are mostly annotated as [-1, -1, -1, -1], but for same cases, the bounding boxes are annotated as 0 in width or height. It would be nice if the bounding boxes are annotated consistently.

Can't access the got10k website today, http://got-10k.aitestunion.com/
Is this temporary？

Does the target exist in every frame of video, and what is the label if the target is occluded or out of view?

hi：
where can I get the fixed annotations and data.

As the title said, there is a compilation error in the file got10k/datasets/lasot.py. The code is self._check_integrity(root_dir, subset) at the line # 31, passing 2 parameters into the function self._check_integrity. However, it just receives one argument in its definition. So, a compilation error happened when I run my code.
The error info is as the following.

Traceback (most recent call last):
  File "train.py", line 18, in <module>
    main()
  File "train.py", line 14, in main
    tracker.train1(data_path, net_storage_dir)
  File ".../.../xxx.py", line 177, in train1
    LaSOT(os.path.join(data_path, 'LaSOT/LaSOTBenchmark'), subset='train')
  File ".../.../anaconda3/lib/python3.6/site-packages/got10k/datasets/lasot.py", line 31, in __init__
    self._check_integrity(root_dir, subset)
TypeError: _check_integrity() takes 2 positional arguments but 3 were given

你好，在项目主页上提交了下载的邮箱，可是没有收到下载数据集的链接，无法下载。这是邮箱，非常感谢。[email protected] 。
用来测试的。

The validation set sample, GOT-10k_Val_000136 changes frame resolution between frames 00000111.jpg -> 00000112.jpg (1920 x 1080 --> 1280 x 720).

Was this intended?

When I try to download the GOT-10K from the website Downloads - GOT-10k, I have not receive the dataset links in my email even now.
Can anyone share me the dataset links? Thanks!

    def _corner2rect(self, corners, center=False):
        cx = np.mean(corners[:, 0::2], axis=1)
        cy = np.mean(corners[:, 1::2], axis=1)

        x1 = np.min(corners[:, 0::2], axis=1)
        x2 = np.max(corners[:, 0::2], axis=1)
        y1 = np.min(corners[:, 1::2], axis=1)
        y2 = np.max(corners[:, 1::2], axis=1)

        area1 = np.linalg.norm(corners[:, 0:2] - corners[:, 2:4], axis=1) * \
            np.linalg.norm(corners[:, 2:4] - corners[:, 4:6], axis=1)
        area2 = (x2 - x1) * (y2 - y1)
        scale = np.sqrt(area1 / area2)
        w = scale * (x2 - x1) + 1  # <-- This doesn't look right.
        h = scale * (y2 - y1) + 1

This actually mimics an old version of VOT that had did not have REGION_LEGACY_RASTERIZATION. In any case, if corners is already axis aligned then in the above code scale = 1.0 and then width and height have an extra pixel. It's a small error, but I think this is a bug.

Additional question 1: Why are you selecting a bounding box with the same area as the oriented bounding box? Does this maximize iou in some meaningful / principled way? Or is this just a heuristic?

Additional question 2: VOT + TRAX does not do this conversion if a Rectangular region is requested. VOT does a vanilla min and max over the input vertices. If you want to be consistent with the VOT toolkit maybe the above implementation should be opt-in to avoid confusion?

I found that there are 4151 frames in Nfs/dog_2/240/dog_2 but the max name of pictures is 04745.jpg. And dog_2 of 30fps and dog_2 of 240fps is not the same sequence. So how to deal with this situation?

In line 137 above, why did you subtract -1 to calculate width and height?

Shouldn't it be just anno[:, 2:] -= anno[:, :2] which is usually the standard practice while dealing with Pascal VOC format [xmin, ymin, xmax, ymax] or anno[:, 2:] -= anno[:, :2] + 1 if you wish to count the number of pixels in the bounding box?

Please let me know if I am wrong somewhere. :)

After the following process
pip3 install -r requirements.txt
I still encountered the error

ModuleNotFoundError: No module named 'tkinter' 

solved by

sudo apt-get install python3-tk 

and the error

ModuleNotFoundError: No module named 'wget' 

solved by

pip3 install wget --user

So would you mind add these solutions in the requirement.txt so that the aforementioned errors would not be encountered by other users again ? Thanks a lot.

Hi Lianghua,
The visualization method in the experiments is slow. I timed the show_frame function and it took 0.5 seconds to visualize one frame of size 1280*720. I tried to improve the speed but still cannot find any good solutions. Do you know any possible alternative methods or libraries to implement the same function?

Best,
Yiming

Hi, my question is about how OTB experiment results are calculated, i.e. success and precision scores.

In the OTB benchmark, areas under success and precision curves are used, while in your code I see that you take the mean to calculate succ_score. Same with prec_score, I don't understand why do you calculate them like that.

Here is how metrics are calculated in ExperimentOTB:

succ_curve = np.mean(succ_curve, axis=0)
prec_curve = np.mean(prec_curve, axis=0)
succ_score = np.mean(succ_curve)
prec_score = prec_curve[20]

Shouldn't you take AUC instead of the mean value?

Hi,
Thank for your excellent toolkit for object tracking.
I try to run my code on VOT2018_LT dataset, but it can not work. I find a problem that the toolkit can not work when the groundtruth is 'nan'.
The exception message is here:
IllegalArgumentException: Points of LinearRing do not form a closed linestring
Shell is not a LinearRing
I'm glad to receive your respond.

Recently I use got10k.datasets scripts in my project and I am very grateful for that.
But I really advise you to change special seq names in OTB dataset: like Jogging.1, Jogging.2, Skating2.1, Skating2.2 to Jogging_1, Jogging_2, Skating2_1, Skating2_2.
Because many algorithms' codes choose the latter, so I need to rename them in many places, so can you improve it while give a better user experience. Just modify a string:
from
[renamed_seqs.append('%s.%d' % (seq_name, ind))]
to
[renamed_seqs.append('%s_%d' % (seq_name, ind))]

@huanglianghua

Hi, thanks for your toolkit. How to use it to print IOU?

Hi
It seems that there is a potential issue in VOT evaluation or I missed something.
A unsupervised VOT evaluation will require results from experiments='supervised'
could I only run unsupervised evaluation?

ValueError                                Traceback (most recent call last)
<ipython-input-14-d8517b2a4749> in <module>
      2 experiment = ExperimentVOT(root_dir=VOT_DIR, experiments='unsupervised', version=2018)
      3 experiment.run(tracker, visualize=False)
----> 4 experiment.report([tracker.name])

~/.local/lib/python3.6/site-packages/got10k/experiments/vot.py in report(self, tracker_names)

ValueError: could not broadcast input array from shape (0) into shape (0,325)

When I evaluate my tracker on VOT, it gives the result of robustness of 50.53 which is confusing to me because robustness in VOT result should be < 1. Is there any difference of how robustness metric is calculated in GOT10k toolkit

Hi:
Thanks for your awesome project. The server is not available right now. Can you please fix the server? And I want to compare my results with some other trackers. Could you please tell me how to plot curves for multiple trackers in the same panel like what you have done in the paper? Thanks in advance!

Hi there, my qusetion is about the VOT evaluation problem. When I try to run the VOT2019 evaluation, got10k raises an assertion error.

File "/usr/local/anaconda3/lib/python3.6/site-packages/got10k-0.1.3-py3.6.egg/got10k/datasets/vot.py", line 81, in __getitem__                                                           
    assert len(img_files) == len(anno), (len(img_files), len(anno))                         
AssertionError: (0, 100)   

And I guess problem may be caused by got10k/datasets/vot.py, line 78:

img_files = sorted(glob.glob(os.path.join(self.seq_dirs[index], 'color', '*.jpg')))

I also check the former got10k version because I didn't encounter this problem before update, and it looks like most VOT (maybe except RGBT2019) don't need path "/color/" and it causes img_files become empty. Thus I wonder whether there may be a conflict bug after adding the VOTRGB support.

下载完train数据，好像没有list文件

hello, I write my email([email protected]) in http://got-10k.aitestunion.com/downloads_dataset/full_data, but i don't get the response, Could you send a download link to my email : [email protected]?

Hi,
Really thanks for providing this dataset.
During analysis of the bounding boxes in the videos, I find the visible levels in 'cover.label' may have problems in some videos like 'GOT-10k_Train_000385' and 'GOT-10k_Train_000602'.
They are all 0 in the files.
Please correct me if I have wrong understanding!

hi, i have requested the link for dataset multiple times, but i havent recieved it yet. can anyone please forward it to me at [email protected]

vot.py line 112: http://data.votchallenge.net/vot2017/vot2017.zip <-- Returns 404.

Sorry for bothering, after I read the paper in arXiv, I am really interested in this work. I would like to know how soon will this wonderful work accessible to the vision tracking community?

The report gives "success_rate", "speed_fps", "success_score" and "precision_score", but how do these compare to the numbers quoted on this page:

https://github.com/foolwood/DaSiamRPN

They quote (for otb2015) "OP" and "DP".

I think this line should be image = image.convert('RGB') to make sure that gray images are successfully converted. Otherwise the evaluation on OTB will fail.

相对比自己的tracker和其他tracker的结果，但是都需要python3版本的其他tracker。所以想问一下有没有现成的其他tracker的结果，在got10k 和 OTB，UAV等

Hello @huanglianghua ,

Thank you for this amazing toolkit. I'm trying to use UAV123 dataset from your toolkit and ran into the error says that:

Traceback (most recent call last):
  File "run_tracking.py", line 20, in <module>
    ExperimentUAV123('data/UAV123', version='UAV123'),
  File "/home/sibobobo/.local/lib/python3.6/site-packages/got10k/experiments/uav123.py", line 24, in __init__
    self.dataset = UAV123(root_dir, version)
  File "/home/sibobobo/.local/lib/python3.6/site-packages/got10k/datasets/uav123.py", line 29, in __init__
    self._check_integrity(root_dir, version)
  File "/home/sibobobo/.local/lib/python3.6/site-packages/got10k/datasets/uav123.py", line 98, in _check_integrity
    raise Exception('Dataset not found or corrupted.')
Exception: Dataset not found or corrupted.

Is it my code that's incorrect?

Best,
Sibo

Thanks for releasing this great evaluation toolkit.
I observe that this toolkit includes the VOT2018LT benchmark. I can successfully evaluated a tracker on the standard VOT2018 benchmark. However, I fail to evaluate the tracker on long-term benchmark using the same manner. Should I change the codes (e.g., the tracker definition)? Thanks!

e = ExperimentVOT(root_dir,version=2018,experiments='unsupervised')

File "G:\practice\TripleFC\tools\test.py", line 37, in
e.report([tracker.name])

File "E:\anaconda\1\envs\siamfc\lib\site-packages\got10k\experiments\vot.py", line 361, in report
ious[seq_name][:len(seq_ious), :] = seq_ious

ValueError: could not broadcast input array from shape (0) into shape (0,325)

Hello, I obtain different precision and success scores for different runs of the same model. I am assuming that there is some randomness in the experiment otb? However some kind of randomness in the experiment code does not make sense to me. Could someone please clarify this issue?

Hello,
Many thanks to your great work. There is one detail about the SiamFC retraining I want to ask: when you train SiamFC on the training part in GOT-10K, do you reinitialize SiamFC or use its pre-trained model on VID to further perform training on GOT-10K?