[Refactor][Bitbucket_Server] Speed up PR collector/extractor about incubator-devlake

[sstojak1]

@klesh can you please assign this task to me? I have started to work on it

[Startrekzky]

Hi @sstojak1 , it's a valid refactore, but may I know why the same data keeps been adding to the RAW_PULL_REQUEST_TABLE after each run?

[sstojak1]

Hi @Startrekzky I would say that this is because BB Server API doesn't support date query parameter. So you cannot fetch PRs that were updated/created after last import job.

Because of that Devlake is importing all PRs during each job run (check how BB Cloud is not doing that since date query parameter is supported for that tool).

[klesh]

Based on the information you provided, it seems the collector would benefit from using the simpler ApiCollector since it would purge related records from the raw data table before saving new PR information. This might be more efficient compared to the StatefulApiCollector in this context.

[sstojak1]

@klesh That might work! Is there a way to run Devlake locally in debug mode? I'd like to go through the ApiCollector impl to understand its impact on the rest of the steps for importing BB server data...

[klesh]

Yes, sure. You may follow this guide.

In case you wanna execute specific subtasks, you may go to the backend folder and run sth like the following:

go run plugins/jira/jira.go -c 2 -b 8 -t "extractWorklogs"

remember to change the plugin name and arguments accordingly.

[klesh]

@sstojak1 Thanks for the reminder. Done. 🤝

[sstojak1]

@klesh ready to be reviewed