ReferencesDataFrame files property about jgit-spark-connector

[ajnavarro]

Hi @sakalouski

You cannot access files anymore. You need to get blobs or tree_entries, depending on what do you want to do.

Don't hesitate to ask if you have some other doubts.

[sakalouski]

@ajnavarro Thanks a lot - it worked - got the access to the file contents.
Do You probably know, how to get git blame data? In other words, I want to get the code of each commit. I could probably do it as difference of previous version and the current one, but do not think that it is the most efficient way.
Another question: searching over the .siva database, I have noticed that spark uses anomalously tiny portion of resources (10% of all the cores), while on .parquet data it uses around 80-90%. As a result, materializing 1 entry from 3 Gb folder (something.filter().take(1)) takes ~10 seconds. Any ideas, how to fix it?

Changing parameters of spark, like core numbers and executor memory does not give anything. Repartitioning does not improve the performance.

Thank You!

[ajnavarro]

Do You probably know, how to get git blame data?

We don't have any specific logic for that. To emulate git blame you should go over a reference history reading the specific file and getting modified lines comparing blob contents and the commit committer name and email (standard git blame is doing basically that).

Another question: searching over the .siva database, I have noticed that spark uses anomalously tiny portion of resources (10% of all the cores), while on .parquet data it uses around 80-90%.

When you are processing Siva files, most of the time it will be reading content from that Siva files. Essentially they are git repositories with packfiles, so data is heavily compressed. Also, we parallelize per repository, that means that your process will be as slow as the biggest repository or siva file. I recommend you to preprocess your data and save it to parquet.

[sakalouski]

Thanks for the detailed answer!

…

On Tue, Sep 18, 2018 at 4:05 PM Antonio Navarro Perez < ***@***.***> wrote: Do You probably know, how to get git blame data? We don't have any specific logic for that. To emulate git blame you should go over a reference history reading the specific file and getting modified lines comparing blob contents and the commit committer name and email (standard git blame is doing basically that). Another question: searching over the .siva database, I have noticed that spark uses anomalously tiny portion of resources (10% of all the cores), while on .parquet data it uses around 80-90%. When you are processing Siva files, most of the time it will be reading content from that Siva files. Essentially they are git repositories with packfiles, so data is heavily compressed. Also, we parallelize per repository, that means that your process will be as slow as the biggest repository or siva file. I recommend you to preprocess your data and save it to parquet. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#424 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AfOlL8Ufpvqy_S87bVN69BRbI7DRboWWks5ucQurgaJpZM4WnUvV> .