pytx fetch error in fetching malware_connection_families about threatexchange

[hammem]

Can you post the ID of the malware sample you were attempting to fetch the families for? It'll be easier for us reproduce the error.

[cybercuffs]

The script first collects the IDs by calling malware object as

"for result in Malware.objects(since=1433167932,until=1433171532, dict_generator=True):"

Values in result['id'] are:
814109225352881
1150492118301411
854234691335667
913965018662359
910996378970454
689470754492122
873962145973716
1081450491870663
838455449543136
829052140506010
908174619229394
798787550237174
1059883600696011
719527378156947

[jessek]

I took a look and those samples don't appear to have any families. Perhaps pytx doesn't handle an empty results set?

[mgoffin]

It's possible, but why would it be detecting a 500 response coming from the server?

[jessek]

Gah, sorry, missed that part. I'm not sure what's happening there. I'm unable to reproduce the problem. From https://developers.facebook.com/tools/explorer/, and attempting to visit, for example, /838455449543136/families, I get an empty "data" block, but no error.

[cybercuffs]

jessek: Here is the how the code looks like https://gist.github.com/cybercuffs/bc8f4b0d776f6a9e7baa. This might help you in reproducing the issue. (I've commented out families and variants pull in the code)

[cybercuffs]

There is another problem in regards to the time it takes to fetch the data.

In the code you see I've commented out the lines that are pulling families and variants. But even when I pull only Malware, Connection_dropped and dropped_by just for an hour duration, it is taking a very long time, almost 4 hours. I did couple of tests and the execution start and stop time is as below.
Test1:
Execution Started: 2015/06/23 21:08:22
Execution Ended: 2015/06/24 00:54:46
The files' size I got at the end of this completion were: Connection_dropped=954K, Connection_dropped_by: 409K, Malware_Dump=2.6M.

Test2:
Execution Started: 2015/06/28 05:42:25
Execution Ended: 2015/06/28 09:24:18
The files' size I got at the end of this completion were: Connection_dropped=955K, Connection_dropped_by: 409K, Malware_Dump=2.6M.

[hammem]

Just pinging on this open issue. Please let us know if things were resolved or not. Thanks!

[jessek]

With no response after three weeks, I am closing out. @cybercuffs , please reopen if you are still having problems!