pithus / bazaar Goto Github PK
View Code? Open in Web Editor NEWAndroid security & privacy analysis for the masses
Home Page: https://beta.pithus.org/
License: GNU Affero General Public License v3.0
Android security & privacy analysis for the masses
Home Page: https://beta.pithus.org/
License: GNU Affero General Public License v3.0
Add the following information in the similar sample block for better pivoting:
From: https://cryptax.medium.com/investigating-android-malware-with-pithus-17d2143cc528
Detect if a sample is packed and unpack it.
note: maybe there is something to do with jadx here.
As a user, I want to be able to save bookmarks of samples. Those bookmarks will be set in the workspace section.
Wouldn't it be nice to have a darker version of Pithus? :)
Plus it's great for reducing eye strain!
From: https://cryptax.medium.com/investigating-android-malware-with-pithus-17d2143cc528
Show the MainActivity
From: https://cryptax.medium.com/investigating-android-malware-with-pithus-17d2143cc528
In Behaviour analysis > Permission analysis, show which section of the code request the permissions instead of the description of the permission.
note: we could keep the description as a beginner friendly enhancement.
Implement public feeds of trends
65 Mb is limiting. for example, impossible to analyse popular apps like twitter
Merci
Most of the items shown on the website are quite technical and not at all easy to understand for the average user (like myself). I mean at least 90% to 95% of the whole content of your website is vague or worse, not at all understandable (at least to me). I am not just throwing a number; I carefully read the whole report of various aspects of the analyses shown by Pithus for a couple of APK files, and thought I have no clue what the website is actually talking about.
So if possible, please create popup (as well as hyperlinked) help messages for almost every item shown on the website, so that a confused user can understand what is going on.
For example, every text item can have a small circle with an "i" in the beginning of it, hovering which will explain that item.
The items that look like buttons (for example, they are encircled within square frames) can themselves trigger popup messages upon hovering.
Or they can be clicked, so that a new browser tab opens, showing details about that particular button.
There can also be some links within the popup help messages for those who want to read much more details about that particular item shown on the website.
Pithus is being recommended on many forums and circles to average or below-the-average (technically illiterate) users.
I think you can (or perhaps should) put a disclaimer on your website about the fact that Pithus is only for professionals and is not supposed to be understandable by the general public, in the first place. That may save many users a lot of headache and confusion.
Besides that, putting popup (or link) help messages wouldn't hurt anybody. It wouldn't reduce the efficiency of your website, or wouldn't change its purpose. It is just a free extra (but very awesome) feature at no cost. I know it will drain your time to implement it, but I think it is worth investing in. This is because such popup help boxes will (1) make your website become understandable to a MUCH broader audience, and therefore, (2) it would help even those few technical superusers fight and investigate APK malware much more effectively because a larger pool of users means much more APK samples submitted to your website.
If you remove a large part of your audience (the general public), you will automatically lose a lot of potential malware that could have been otherwise submitted to Pithus and its professional superusers.
Hi,
I just installed a new test env with docker compose.
The app starts
I can get to the home page
I click on Sign In then Signing with Github and then I get the following error:
Environment:
Request Method: GET
Request URL: http://localhost:8001/accounts/github/login/?process=login
Django Version: 3.0.11
Python Version: 3.8.10
Installed Applications:
['whitenoise.runserver_nostatic',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.messages',
'django.contrib.staticfiles',
'django.contrib.admin',
'django.forms',
'crispy_forms',
'allauth',
'allauth.account',
'allauth.socialaccount',
'allauth.socialaccount.providers.github',
'rest_framework',
'rest_framework.authtoken',
'corsheaders',
'django_q',
'minio_storage',
'bazaar.users.apps.UsersConfig',
'bazaar.core',
'bazaar.front',
'compressor',
'django_extensions']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
'corsheaders.middleware.CorsMiddleware',
'whitenoise.middleware.WhiteNoiseMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.locale.LocaleMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.common.BrokenLinkEmailsMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware']
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/usr/local/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/usr/local/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/python3.8/contextlib.py", line 75, in inner
return func(*args, **kwds)
File "/usr/local/lib/python3.8/site-packages/allauth/socialaccount/providers/oauth2/views.py", line 77, in view
return self.dispatch(request, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/allauth/socialaccount/providers/oauth2/views.py", line 105, in dispatch
app = provider.get_app(self.request)
File "/usr/local/lib/python3.8/site-packages/allauth/socialaccount/providers/base.py", line 49, in get_app
return adapter.get_app(request, self.id)
File "/usr/local/lib/python3.8/site-packages/allauth/socialaccount/adapter.py", line 207, in get_app
app = SocialApp.objects.get_current(provider, request)
File "/usr/local/lib/python3.8/site-packages/allauth/socialaccount/models.py", line 32, in get_current
app = self.get(sites__id=site.id, provider=provider)
File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 415, in get
raise self.model.DoesNotExist(
Exception Type: DoesNotExist at /accounts/github/login/
Exception Value: SocialApp matching query does not exist.
We'd like to run a Pithus instance, but with our own branding ♥
When looking for similar samples under the threat intel tab, the first one is always the current sample shown with 100% similarity to itself; it would be best to hide it
Hello
i connect to Github it's ok. next, i connect with the same browser to https://beta.pithus.org/ and when i upload a package and i get this error:
Why it is ok to log to github and not to https://beta.pithus.org/ please?
Regards
From: https://cryptax.medium.com/investigating-android-malware-with-pithus-17d2143cc528
Reduce false positives, by ignoring 3rd party SDKs. DroidLysis does it apparently.
From: https://cryptax.medium.com/investigating-android-malware-with-pithus-17d2143cc528
In APK Analysis > Receivers, show which intent/actions/filters apply to the receiver.
worker_1 | 15:59:06 [Q] ERROR Failed [whiskey-three-quiet-football] - lost synchronization with server: got message type "5", length 1714972005
worker_1 | : Traceback (most recent call last):
worker_1 | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
worker_1 | return self.cursor.execute(sql, params)
worker_1 | psycopg2.OperationalError: lost synchronization with server: got message type "5", length 1714972005
worker_1 |
worker_1 |
worker_1 | The above exception was the direct cause of the following exception:
worker_1 |
worker_1 | Traceback (most recent call last):
worker_1 | File "/usr/local/lib/python3.8/site-packages/django_q/cluster.py", line 436, in worker
worker_1 | res = f(*task["args"], **task["kwargs"])
worker_1 | File "/app/bazaar/core/tasks.py", line 195, in yara_analysis
worker_1 | for rule in Yara.objects.all():
worker_1 | File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 276, in __iter__
worker_1 | self._fetch_all()
worker_1 | File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 1261, in _fetch_all
worker_1 | self._result_cache = list(self._iterable_class(self))
worker_1 | File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 57, in __iter__
worker_1 | results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
worker_1 | File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1154, in execute_sql
worker_1 | cursor.execute(sql, params)
worker_1 | File "/usr/local/lib/python3.8/site-packages/sentry_sdk/integrations/django/__init__.py", line 489, in execute
worker_1 | return real_execute(self, sql, params)
worker_1 | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 68, in execute
worker_1 | return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
worker_1 | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
worker_1 | return executor(sql, params, many, context)
worker_1 | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
worker_1 | return self.cursor.execute(sql, params)
worker_1 | File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
worker_1 | raise dj_exc_value.with_traceback(traceback) from exc_value
worker_1 | File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
worker_1 | return self.cursor.execute(sql, params)
worker_1 | django.db.utils.OperationalError: lost synchronization with server: got message type "5", length 1714972005
worker_1 |
worker_1 |
worker_1 | 16:01:08 [Q] INFO Process-1:433 stopped doing work
worker_1 | 16:01:08 [Q] INFO Processed [avocado-april-sixteen-michigan]
worker_1 | 16:01:47 [Q] INFO Process-1:413 stopped doing work
worker_1 | 16:01:47 [Q] INFO Processed [nineteen-cold-maine-berlin]
worker_1 | 16:01:54 [Q] INFO Process-1:423 stopped doing work
worker_1 | 16:01:54 [Q] INFO Processed [ceiling-fifteen-don-helium]
worker_1 | PostgreSQL is available
worker_1 | /usr/local/lib/python3.8/site-packages/django_q/conf.py:136: UserWarning: Retry and timeout are misconfigured. Set retry larger than timeout,
worker_1 | failure to do so will cause the tasks to be retriggered before completion.
worker_1 | See https://django-q.readthedocs.io/en/latest/configure.html#retry for details.
worker_1 | warn("""Retry and timeout are misconfigured. Set retry larger than timeout,
I've noticed that in the case of some samples where VT has no detection but there are other detection, the display on threat level is inconsistent between the card and the report. The reproducibility is quite low (need a non detected VT but with malicious things).
My idea is to have the same display of threat level on the card and on the report.
Sample: https://beta.pithus.org/report/3446ccbf96a485c8a95febd5d81d45010f2ac2b6ef48b8531ce07a209ccd4d73
Adding androguard-yara plugin and generating report to feed androguard module.
Since androguard already used in project this will be easy to implement. Will it be usefull than current yara module ? That can be discussed.
Features of androguard-yara is here . Most of features can be search via pithus search section.
Adding following links to be used as a reference point.
http://pavelsimecek.cz/custom-matching-of-koodous-yara-rules/
https://github.com/eybisi/hacky-yara-androguard
Case:
From: https://cryptax.medium.com/investigating-android-malware-with-pithus-17d2143cc528
In Fingerprints > APKiD, show which part of the code implements the features that are detected.
Add an option to compare analysis of two samples based on a diff of the json report or something
add the option to look for samples that have a lot of similar samples
I'm thinking about a way to be able to "follow" public yara rules. I.e. to be able to have a list or matches for a specific rules, not necessarily being able to see it, but I'm for it.
The idea behind it is to follow the activity of specific groups and give a bit sense of community in threat hunting on Pithus.
Private Yara rules will of course not allow such a feature.
From: https://cryptax.medium.com/investigating-android-malware-with-pithus-17d2143cc528
Compute the different names from the AV from VT and show the compiled results.
Hi!
I am currently working on my local DB and noticed that when a samples has only one similar sample, it doesn't show in the threat intel tab "Similar Samples" although it is shown in the index with the nice cards.
This line prevents similar samples when the array in >1. It was probably made to prevent the same sample to be displayed however, when testing on local, I noticed that the main sample is not part of the similar_samples
array.
I'd like confirmation that this behavior can also be observed with the main version of the code, so I can maybe fix that.
Thank you!
On it
From: https://cryptax.medium.com/investigating-android-malware-with-pithus-17d2143cc528
For the similarity, problem is we don't know what is similar between the samples. Suggestion: using Quark's chart radar to compare malware labels.
x@pithus ~/bazaar # docker-compose -f production.yml run --rm django python manage.py update_reports "126a2cab78aca9c78be086eb6d1e363c6927129d62f12b49ae0c15b89107a28c" mbfascqyv
Creating bazaar_django_run ... done
PostgreSQL is available
/usr/local/lib/python3.8/site-packages/django_q/conf.py:136: UserWarning: Retry and timeout are misconfigured. Set retry larger than timeout,
failure to do so will cause the tasks to be retriggered before completion.
See https://django-q.readthedocs.io/en/latest/configure.html#retry for details.
warn("""Retry and timeout are misconfigured. Set retry larger than timeout,
WARNING 2021-06-06 17:25:37,451 base 1 140546873874240 PUT http://elasticsearch:9200/google_play_details [status:400 request:0.003s]
RequestError(400, 'resource_already_exists_exception', 'index [google_play_details/hjyogKnWRzeSavwVbcUa5g] already exists')
WARNING 2021-06-06 17:25:37,453 base 1 140546873874240 PUT http://elasticsearch:9200/dexofuzzy_apk [status:400 request:0.002s]
RequestError(400, 'resource_already_exists_exception', 'index [dexofuzzy_apk/LBmeN4QNTuG9zzSNICS2aQ] already exists')
Start mobsf_analysis for 126a2cab78aca9c78be086eb6d1e363c6927129d62f12b49ae0c15b89107a28c
17:25:38 [Q] INFO Enqueued 1
Start malware_bazaar_analysis for 126a2cab78aca9c78be086eb6d1e363c6927129d62f12b49ae0c15b89107a28c
Start frosting_analysis for 126a2cab78aca9c78be086eb6d1e363c6927129d62f12b49ae0c15b89107a28c
Start vt_analysis for 126a2cab78aca9c78be086eb6d1e363c6927129d62f12b49ae0c15b89107a28c
Start apkid_analysis for 126a2cab78aca9c78be086eb6d1e363c6927129d62f12b49ae0c15b89107a28c
17:25:54 [Q] INFO Enqueued 1
Start ssdeep_analysis for 126a2cab78aca9c78be086eb6d1e363c6927129d62f12b49ae0c15b89107a28c
ERROR 2021-06-06 17:25:54,944 update_reports 1 140546873874240 [Errno 36] File name too long: '/tmp/tmpj6200kcx/META-INF/services/o0ooo0O0o0Oo0OOoO000o0oOO0O000O0OOOo.OooOoO0O0oooooo0OO0ooOOooooo00O00OO0.O0000Oo0oOOO0oooooooo0OO00o0OoO0oOOO.oOoo00OOOoOOOo00O0oO00o0ooOOoooooooO.oOoooOo0OOOoOOOo000O0oo0OO0OOo0OOO0o.oo0O0oOoOOoOoOoOOo0o0o0OOOoo000oOOo0.oOoo00OOOoOOOo00O0oO00o0ooOOoooooooO'
Traceback (most recent call last):
File "/app/bazaar/core/management/commands/update_reports.py", line 46, in _handle_sample
ssdeep_analysis(sha256)
File "/app/bazaar/core/tasks.py", line 404, in ssdeep_analysis
apk.extractall(tmp_dir)
File "/usr/local/lib/python3.8/zipfile.py", line 1647, in extractall
self._extract_member(zipinfo, path, pwd)
File "/usr/local/lib/python3.8/zipfile.py", line 1701, in _extract_member
open(targetpath, "wb") as target:
OSError: [Errno 36] File name too long: '/tmp/tmpj6200kcx/META-INF/services/o0ooo0O0o0Oo0OOoO000o0oOO0O000O0OOOo.OooOoO0O0oooooo0OO0ooOOooooo00O00OO0.O0000Oo0oOOO0oooooooo0OO00o0OoO0oOOO.oOoo00OOOoOOOo00O0oO00o0ooOOoooooooO.oOoooOo0OOOoOOOo000O0oo0OO0OOo0OOO0o.oo0O0oOoOOoOoOoOOo0o0o0OOOoo000oOOo0.oOoo00OOOoOOOo00O0oO00o0ooOOoooooooO'
[ERROR ] root: [Errno 36] File name too long: '/tmp/tmpj6200kcx/META-INF/services/o0ooo0O0o0Oo0OOoO000o0oOO0O000O0OOOo.OooOoO0O0oooooo0OO0ooOOooooo00O00OO0.O0000Oo0oOOO0oooooooo0OO00o0OoO0oOOO.oOoo00OOOoOOOo00O0oO00o0ooOOoooooooO.oOoooOo0OOOoOOOo000O0oo0OO0OOo0OOO0o.oo0O0oOoOOoOoOoOOo0o0o0OOOoo000oOOo0.oOoo00OOOoOOOo00O0oO00o0ooOOoooooooO'
Traceback (most recent call last):
File "/app/bazaar/core/management/commands/update_reports.py", line 46, in _handle_sample
ssdeep_analysis(sha256)
File "/app/bazaar/core/tasks.py", line 404, in ssdeep_analysis
apk.extractall(tmp_dir)
File "/usr/local/lib/python3.8/zipfile.py", line 1647, in extractall
self._extract_member(zipinfo, path, pwd)
File "/usr/local/lib/python3.8/zipfile.py", line 1701, in _extract_member
open(targetpath, "wb") as target:
OSError: [Errno 36] File name too long: '/tmp/tmpj6200kcx/META-INF/services/o0ooo0O0o0Oo0OOoO000o0oOO0O000O0OOOo.OooOoO0O0oooooo0OO0ooOOooooo00O00OO0.O0000Oo0oOOO0oooooooo0OO00o0OoO0oOOO.oOoo00OOOoOOOo00O0oO00o0ooOOoooooooO.oOoooOo0OOOoOOOo000O0oo0OO0OOo0OOO0o.oo0O0oOoOOoOoOoOOo0o0o0OOOoo000oOOo0.oOoo00OOOoOOOo00O0oO00o0ooOOoooooooO'
From: https://cryptax.medium.com/investigating-android-malware-with-pithus-17d2143cc528
In Behaviour analysis > threat analysis, show where in the code Quark locates the behaviour.
Add saved searches as example on the landing page to help users to start up their work with Pithus.
I have done my first analysis locally... very impressive results!!! So thanks for that making this tool available.
I did notice an error though in the GUI it says "An error occurred during the analysis, we have been notified"
I can see that on your beta online page too
But looking at the logs in docker I can't see any error.
So a few questions:
Thanks.
B.
See the Network analysis of https://beta.pithus.org/report/7f4dda8c5131fe4f43112f696c1733bb93d3a9f6bef9e1aa4c87891b49874970
We should probably fix that directly in MobSF.
Looking into it.
When pivoting on dexofuzzy hashes of similar samples, it would be interesting to add the dendrogram tree for genetic analysis of those similar samples to give a better insight of the similarities.
Hi,
I am trying to install your framework but the docker compose installation doesn't seem to work.
I am on a Mac, running Docker version 3.3.2 and compose 1.29.1
It is a brand new install of Docker, with no images, nothing.
I git clone bazaar
I ran the docker-compose -f local.yml build
Everything seems to go ok...
However I noticed it starts with:
redis uses an image, skipping
mailhog uses an image, skipping
elasticsearch uses an image, skipping
mobsf uses an image, skipping
kibana uses an image, skipping
minio uses an image, skipping
Building postgres
....
then it build everything and there are no error.
But if I then go to the URL listed in your instruction (http://localhost:8001) nothing happens...
If I look in the docker dashboard, I can see no apps have started.
But there are 3x images:
If I start them manually, they each exit with a very quick error:
backend_local_worker
Error:
/entrypoint: line 10: POSTGRES_USER: unbound variable
bazaar_local_django
Error:
/entrypoint: line 10: POSTGRES_USER: unbound variable
bazaar_production_posgres
Error:
Database is uninitialized and superuser password is not specified.
You must specify POSTGRES_PASSWORD to a non-empty value for the
superuser. For example, "-e POSTGRES_PASSWORD=password" on "docker run".
You may also use "POSTGRES_HOST_AUTH_METHOD=trust" to allow all
connections without a password. This is not recommended.
See PostgreSQL documentation about "trust":
https://www.postgresql.org/docs/current/auth-trust.html
So it looks I am missing some steps...
I am not too familiar with docker and docker-compose but should I be running some docker-compose up first?
Any help welcome!
Thanks.
Bugs.
Missed a possibility of having various MainActivities: https://beta.pithus.org/report/ae05bbd31820c566543addbb0ddc7b19b05be3c098d0f7aa658ab83d6f6cd5c8
Need to put them in a proper list
Build and extend the hunting section to set up an entire workspace allowing users to track, bookmark sample, generate reports etc.
Workspace:
sections:
From: https://cryptax.medium.com/investigating-android-malware-with-pithus-17d2143cc528
In Control Flow Analysis, show source code that can be parsed and not images. Adding also the possibility to highlight key lines (MobSF does it).
Clean install with docker-compose -f local.yml up
results in following error:
ModuleNotFoundError: no module found "tld"
Similar to VirusTotal, please check the HASH of the to-be-uploaded file, before attempting the upload. It is possible that you have already received a certain APK file before. So the rest of the uploads from many users would be redundant. This will go hard on both the website server as well as those non-first-uploader users (especially those with not-so-fast internet connections who need to wait a lot).
This way, if you have already analyzed that file before, you can simply show the results of that previous analysis each time a new user intends to upload a file, simply by checking the HASH of the to-be-uploaded file (instead of really doing the upload).
I know now that Pithus does check the HASH upon receiving a file on the server, but I really think this HASH-checking should be client-side. It should be checked Not AFTER the file was uploaded by the user, but BEFORE upload has begun (immediately when the user selects a file to upload).
Please have a look at VirusTotal: It downloads a small piece of software to the client's device, and calculates the HASH right on the very client side.
This way, VirusTotal saves a lot of (unnecessary and redundant) internet traffic for both the users and itself. Also, it would be extremely helpful for many users who don't have very fast upload connections.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.