sampom100 / unusualvolumedetector

Gets the last 5 months of volume history for every ticker, and alerts you when a stock's volume exceeds 10 standard deviations from the mean within the last 3 days

Unusual Volume Detector --- New Website! :

This scans every ticker on the market, gets their last 5 months of volume history, and alerts you when a stock's volume exceeds 10 standard deviations from the mean within the last 3 days. (these numbers are all adjustable). Helps find anomalies in the stock market

Easiest way to see this:

Go to the website

How to run the script:

• Download your favorite Python IDE. (I use VSCode)
• Get my script from GitHub
• Open the script in your IDE and install all required dependancies by typing pip install -r requirements.txt into the IDE's terminal. You can get to the the terminal on VSC by pressing CMD and ` at the same time.
• Run the market_scanner.py and it will print out results into the terminal
• You can also graph any ticker's volume in grapher.py

Controlling the Script

• Line 21 controls the amount of months of historical volume the script gets
• Line 22 controls the amount of days before today that it will alert you
• Line 23 controls the number of standard deviations away from the mean volume
• Line 116, "n-jobs" controls the number of threads the script runs on, which I lowered to avoid new rate limits

Websites

Main Website

alternate website

alternate credit

Donations

If you enjoy my work, please donate here

unusualvolumedetector's Issues

Yahoo Finance Not Recording Ticker History

Missing data from Sep 07 - Sep 14

Possible Suggestion for Faster Runtime

Anyone wanna change it so that it only downloads new data saving the old yahoo finance data? What I mean is storing the mean volume, number of data points, as well as the variance and just updating it depending on the day? This will probably decrease runtime to less than a few seconds. I can try to implement it too if there's enough interest.

UPDATE: I wrote a proof of concept it runs faster amortized. Enough changes needed to be made so I rewrote the main script. Its on my github if anyone is curious. Just a heads up though its very scuffed. File storage is in json so if anyone can fork my code to turn it into csv would be perfect. Runtime is still largely gated by yfinance download speed.

Scanning slows to a crawl

Running on an i7-6850K @ 3.6GHz

Scanning starts out really quickly getting up to around 20% within a minute. After that it starts slowing down a lot taking 10 minutes to get to 36%

Random ChunkedEncodingError on mac [email protected]

It looks like it could be a ChunkedEncodingError originating in yfinance that is not being handled properly. Sometimes this manifests as a JSONDecode error, but is always random.

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/response.py", line 436, in _error_catcher
    yield
  File "/usr/local/lib/python3.8/site-packages/urllib3/response.py", line 766, in read_chunked
    chunk = self._handle_chunk(amt)
  File "/usr/local/lib/python3.8/site-packages/urllib3/response.py", line 719, in _handle_chunk
    returned_chunk = self._fp._safe_read(self.chunk_left)
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/http/client.py", line 614, in _safe_read
    raise IncompleteRead(data, amt-len(data))
http.client.IncompleteRead: IncompleteRead(2776 bytes read, 855 more expected)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 751, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/usr/local/lib/python3.8/site-packages/urllib3/response.py", line 571, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/usr/local/lib/python3.8/site-packages/urllib3/response.py", line 792, in read_chunked
    self._original_response.close()
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.8/site-packages/urllib3/response.py", line 454, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(2776 bytes read, 855 more expected)', IncompleteRead(2776 bytes read, 855 more expected))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/site-packages/multitasking/__init__.py", line 102, in _run_via_pool
    return callee(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/yfinance/multi.py", line 166, in _download_one_threaded
    data = _download_one(ticker, start, end, auto_adjust, back_adjust,
  File "/usr/local/lib/python3.8/site-packages/yfinance/multi.py", line 178, in _download_one
    return Ticker(ticker).history(period=period, interval=interval,
  File "/usr/local/lib/python3.8/site-packages/yfinance/base.py", line 150, in history
    data = _requests.get(url=url, params=params, proxies=proxy)
  File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 685, in send
    r.content
  File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 829, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 754, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(2776 bytes read, 855 more expected)', IncompleteRead(2776 bytes read, 855 more expected))```

Yfinance not found error

Install without any issues using install -r requirements.txt

When running python market_scanner.py I am met with the following error:

Traceback (most recent call last): File "market_scanner.py", line 3, in <module> import yfinance as yf ImportError: No module named yfinance

Running pip show yfinance returns
Name: yfinance Version: 0.1.54 Summary: Yahoo! Finance market data downloader Home-page: https://github.com/ranaroussi/yfinance Author: Ran Aroussi Author-email: [email protected] License: Apache Location: /usr/local/lib/python3.6/site-packages Requires: pandas, numpy, requests, multitasking

tqdm not found

ModuleNotFoundError: No module named 'tqdm'

Didn't print anything

I ran the marketscanner.py but it didn't print anything except the time it took

Text message notification

Will anyone be interested in having this script notify you through text message when a ticker is found? it'll require connecting your own twilio account (free to send messages to your own cellphone).

Time to load > 5 min

Is this thing taking forever to load for anyone else?

Off-by-one bug (tickers delayed by 1 day)

fixed in #41

when you give yfinance a range, it doesn't download the data for the enddate
data = yf.download(ticker, pastDate, currentDate)

Volume detector

Add License

Hello,

Thanks for the great stock tool.

Just wondering what License this code is.

I'm assuming I can copy it and modify it in any way shape or form.

market_scanner.py typo

line 30 should be MONTH_CUTTOFF instead of MONTH_CUTOFF as the variable, although the "CUTTOFF" is a typo itself

alternatively all the CUTTOFF variables should be renamed CUTOFF for spelling aesthetics

not sure if there's any technical problem with not calling that global

Forecasting Add On

Hey! No issue. I'm trying to add the functionality to compute a forecasting projection of any anomalous stocks found.

I plan on using scikit-learn for the all the decision making algos. Currently, I am having an issue that the target array isn't in the right classification type. I'll add a screenshot of the error soon. Might see if weka provides a python library as an alternative to scikit-learn.

If anyone would be interesting in collaborating, let me know!

NA

market_scanner.py problems

I thought I had everything set up correctly but I got these two problems when running market_scanner.py

{
"resource": "/i:/UnusualVolumeDetector-master/UnusualVolumeDetector-master/market_scanner.py",
"owner": "python",
"code": "unused-variable",
"severity": 4,
"message": "Unused variable 'anomalies'",
"source": "pylint",
"startLineNumber": 24,
"startColumn": 9,
"endLineNumber": 24,
"endColumn": 9
}

{
"resource": "/i:/UnusualVolumeDetector-master/UnusualVolumeDetector-master/market_scanner.py",
"owner": "python",
"code": "import-error",
"severity": 8,
"message": "Unable to import 'tqdm'",
"source": "pylint",
"startLineNumber": 10,
"startColumn": 1,
"endLineNumber": 10,
"endColumn": 1
}

Cleanups

Kind of treating this issue ticket as a PR/MR

• Remove find_anomalies and rename find_anomalies_two to find_anomalies
• Instead of the for-loops + append, you might be better served with a list comprehension or a generator. I'm going to try this on my machine, and I'll let you know what I find

The Nasdaqcontroller class appends on the global variable

if the get list method is called twice it will append

fix

exportList[:] = []

before looping

Legacy code left as comments

Git completely nullifies the reason to do this. If you want to show legacy code, create a tag, otherwise you're making files needlessly large with stuff that people probably won't read.

Term 'pip' not recognised

Hey I have absolutely no idea what I'm doing, and when I type pip requirements etc. into the terminal I get:

pip install -r requirements.txt
pip : The term 'pip' is not recognized as the name of a cmdlet, function, script file, or
operable program. Check the spelling of the name, or if a path was included, verify that the
path is correct and try again.
At line:1 char:1

• pip install -r requirements.txt

•   + CategoryInfo          : ObjectNotFound: (pip:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException
  
  

I'm guessing there's something very basic I'm doing wrong here, like running the script properly.

Yahoo Finance link as an option in addition to Robinhood

Hi I was wondering if Yahoo finance link can also be added in addition to Robinhood? for example like below:
AAPL (RH, Y!)

intraday support

yfinance surprisingly supports intraday data (within a 60 day window), I don't think it would be too hard to adapt the scanner script for 15m/30m/60m intervals and have the flask script schedule a thread to update the results list. Alternatively we could probably keep our 5/6 month window and try to extrapolate the intraday volume to a full day's volume

Increased speed on Intel when n_jobs is increased to 2x.

Specifically here:
with parallel_backend('loky', n_jobs=multiprocessing.cpu_count()*2):

Instead of 20-30 It/s as reported by TDQM, I'm seeing 40-50 It/s with spikes to 120it/s

I think this is because my connection is the bottleneck more than my processor, so I have 16 requests at a time waiting, and as they come in, my processor saturates. Do anyone else (Tyler), see this behavior?

Connection refused

Hi there, love the idea of this! I installed all the requirements with the text file using VSCode, but now when I run market_scanner.py I get "connection refused" errors on lines that don't even exist like 117 for example. Any idea why that could be? Thank you!!!

Option to open link with another website like Yahoo Finance

Feature request
It would be nice if there was some kind of slider/dropdown where the user can select if he want's to open the URL from the dashboard with another website for example Yahoo Finance or some other third party site.

Example url Yahoo Finance
https://finance.yahoo.com/quote/TSLA?p=TSLA

upload images

Taking over 24 hrs to run for the first time

Any idea why it could be taking over 24 hrs to execute for the first time? I had no issues pip installing all the requirements. I tried to execute the code but it is painfully slow. It did not error out, I ended the execution myself after I let it run for a day and it didn't finish. I have a fast internet connection, connected via ethernet.

Thanks in advance for your help!

UnusualVolumeDetector

Exclude large float tickers

Is this tool only useful for low float tickers (less than 20M)? If so should we have an exclude_tickers.txt list to have all large float tickers that can be excluded? That way we could have 1k tickers to go through instead of 9k. Considerably saving time. I can work on adding the code to exclude tickers from a file but I don't have a complete list or a way to get them

Feature request: include the % off of standard Dev

It saves an extra step for the user, to find out how much the volume differs from the expectation.