bennythadikaran / eod2 Goto Github PK
View Code? Open in Web Editor NEWA fully automated script to download and update NSE EOD Historical stock, index and delivery data with added features
License: GNU General Public License v3.0
A fully automated script to download and update NSE EOD Historical stock, index and delivery data with added features
License: GNU General Public License v3.0
@BennyThadikaran a feature request
can there be a generic mechanism for saving data that I can extend to write to other datalakes?
Originally posted by @Prady04 in #54 (comment)
Hello Benny,
Today while syncing error is coming.
Error:
saurabhgarg@MacBook-Pro src % python3 --version
Python 3.9.6
saurabhgarg@MacBook-Pro src % python3 init.py
sh: color: command not found
Downloading Files
Starting Data Sync
EOD sync complete
Index sync complete.
Makings adjustments for splits and bonus
Error while making adjustments. KeyError(datetime.datetime(2024, 1, 25, 0, 0))
All adjustments have been discarded.
Rolling back changes from 2024-01-25: /Users/saurabhgarg/shivshakti/eod2/src/eod2_data/daily
Rollback successful
saurabhgarg@MacBook-Pro src %
Can you pls check your repo.
Thanks
Saurabh
A user contacted me on Friday with an error while running EOD2 sync. The error seem to point to duplicate entries in one of the csv files.
I couldn't get the source of the error, but i decided to check my own data for any such issues.
daily
or delivery folders
.DELIV_PER
in some delivery files. Source of the error was in src/defs/defs.py
in the header_text
variable. This defines the column headers for delivery files whenever a new file is created.
I have written a script diagnostics.py
which will look for common errors. I intend to run this weekly before updating data on the repo.
Users can use diagnostics.py to run checks on their own data and report issues.
In rare cases, NSE may delay updating its delivery reports by a day or more. EOD2 will wait for all reports to update before syncing data. See issue #59 for details.
This is inconvenient as latest EOD data is not available because of a single report.
To remedy this issue and prevent future disruptions, here is the final protocol EOD2 will follow going forward:
This feature has been implemented and will be updated on the repo soon. ๐บ
Hello,
Makings adjustments for splits and bonus is not working.
Example 360one.csv. The stock was adjusted on 2nd march 2023 but the same was not synced by the program.
Thanks
Saurabh
Error log
CRITICAL: Unhandled exception
Traceback (most recent call last):
File "D:\python\eod2-main\src\init.py", line 58, in
if not defs.dates.nextDate():
^^^^^^^^^^^^^^^^^^^^^
File "D:\python\eod2-main\src\defs\defs.py", line 95, in nextDate
t_str = local_time.strftime("%-I:%M%p") # 7:00PM
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Invalid format string
CRITICAL: Unhandled exception
Traceback (most recent call last):
File "D:\python\eod2-main\src\init.py", line 58, in
if not defs.dates.nextDate():
^^^^^^^^^^^^^^^^^^^^^
File "D:\python\eod2-main\src\defs\defs.py", line 95, in nextDate
t_str = local_time.strftime("%-I:%M%p") # 7:00PM
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
I have recently created NseIndiaApi and released a pip package NSE pypi
Use this package and replace all NSE related functionality in EOD2 like downloading reports, corporate actions etc.
This way, i can make changes or bug fixes in NseIndiaApi and propagate the changes to other repos with minimal work or duplication.
Thanks Benny. Will go through the same as it seems i need to update python on my system to 3.10 and if do upgrade all my automation will break with brokers so creating virtual env to make eod2 module work.
Thanks
Originally posted by @cotton1234 in #78 (comment)
Hello,
There seems to be another issue related to holiday.json.
{
"tradingDate": "28-Jun-2023",
"weekDay": "Wednesday",
"description": "Bakri Id",
"Sr_no": 10
},
This is the holiday.json which is incorrect as the holiday was changed yesterday by NSE.
I have fixed it by removing the file and it downloaded the same but try to see if you can get the latest file everyday.
Thanks
Saurabh
Hello,
The program was running file till now but today (06/03/2023) the issue is coming in cleanup of the files.
in
Downloading Files
Starting Data Sync
EOD sync complete
Delivery Data updated
Delivery sync complete
Index sync complete.
Makings adjustments for splits and bonus
Cleaning up files
Traceback (most recent call last):
File "/Users/saurabhgarg/shivshakti/eod2/init.py", line 51, in
defs.cleanup([bhavFile])
File "/Users/saurabhgarg/shivshakti/eod2/defs.py", line 553, in cleanup
lastUpdated = datetime.strptime(getLastDate(entry.path), fmt)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/_strptime.py", line 568, in _strptime_datetime
tt, fraction, gmtoff_fraction = _strptime(data_string, format)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/_strptime.py", line 349, in _strptime
raise ValueError("time data %r does not match format %r" %
ValueError: time data '' does not match format '%Y-%m-%d'
Since the lastupdate.txt file get updated in the end after cleanup so running the prog will make it double entries in daily and delivery folder.
Thanks
Saurabh
Hi Benny,
I could see BhavCopy available at NSE site, but script seems to be having some fetching issues, any idea why so?
INFO: Downloading Files
WARNING: NSE file is unavailable or not yet updated.
Thanks,
Rohit
Nse Indices report use the -
character when no data is available. It messes up the data types in pandas.
I have edited defs.py:updateIndexEOD
to replace it with zeros. src/eod2_data/daily/India Vix.csv
has been manually edited to add zeros.
No other files have this issue so far. Update to follow shortly.
I tried many programs in python to convert the downloaded CSV data to metastock or text file. But it's not working. The data is not working with metastock. If you kindly help with that I'll be grateful.
plot --watch hold-w --tf weekly --sma 30 --m-rs --preset-save hold
Preset saved as 'hold'
Traceback (most recent call last):
File "/home/benny/Documents/python/eod2/src/plot.py", line 176, in <module>
plotter.plot(symList[plotter.idx])
File "/home/benny/Documents/python/eod2/src/defs/Plotter.py", line 218, in plot
df = self._prepData(sym)
File "/home/benny/Documents/python/eod2/src/defs/Plotter.py", line 742, in _prepData
df = getDataFrame(
File "/home/benny/Documents/python/eod2/src/defs/utils.py", line 176, in getDataFrame
df = csv_loader(fpath, candle_count, end_date=toDate)
File "/home/benny/Documents/python/eod2/src/defs/utils.py", line 128, in csv_loader
current_dt = get_date(start, chunk)
File "/home/benny/Documents/python/eod2/src/defs/utils.py", line 46, in get_date
date = datetime.strptime(date_str, date_fmt)
File "/usr/lib/python3.10/_strptime.py", line 568, in _strptime_datetime
tt, fraction, gmtoff_fraction = _strptime(data_string, format)
File "/usr/lib/python3.10/_strptime.py", line 349, in _strptime
raise ValueError("time data %r does not match format %r" %
ValueError: time data 'Date' does not match format '%Y-%m-%d'
The error occurs when loading a file using the utils.py:csv_loader
function, as a chunk is read from the start of the file.
Under some circumstances the final chunk may read something like this b'Date,Open,High,Low,Close,Volume,TOT'
Trying to parse the Date
string as a Datetime value results in a ValueError. Issue was fixed by catching the ValueError, appending the chunk and returning the data as a DataFrame.
Updates to be posted soon.
If src/data
folder does not exist, drawing lines on chart and quitting or switching charts will raise a FileNotFoundError
Chart lines are saved on quitting or switching charts. In this case if data
folder does not exists, operation fails.
Error has been fixed, Update to be released shortly.
Kansai Nerolac under symbol code KANSAINER
was not adjusted on 4th July. I could not figure the reason for this.
I ran the adjustment in isolation and found no issues with the code.
The CSV file has been adjusted and added to src/eod2_data/daily
. Repo will be updated shortly
Hi Benny,
Firstly, this is amazing !, I have never seen something prepared so neatly. Kudos !
I tried this few days back and it ran well, however today I tried updating the data and encountered below error, couldn't figure out what's missing as internet connection is running well. I can download other stuff manually. Can you please suggest.
D:\eod2-main\src>init.py
Downloading Files
Download error. Try again later: ChunkedEncodingError(ProtocolError('Connection broken: IncompleteRead(271 bytes read, 138 more expected)', IncompleteRead(271 bytes read, 138 more expected)))
H:\python\eod2\src>py init.py
CRITICAL: Unhandled exception
Traceback (most recent call last):
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 1042, in validate_conn
conn.connect()
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connection.py", line 414, in connect
self.sock = ssl_wrap_socket(
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\ssl.py", line 449, in ssl_wrap_socket
ssl_sock = ssl_wrap_socket_impl(
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\ssl.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 512, in wrap_socket
return self.sslsocket_class._create(
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 1070, in _create
self.do_handshake()
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 1341, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\adapters.py", line 486, in send
resp = conn.urlopen(
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\packages\six.py", line 769, in reraise
raise value.with_traceback(tb)
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 1042, in validate_conn
conn.connect()
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connection.py", line 414, in connect
self.sock = ssl_wrap_socket(
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\ssl.py", line 449, in ssl_wrap_socket
ssl_sock = ssl_wrap_socket_impl(
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\ssl.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 512, in wrap_socket
return self.sslsocket_class._create(
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 1070, in _create
self.do_handshake()
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 1341, in do_handshake
self._sslobj.do_handshake()
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "H:\python\eod2\src\init.py", line 34, in
special_sessions = defs.downloadSpecialSessions()
File "H:\python\eod2\src\defs\defs.py", line 157, in downloadSpecialSessions
res = requests.get(f"{base_url}/main/special_sessions.txt")
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "C:\Users\JashvikKrishna\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\adapters.py", line 501, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))
User @ramkdlr raised a suggestion to merge the OHLCV and delivery files into a single file.
See discussion here
The new columns will be like
Open, High, Low, Close, Volume, TOTAL_TRADES, QTY_PER_TRADE, DLV_QTY
Changes will be reflected in plot.py
and dget.py
(No changes in their interface)
When no user defined watchlists or preset exists, running py plot.py --ls
throws the below error.
`AttributeError: 'Config' object has no attribute 'PRESET'
Issue will be fixed in the upcoming update.
Always reproducible.
There is no issue with the flag set to False.
If I rerun the program, it works fine and will fail for the next date.
Following is the log.
Downloading Files
Starting Data Sync
Converting to AmiBroker format
EOD sync complete.
Index sync complete.
Makings adjustments for splits and bonus
Error while making adjustments. TypeError('cannot do positional indexing on DatetimeIndex with these indexers [[False False False ... False False True]] of type ndarray')
All adjustments have been discarded.
Rolling back changes from 2024-03-22: src\eod2_data\daily
Hello Benny,
I was looking at the eod daily folder and could find only some indices and not all the indices.
where as in the file ind_close_all_01082023.csv there are lot of indices.
Can i define somewhere what all indices i need to sync.
Also i am planning to use the same logic for downloading stock data from BSE as some stocks are only listed on it.
Thanks
Saurabh
Hello Benny,
Getting error today while syncing.
2024-07-18 20:12:30,872 - main - ERROR - Error while making adjustments.
All adjustments have been discarded.
Traceback (most recent call last):
File "/Users/saurabhgarg/shivshakti/eod2/src/init.py", line 123, in
defs.adjustNseStocks()
File "/Users/saurabhgarg/shivshakti/eod2/src/defs/defs.py", line 846, in adjustNseStocks
raise e
File "/Users/saurabhgarg/shivshakti/eod2/src/defs/defs.py", line 804, in adjustNseStocks
commit = makeAdjustment(
^^^^^^^^^^^^^^^
File "/Users/saurabhgarg/shivshakti/eod2/src/defs/defs.py", line 681, in makeAdjustment
last = df.iloc[idx:]
~~~~~~~^^^^^^
File "/Users/saurabhgarg/.venv/lib/python3.12/site-packages/pandas/core/indexing.py", line 1191, in getitem
return self._getitem_axis(maybe_callable, axis=axis)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/saurabhgarg/.venv/lib/python3.12/site-packages/pandas/core/indexing.py", line 1729, in _getitem_axis
return self._get_slice_axis(key, axis=axis)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/saurabhgarg/.venv/lib/python3.12/site-packages/pandas/core/indexing.py", line 1764, in _get_slice_axis
labels._validate_positional_slice(slice_obj)
File "/Users/saurabhgarg/.venv/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 4197, in _validate_positional_slice
self._validate_indexer("positional", key.start, "iloc")
File "/Users/saurabhgarg/.venv/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 6697, in _validate_indexer
self._raise_invalid_indexer(form, key)
File "/Users/saurabhgarg/.venv/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 4301, in _raise_invalid_indexer
raise TypeError(msg)
TypeError: cannot do positional indexing on DatetimeIndex with these indexers [slice(737, 739, None)] of type slice
Thanks
Saurabh
Hi Benny, not sure what's not right.. files are available at NSE page for eod however code throwing back this error.. please have a look whenever convenient. Tx
INFO: Updating NSE EQUITY actions
INFO: Updating NSE SME actions
INFO: Downloading Files
WARNING: NSE file is unavailable or not yet updated.
Hello Benny,
Today session 20 Jan 2024 is not getting downloaded and saying All upto date.
Can you pls check the same.
Thanks
Saurabh
C:\Users\Admin\Documents\Stocks\Code\PythonRS\eod2\src>pip install nse
Collecting nse
Downloading nse-0.2.6.post2-py3-none-any.whl.metadata (2.6 kB)
Requirement already satisfied: mthrottle>=0.0.1 in c:\python311\lib\site-packages (from nse) (0.0.1)
Requirement already satisfied: requests>=2.31.0 in c:\python311\lib\site-packages (from nse) (2.31.0)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\python311\lib\site-packages (from requests>=2.31.0->nse) (3.2.0)
Requirement already satisfied: idna<4,>=2.5 in c:\python311\lib\site-packages (from requests>=2.31.0->nse) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\python311\lib\site-packages (from requests>=2.31.0->nse) (2.0.3)
Requirement already satisfied: certifi>=2017.4.17 in c:\python311\lib\site-packages (from requests>=2.31.0->nse) (2023.5.7)
Downloading nse-0.2.6.post2-py3-none-any.whl (21 kB)
Installing collected packages: nse
Successfully installed nse-0.2.6.post2
C:\Users\Admin\Documents\Stocks\Code\PythonRS\eod2\src>python init.py
NSE Holiday list updated
Downloading Files
RuntimeError('NSE file is unavailable or not yet updated.')
defs.toAmiBrokerFormat uses TOTTRDVAL
instead of TOTTRDQTY
for volume.
Issue has been fixed. Repo will be updated shortly.
ETFs are getting popular and NSE is pushing more of them into the market.
EOD2 currently does not sync ETF stocks. Sync data for existing and future ETF stocks.
Some of recent splits announced are not getting adjusted into EOD2 data. This was due to the regex not detecting the split. This has been fixed. However i need to check for any splits that were missed and action them. I will post an annoucement once the data has been corrected.
Hello Benny,
It just a suggestion, you are filtering the EQ, BE and BZ. It does not have SME based stocks.
Can you include the same in the code.
Thanks
Saurabh
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.