Comments (12)
Hi @assaron
I just tried this from a clean environment on my Linux notebook and didn't have any problems.
Do you have /usr/local/bin/ in your PATH? If you run: "hsds-node" on the command line, what happens?
from hsds.
@jreadey Yes, I have it, that's why I don't understand what's going on:
$ which hsds-node
/usr/local/bin/hsds-node
$ hsds-node --node_type=sn
hsds node main for node_type: sn
python version: 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110]
sys path: ['/usr/local/bin', '/usr/lib/python39.zip', '/usr/lib/python3.9', '/usr/lib/python3.9/lib-dynload', '/home/alserg/.local/lib/python3.9/site-packages', '/usr/local/lib/python3.9/dist-packages', '/usr/local/lib/python3.9/dist-packages/requests_unixsocket-0.3.0-py3.9.egg', '/usr/local/lib/python3.9/dist-packages/PyJWT-2.4.0-py3.9.egg', '/usr/local/lib/python3.9/dist-packages/numcodecs-0.10.2-py3.9-linux-x86_64.egg', '/usr/local/lib/python3.9/dist-packages/aiohttp_cors-0.7.0-py3.9.egg', '/usr/local/lib/python3.9/dist-packages/aiohttp-3.8.1-py3.9-linux-x86_64.egg', '/usr/local/lib/python3.9/dist-packages/aiofiles-0.8.0-py3.9.egg', '/usr/local/lib/python3.9/dist-packages/aiobotocore-2.1.0-py3.9.egg', '/usr/local/lib/python3.9/dist-packages/yarl-1.8.1-py3.9-linux-x86_64.egg', '/usr/local/lib/python3.9/dist-packages/multidict-6.0.2-py3.9-linux-x86_64.egg', '/usr/local/lib/python3.9/dist-packages/frozenlist-1.3.1-py3.9-linux-x86_64.egg', '/usr/local/lib/python3.9/dist-packages/charset_normalizer-2.1.0-py3.9.egg', '/usr/local/lib/python3.9/dist-packages/async_timeout-4.0.2-py3.9.egg', '/usr/local/lib/python3.9/dist-packages/aiosignal-1.2.0-py3.9.egg', '/usr/local/lib/python3.9/dist-packages/wrapt-1.14.1-py3.9-linux-x86_64.egg', '/usr/local/lib/python3.9/dist-packages/botocore-1.23.24-py3.9.egg', '/usr/local/lib/python3.9/dist-packages/aioitertools-0.10.0-py3.9.egg', '/usr/local/lib/python3.9/dist-packages/jmespath-0.10.0-py3.9.egg', '/usr/local/lib/python3.9/dist-packages/hsds-0.7.2-py3.9.egg', '/usr/lib/python3/dist-packages']
INFO> Service node initializing
INFO> service node initializing
setLogConfig - level=INFO
INFO> Application baseInit
INFO> using node port: 5101
INFO> setting node_id to: sn-a2a4b
INFO> using bucket: hsdstest
INFO> using node port: 5101
INFO> aws_iam_role set to: hsds_role
INFO> aws_secret_access_key not set
INFO> aws_access_key_id not set
INFO> aws_region set to: us-east-1
INFO> Using metadata memory cache size of: 134217728
INFO> allow_noauth = True
INFO> initUserDB
WARN> No password file, file /config/passwd.txt not found
INFO> user_db initialized: 0 users
INFO> initgroupDB
INFO> No groups file
INFO> group_db initialized: 0 groups
INFO> run_app on port: 5101
======== Running on http://0.0.0.0:5101 ========
(Press CTRL+C to quit)
INFO> health check start
INFO> healthCheck - node_state: WAITING
INFO> register: http://head:5100/register
INFO> register req: http://head:5100/register body: {'id': 'sn-a2a4b', 'port': 5101, 'node_type': 'sn'}
INFO> http_post('http://head:5100/register', 3 bytes)
INFO> Initiating TCPConnector for http://head:5100/register with limit 100 connections
WARN> ClientError for http_post(http://head:5100/register): Cannot connect to host head:5100 ssl:default [Name or service not known]
ERROR> HEAD node seems to be down.
ERROR> Unexpected UnboundLocalError exception in doHealthCheck: local variable 'rsp_json' referenced before assignment
I've added some debug output, and for some reason cmd_path
being checked in hsds_app.py
is /usr/bin/hsds-node
, not /usr/local/bin/hsds-node
.
The debug code:
cmd_path = os.path.join(sys.exec_prefix, "bin")
cmd_path = os.path.join(cmd_path, "hsds-node")
logging.info(f"sys.exec_prefix = {sys.exec_prefix}")
logging.info(f"cmd_path = {cmd_path}")
print(os.path.isfile(cmd_path))
if not os.path.isfile(cmd_path):
# search corresponding location for windows installs
cmd_path = os.path.join(sys.exec_prefix, "Scripts")
cmd_path = os.path.join(cmd_path, "hsds-node-script.py")
if not os.path.isfile(cmd_path):
raise FileNotFoundError("can't find hsds-node executable")
The output of hsds
command:
$ hsds --root_dir /home/alserg/workspace/hsds/data --password_file admin/config/passwd.default --logfile hs.log --socket_dir /tmp/hs --loglevel INFO --config_dir=admin/config --count=4
set logging to: 20
logfile: /home/alserg/workspace/hsds/hs.log
INFO:root:using cmd_dir: /usr/local/bin
INFO:root:sys.exec_prefix = /usr
INFO:root:cmd_path = /usr/bin/hsds-node
False
Traceback (most recent call last):
File "/usr/local/bin/hsds", line 33, in <module>
sys.exit(load_entry_point('hsds==0.7.2', 'console_scripts', 'hsds')())
File "/usr/local/lib/python3.9/dist-packages/hsds-0.7.2-py3.9.egg/hsds/app.py", line 333, in main
app.run()
File "/usr/local/lib/python3.9/dist-packages/hsds-0.7.2-py3.9.egg/hsds/hsds_app.py", line 275, in run
raise FileNotFoundError("can't find hsds-node executable")
FileNotFoundError: can't find hsds-node executable
from hsds.
I have no idea why sys.exec_prefix
for me is /usr
, and not the default (according to documentation) /usr/local
. I'm using system-installed (Debian) python package python3.9, version 3.9.2-1
.
Still, is there a reason why sys.exec_prefix
is checked instead of cmd_dir
?
from hsds.
And I've checked some other Ubuntu-based installations, there for system-wide Python sys.exec_prefix
is /usr/
from hsds.
I've only ever tested the no-docker option with Anaconda python. In Anaconda the sys.exec_prefix points to something like: /home/jreadey/anaconda3/envs/my_conda_env/bin
which is where the hsds executables get dumped.
I guess it would be easy enough to have the code look at /usr/local/bin/ if it doesn't find anything in the exec_prefix directory.
I'll give this a try.
Is there a specific reason you need to run with native python?
from hsds.
No, there is no specific reason why I need to use native python, but it was also a bit unexpected that it doesn't work, while all of the dependencies are installed.
As for the fix, doesn't it make sense just to use cmd_dir
like below? It looks like it fixes the issue for me (at least being able to start the server).
diff --git a/hsds/hsds_app.py b/hsds/hsds_app.py
index 9e4310a..d236bcd 100644
--- a/hsds/hsds_app.py
+++ b/hsds/hsds_app.py
@@ -262,8 +262,7 @@ class HsdsApp:
common_args.append(f"--log_level={self._loglevel}")
py_exe = sys.executable
- cmd_path = os.path.join(sys.exec_prefix, "bin")
- cmd_path = os.path.join(cmd_path, "hsds-node")
+ cmd_path = os.path.join(self._cmd_dir, "hsds-node")
if not os.path.isfile(cmd_path):
# search corresponding location for windows installs
cmd_path = os.path.join(sys.exec_prefix, "Scripts")
from hsds.
Glad you were able to get it working.
I've always been a little leery of changing the system python packages (e.g. what if a different python app requires different package versions?), but I'm fine with making this change. I'd reverse the search order though - look in exec_prefix, then cmd_dir. If I don't have an active conda environment, it won't find anything in exec_prefix. If I am running conda, I'd expect to use the version in the conda path.
from hsds.
If I am running conda, I'd expect to use the version in the conda path.
I thought that was the point of cmd_dir, it tries to find the appropriate location of hsds installation, be it in conda or native, no?
from hsds.
I've tried to install hsds with my change inside a conda environment and it seems to work just fine:
$ hsds --root_dir /home/alserg/workspace/hsds/data --password_file admin/config/passwd.default --logfile hs.log --socket_dir /tmp/hs --loglevel INFO --config_dir=admin/config --count=1
set logging to: 20
logfile: /home/alserg/workspace/hsds/hs.log
INFO:root:using cmd_dir: /home/alserg/miniconda3/envs/hsds/bin
INFO:root:all processes ready!
INFO:root:Ready after: 0.00 s
from hsds.
@assaron - I've updated the unbloundlocal branch to fix this issue and some other updates for running without docker. Can you give it a try?
from hsds.
@jreadey Thanks! It seems to work well! My issue is resolved then
from hsds.
Great! I've merged the changes into master.
from hsds.
Related Issues (20)
- Errors Running TestAll After Install on Windows 10 (http://localhost:5101 and root at C:\hsds_data) HOT 8
- Return group creation properties on GET request to group HOT 2
- Variable Length Strings HOT 4
- Point selections - how to write correct query? HOT 4
- Support for H5D_CONTIGUOUS / H5D_COMPACT chunk layouts HOT 2
- swapped arguments for HTTPRequestEntityTooLarge HOT 3
- Enable query for creation order for links and attributes HOT 1
- Creating Server and Running Tests HOT 2
- Set up a project under Projects and make a plan for the next release.
- [Question] Storage buckets supported by HSDS service HOT 2
- Building HSDS Service with Tencent COS HOT 7
- There is no ETag in the response header HOT 10
- Setting up local HSDS server: error on first runall.bat in Windows HOT 1
- Update setup.py to remove setuptools deprecation HOT 1
- Enable dataset shape extent to be reduced HOT 1
- Support object copy operations
- Allow hdf5:// prefix in domain names HOT 1
- Can't write to H5T_ARRAY datasets
- 'Verbose' parameters for dset reads returns inaccurate until scan HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hsds.