Error on RTCClient access. about rtcclient

[dixudx]

@arun017 From the status code 302, it seems that your rtc server url has been redirected.

However currently in client.py#L79-L83, _allow_redirects will be set to True only when ends_with_jazz=True.

So I would suggest you have a test by setting _allow_redirects=True (just modify client.py) to see what will happen.

[arun017]

On setting ends_with_jazz=True below mentioned exception is reported.
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://https://xxyy.com/ccm/authenticated/

RTC_Clientaccessissue2.txt

[dixudx]

@arun017 Have you cut off the response content. I want to know more from the response content.

[kevinhe2017]

@arun017 Have you solved it? I have the exact same issue.

[dixudx]

@kevinhe2017 Can you try to set _allow_redirects=True in client.py#L79-L83. Just hard code it to True (modify client.py) to see what will happen.

[kevinhe2017]

@dixudx I have set _allow_redirects=True since beginning. Don't know what's really going on. Here are the ouput:
2018-01-28 19:12:58,431 DEBUG client.RTCClient: Get response from https://rtc-ccm-1.int.xxx.com:9443/ccm/authenticated/identity
2018-01-28 19:12:58,431 DEBUG urllib3.connectionpool: Starting new HTTPS connection (1): rtc-ccm-1.int.xxx.com
2018-01-28 19:12:58,914 DEBUG urllib3.connectionpool: https://rtc-ccm-1.int.xxx.com:9443 "GET /ccm/authenticated/identity HTTP/1.1" 302 0
2018-01-28 19:12:58,946 DEBUG urllib3.connectionpool: https://rtc-ccm-1.int.xxx.com:9443 "GET /ccm/auth/authrequired HTTP/1.1" 200 None
2018-01-28 19:12:58,961 DEBUG client.RTCClient: Post a request to https://rtc-ccm-1.int.xxx.com:9443/ccm/authenticated/j_security_check with data: j_username=xxxx&j_password=xxxx and json: None
2018-01-28 19:12:58,961 DEBUG urllib3.connectionpool: Starting new HTTPS connection (1): rtc-ccm-1.int.xxx.com
2018-01-28 19:12:59,134 DEBUG urllib3.connectionpool: https://rtc-ccm-1.int.xxx.com:9443 "POST /ccm/authenticated/j_security_check HTTP/1.1" 302 0
2018-01-28 19:12:59,164 DEBUG urllib3.connectionpool: https://rtc-ccm-1.int.xxx.com:9443 "GET /ccm/authenticated/ HTTP/1.1" 404 None
2018-01-28 19:12:59,164 ERROR client.RTCClient: Failed POST request at https://rtc-ccm-1.int.xxx.com:9443/ccm/authenticated/j_security_check with response: Error 404: Not Found

2018-01-28 19:12:59,164 INFO client.RTCClient: 404
Traceback (most recent call last):
File "C:/Users/xxxx/PycharmProjects/DFSS_Leon/Example.py", line 22, in
myclient = RTCClient(url, username, password, ends_with_jazz=False)
File "C:\Python27\lib\site-packages\rtcclient\client.py", line 66, in init
self.headers = self._get_headers()
File "C:\Python27\lib\site-packages\rtcclient\client.py", line 105, in _get_headers
allow_redirects=_allow_redirects)
File "C:\Python27\lib\site-packages\rtcclient\utils.py", line 22, in wrapper
return func(*args, **kwargs)
File "C:\Python27\lib\site-packages\rtcclient\base.py", line 125, in post
response.raise_for_status()
File "C:\Python27\lib\site-packages\requests\models.py", line 936, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://rtc-ccm-1.int.xxx.com:9443/ccm/authenticated/

[kevinhe2017]

I opened the page https://rtc-ccm-1.int.xxx.com:9443/ccm/authenticated/j_security_check, it gives me the following warning. Is the sever has some special security thing which prevents authentication directly?
"You have followed a direct link to log in to a Jazz server. This page has been presented to ensure that a malicious website cannot use cleverly crafted content to circumvent security. Please log in if you would like to access the server."

[dixudx]

@kevinhe2017 What's your RTC server version?

[kevinhe2017]

RTC 6.0.4

[dixudx]

This library has not been fully tested against 6.0.4 yet.

Seems the login url has slightly got changed. @kevinhe2017 Would you please help fix this? Thanks in advance.

[kevinhe2017]

I found by using Firefox Chrome poster to send just two requests like the following, I can get a correct work item response (Status 200 OK with xml format response content with ID 1643428 in it).

1. POST (to get authenticated):
   In URL field: https://rtc-ccm-1.int.xxx.com:9443/ccm/oslc/workitems/catalog/j_security_check?j_username=myusername&j_password=mypassword

2. GET (to get work item):
   In URL field: https://rtc-ccm-1.int.xxx.com:9443/ccm/oslc/workitems/1643428
   In Headers tab:
   Accept: text/xml
   OSLC-Core-Version: 2.0

I tried to modify your scripts to send those two exact requests. So far, I haven't succeeded. Could you give me some directions or some code so I can use?

[kevinhe2017]

Still can't get authenticated. I'm not sure if the following code is equivalent to the above posting the following working URL:
https://rtc-ccm-1.int.xxx.com:9443/ccm/oslc/workitems/catalog/j_security_check?j_username=myusername&j_password=mypassword

Python code:
credentials = urlencode({"j_username": self.username,
"j_password": self.password})

    resp = self.post(self.url + "/oslc/workitems/catalog/j_security_check",
                     data=credentials,
                     verify=False,
                     headers=_headers,
                     proxies=self.proxies,
                     allow_redirects=True)

[kevinhe2017]

Here is the debug info:

2018-01-29 16:25:22,137 DEBUG client.RTCClient: Get response from https://rtc-ccm-1.int.xxx.com:9443/ccm/authenticated/identity
2018-01-29 16:25:22,137 DEBUG urllib3.connectionpool: Starting new HTTPS connection (1): rtc-ccm-1.int.xxx.com
2018-01-29 16:25:22,510 DEBUG urllib3.connectionpool: https://rtc-ccm-1.int.xxx.com:9443 "GET /ccm/authenticated/identity HTTP/1.1" 302 0
2018-01-29 16:25:22,510 DEBUG urllib3.connectionpool: https://rtc-ccm-1.int.xxx.com:9443 "GET /ccm/auth/authrequired HTTP/1.1" 200 None
2018-01-29 16:25:22,526 DEBUG client.RTCClient: Post a request to https://rtc-ccm-1.int.xxx.com:9443/ccm/oslc/workitems/catalog/j_security_check with data: j_username=myusername&j_password=mypassword and json: None
2018-01-29 16:25:22,526 DEBUG urllib3.connectionpool: Starting new HTTPS connection (1): rtc-ccm-1.int.xxx.com
2018-01-29 16:25:22,667 DEBUG urllib3.connectionpool: https://rtc-ccm-1.int.xxx.com:9443 "POST /ccm/oslc/workitems/catalog/j_security_check HTTP/1.1" 302 0
2018-01-29 16:25:22,697 DEBUG urllib3.connectionpool: https://rtc-ccm-1.int.xxx.com:9443 "GET /ccm/auth/authfailed HTTP/1.1" 200 None

[lvfanqie]

kevinhe2017, are you able to solve your issue wtih authentication? I got the same issue with you.

[kevinhe2017]

@lvfanqie yes, I already solved this issue. Please refer my post "fix issue #103: Authentication not working for RTC 6.0.4 ccm and other issues" under the above "Pull request" tab to find my solution.

[lvfanqie]

@kevinhe2017 how you fix the authentication issue? For your log for issue 103, I could see the response from security check is 200 but not 302.

[dixudx]

@kevinhe2017 Seems you just describe your fix in the PR description. Would you please finalize them into codes? Thanks so much.

[lvfanqie]

for me, the issue is a little bit different. for GET request on /authenticated/identity and POST request on /authenticated/j_security_check, I got HTTP response 302. While I can still get the status from work item. Just wonder how to get rid of error response 302. I tried with Chrome poster using above url. The status is 200.

[kevinhe2017]

@dixudx The directions on your "Creating a pull request" link is very difficult to follow and should be revised. I spent a lot time and finally figured out how to make the changes and submit pull request.

[lvfanqie]

@kevinhe2017 could you let me know how get the fix? I tried to go to error#103 but could not find where to download the fixes.

[kevinhe2017]

@lvfanqie In my experience, there may be many causes. Please refer my two new pull requests to update client.py and base.py. I spent several months, asked many people and searched tons of on-line notes, many try and error, finally figured it out.

In Chrome's poster, enter the following to URL field.
https://rtc-ccm-1.int.ABC.com:9443/ccm/oslc/workitems/catalog/j_security_check?j_username=your_user_name&j_password=your_password

Make sure your authentication url is correct. My url is something like "https://rtc-ccm-1.int.ABC.com:9443/ccm/oslc/workitems/catalog/j_security_check"

Add the following to Headers tab.
Accept: text/xml
OSLC-Core-Version: 2.0

After you sent POST command, the response should be in xml format and status should be 200 OK.

[lvfanqie]

@kevinhe2017 I download your new pythons files and tried it again. But I still get 302 for j_security_check. I tried this url in chrome poster, the status is 200.
https://rtc-ccm2.xxx.com:9443/ccm/web/projects/INFOSW/authenticated/j_security_check?j_username=myID&j_password=mypw

And in client.py, i have following
resp, content = self.http.request(self.url + '/web/projects/INFOSW/authenticated/j_security_check', 'POST', headers=_headers, body=urlencode({'j_username': self.username, 'j_password': self.password}))

By debugging, I got resp status as '302', which is same as before.

[kevinhe2017]

@lvfanqie Your URL doesn't look right. Please ask your RTC administrator to make sure you enter the correct authentication URL. Every organization's authentication URL is unique.

You can check if your URL is correct through Chrome poster. If your URL is correct, you may get response 400 Bad Request. In the response content, it may have something like "You have followed a direct link to log in to a Jazz server. This page has been presented to ensure that a malicious website cannot use cleverly crafted content to circumvent security. Please log in if you would like to access the server." If the response is 200 OK, that means your authentication URL is wrong.

[kevinhe2017]

@lvfanqie There are some mistakes in previous note.
When using chrome poster, after adding the following to Headers tab.
Accept: text/xml
OSLC-Core-Version: 2.0

If entering a correct authentication URL and sending POST command, you may get response 400 Bad Request with note "You have followed a direct link ...". If entering a wrong authentication URL and sending POST command, you may get response 200 OK with note "Javascript is either disabled or not available in your Browser".

[lvfanqie]

@kevinhe2017 I checked with IT and the authentication link is https://rtc-ccm2.xx.com:9443/ccm/authenticated/j_security_check
As I post the request on this link in chrome poster, i got response of 400 band request as you said. But put this in the client.py, I still get response of 302.

[lvfanqie]

@kevinhe2017 I did add header as you said, so chrome poster get 400 response ,while the script get 302 response.

[kevinhe2017]

@lvfanqie Ok, looks like the wrong authentication URL issue is fixed. You can try the following further changes also:

1. In your example Python script, use the following url assignment:
   # Set up the URL for querying the work tickets in RTC
   url = "https://rtc-ccm2.xxx.com:9443/ccm"
   # Initialize and authenticate the RTC client
   client = RTCClient(url, username, password, ends_with_jazz=False)
2. You may have issue to transfer username and password correctly. I found the example can't transfer them correctly. To get around this issue, you can try the following code in client.py:
   resp, content = self.http.request(self.url + '/web/j_security_check', 'POST', headers=_headers,
   body=urllib.urlencode({'j_username': your_username or "your_username", 'j_password': your_password or "your_password"}))

It's really difficult to debug the issues. I had many tries and errors and finally made it work. You may have to figure them out by yourself because every server is a little different. I may not see your issues. If you just have a tiny error, it will not work.

[lvfanqie]

@kevinhe2017 sigh. I already tried what you said and still got 302. So the response for security check need to be 200, right?

[kevinhe2017]

@lvfanqie actually, I'm not sure if the response needs to be 200 because it may involve re-direction which will get 3xx response. For me, I used the example first and found it didn't work. Then I wrote my script for my project and removed the logging feature because it's annoying after I fixed the issue. So I can't see my script's response anymore. For your reference, you can review my issue #103 and my pull request "fix issue #103: Authentication not working for RTC 6.0.4 ccm and other issues" to get some ideas. I hope I can debug it with you, but I can't.

[lvfanqie]

@kevinhe2017 Thanks for your kindly help. I figured out how to solve my problem with authentication issue. As what I said, my issue is a little different from yours. I actually is able to get work item status event it return 302 for security check. And what is really wrong is when I input a wrong ID or pw, it doesn't return authentication error. I change the authentication message check part as below and now it works as expected.
authfailed = resp.headers._store['location'][1].find('authfailed') > 0
if authfailed :
raise exception.RTCException("Authentication Failed: "
"Invalid username or password")

[kevinhe2017]

@lvfanqie Glad to know you solved it quickly.