URL with Temporary Redirect about libbigwig

[lidaof]

Tried add this in the curl callback function, but it's not working

   /* example.com is redirected, so we tell libcurl to follow redirection */
   rv = curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);

[dpryan79]

Oddly, libBigWig isn't getting a redirect, it's getting a "401 Unauthorized" message and I haven't a clue why.

[dpryan79]

BTW, you can add a rv = curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L); and then you'll see the underlying curl queries and responses.

[lidaof]

wondering why testBigBed is working:

./testBigBed https://www.encodeproject.org/files/ENCFF001JBR/@@download/ENCFF001JBR.bigBed

[dpryan79]

I suspect I'll have to use a packet sniffer to debug this when I have time, you're certainly correct that this seems odd.

[lidaof]

Thanks for looking into this.

[dpryan79]

Nevermind, apparently it didn't like the default "basic" authentication method. Please try adding in the following:

rv = curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_ANY);

[lidaof]

we are making progress.

got an error in bwStatsFromZoom in the range 27053199-27059606: Resource temporarily unavailable

[dpryan79]

If you enable CURLOPT_VERBOSE, what does the remote server return just before this? I ask mostly since this solved the issue for me.

[lidaof]

yeah, got tons of output....

 ./querybw https://www.encodeproject.org/files/ENCFF188HKC/@@download/ENCFF188HKC.bigWig chr7 27053199 27373567 50 bwOutput.txt 1
* About to connect() to www.encodeproject.org port 443 (#0)
*   Trying 171.67.205.70... * connected
* found 153 certificates in /etc/ssl/certs/ca-certificates.crt
* 	 server certificate verification OK
* 	 common name: *.encodeproject.org (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: RSA
* 	 certificate version: #3
* 	 subject: OU=Domain Control Validated,CN=*.encodeproject.org
* 	 start date: Thu, 14 Aug 2014 14:12:02 GMT
* 	 expire date: Fri, 02 Aug 2019 08:32:08 GMT
* 	 issuer: C=BE,O=GlobalSign nv-sa,CN=AlphaSSL CA - SHA256 - G2
* 	 compression: NULL
* 	 cipher: AES-128-CBC
* 	 MAC: SHA256
> GET /files/ENCFF188HKC/@@download/ENCFF188HKC.bigWig HTTP/1.1
Range: bytes=0-131071
Host: www.encodeproject.org
Accept: */*

< HTTP/1.1 307 Temporary Redirect
< Server: nginx/1.10.1
< Date: Fri, 17 Mar 2017 21:55:32 GMT
< Content-Type: text/html; charset=UTF-8
< Content-Length: 1192
< Connection: keep-alive
< X-Request-URL: https://www.encodeproject.org/files/ENCFF188HKC/@@download/ENCFF188HKC.bigWig
< X-Stats: es_count=1&es_time=6901&queue_begin=1489787732112648&queue_time=917&rss_begin=504201216&rss_change=0&rss_end=504201216&wsgi_begin=1489787732113565&wsgi_end=1489787732134395&wsgi_time=20830
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: GET, HEAD
< Access-Control-Allow-Headers: Accept, Origin, Range, X-Requested-With
< Access-Control-Expose-Headers: Content-Length, Content-Range, Content-Type
< Location: https://download.encodeproject.org/http://encode-files.s3.amazonaws.com/2017/03/05/d7be9e16-e742-4554-9e9b-347834665817/ENCFF188HKC.bigWig?Signature=h7TAvD1OvLohS282XhnUghbM2Ec%3D&Expires=1489917332&AWSAccessKeyId=ASIAIBGS2LKKZLYIOLVA&response-content-disposition=attachment%3B%20filename%3DENCFF188HKC.bigWig&x-amz-security-token=FQoDYXdzEB4aDKo3b1EtKQK0xzlrqCK3A8jMrctRMooXvbFhPZaBtN46iqYhdsIuZVnmCBYphXlMoRFfa%2B7dyVq1ICoFY7d6wrVj2sKHs4VfVMOYlRJOHonPlRj9BvF5DYR8EHZaItBq4ouDlkzOYcrCNbo36uR1IP%2BsDlX8vwqn7hw6ri/wtQYjReE35P8wyG7D3cN4cHZFm2bAmd4xfS6o7vsgh21LfSHjhKIg%2BoQqPoxZwdNB64qlUBrKYo%2BnhDQdKDceMc/0GB9NJqy1U1n0kaXitFHSwg88LzgXR/CY2Eyk/tQVcScceLERWAupB9nLyVpsVH1uSOumFhwcSf1FXEyqFCKWf4jgUqBHJ7T7kfUHmKcLP8VbJgQs0/TB7q8OY0fn7lzugK4kTXkF3GoGdI8aUwNBo2VuA7Z1S0ldUntTMHeh%2Bl9x9nLETbzmVSOB/qIlP%2BZimKVJTMszPoj57cqhxAUd/%2B6xfdiZbjjoh6NSY5rI%2BfMWfTqZS5my6aCIVXTjkg26Bfm4HGl04bNXhJp7uTiIeorwMex0yfSxFJnybunDaLv9fuFmqsOpY7tOXNzmbina6Tp38euVaLnHD/9rAKNpTu6C7pvyUWQohJ6xxgU%3D
< Strict-Transport-Security: max-age=15768000
< 
* Ignoring the response-body
* Connection #0 to host www.encodeproject.org left intact
* Issue another request to this URL: 'https://download.encodeproject.org/http://encode-files.s3.amazonaws.com/2017/03/05/d7be9e16-e742-4554-9e9b-347834665817/ENCFF188HKC.bigWig?Signature=h7TAvD1OvLohS282XhnUghbM2Ec%3D&Expires=1489917332&AWSAccessKeyId=ASIAIBGS2LKKZLYIOLVA&response-content-disposition=attachment%3B%20filename%3DENCFF188HKC.bigWig&x-amz-security-token=FQoDYXdzEB4aDKo3b1EtKQK0xzlrqCK3A8jMrctRMooXvbFhPZaBtN46iqYhdsIuZVnmCBYphXlMoRFfa%2B7dyVq1ICoFY7d6wrVj2sKHs4VfVMOYlRJOHonPlRj9BvF5DYR8EHZaItBq4ouDlkzOYcrCNbo36uR1IP%2BsDlX8vwqn7hw6ri/wtQYjReE35P8wyG7D3cN4cHZFm2bAmd4xfS6o7vsgh21LfSHjhKIg%2BoQqPoxZwdNB64qlUBrKYo%2BnhDQdKDceMc/0GB9NJqy1U1n0kaXitFHSwg88LzgXR/CY2Eyk/tQVcScceLERWAupB9nLyVpsVH1uSOumFhwcSf1FXEyqFCKWf4jgUqBHJ7T7kfUHmKcLP8VbJgQs0/TB7q8OY0fn7lzugK4kTXkF3GoGdI8aUwNBo2VuA7Z1S0ldUntTMHeh%2Bl9x9nLETbzmVSOB/qIlP%2BZimKVJTMszPoj57cqhxAUd/%2B6xfdiZbjjoh6NSY5rI%2BfMWfTqZS5my6aCIVXTjkg26Bfm4HGl04bNXhJp7uTiIeorwMex0yfSxFJnybunDaLv9fuFmqsOpY7tOXNzmbina6Tp38euVaLnHD/9rAKNpTu6C7pvyUWQohJ6xxgU%3D'
* About to connect() to download.encodeproject.org port 443 (#1)
*   Trying 171.67.205.70... * connected
* found 153 certificates in /etc/ssl/certs/ca-certificates.crt
* 	 server certificate verification OK
* 	 common name: *.encodeproject.org (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: RSA
* 	 certificate version: #3
* 	 subject: OU=Domain Control Validated,CN=*.encodeproject.org
* 	 start date: Thu, 14 Aug 2014 14:12:02 GMT
* 	 expire date: Fri, 02 Aug 2019 08:32:08 GMT
* 	 issuer: C=BE,O=GlobalSign nv-sa,CN=AlphaSSL CA - SHA256 - G2
* 	 compression: NULL
* 	 cipher: AES-128-CBC
* 	 MAC: SHA256
> GET /http://encode-files.s3.amazonaws.com/2017/03/05/d7be9e16-e742-4554-9e9b-347834665817/ENCFF188HKC.bigWig?Signature=h7TAvD1OvLohS282XhnUghbM2Ec%3D&Expires=1489917332&AWSAccessKeyId=ASIAIBGS2LKKZLYIOLVA&response-content-disposition=attachment%3B%20filename%3DENCFF188HKC.bigWig&x-amz-security-token=FQoDYXdzEB4aDKo3b1EtKQK0xzlrqCK3A8jMrctRMooXvbFhPZaBtN46iqYhdsIuZVnmCBYphXlMoRFfa%2B7dyVq1ICoFY7d6wrVj2sKHs4VfVMOYlRJOHonPlRj9BvF5DYR8EHZaItBq4ouDlkzOYcrCNbo36uR1IP%2BsDlX8vwqn7hw6ri/wtQYjReE35P8wyG7D3cN4cHZFm2bAmd4xfS6o7vsgh21LfSHjhKIg%2BoQqPoxZwdNB64qlUBrKYo%2BnhDQdKDceMc/0GB9NJqy1U1n0kaXitFHSwg88LzgXR/CY2Eyk/tQVcScceLERWAupB9nLyVpsVH1uSOumFhwcSf1FXEyqFCKWf4jgUqBHJ7T7kfUHmKcLP8VbJgQs0/TB7q8OY0fn7lzugK4kTXkF3GoGdI8aUwNBo2VuA7Z1S0ldUntTMHeh%2Bl9x9nLETbzmVSOB/qIlP%2BZimKVJTMszPoj57cqhxAUd/%2B6xfdiZbjjoh6NSY5rI%2BfMWfTqZS5my6aCIVXTjkg26Bfm4HGl04bNXhJp7uTiIeorwMex0yfSxFJnybunDaLv9fuFmqsOpY7tOXNzmbina6Tp38euVaLnHD/9rAKNpTu6C7pvyUWQohJ6xxgU%3D HTTP/1.1
Range: bytes=0-131071
Host: download.encodeproject.org
Accept: */*

< HTTP/1.1 206 Partial Content
< Server: nginx/1.10.1
< Date: Fri, 17 Mar 2017 21:55:32 GMT
< Content-Type: binary/octet-stream
< Content-Length: 131072
< Connection: keep-alive
< x-amz-id-2: JpA2cB854K6fVz+112Ovylal2+AirfoNifFZOCKt3I3ke/MQ61/SaT8PEOIEqaOfeHL7Um86PHM=
< x-amz-request-id: D1314CFF89768692
< Last-Modified: Sun, 05 Mar 2017 05:25:16 GMT
< ETag: "27a0a165fbaae852aafdb97860bce35a-141"
< x-amz-version-id: nJS91vm0UxKvoK9zaXN6by.IC9VgKm52
< Content-Disposition: attachment; filename=ENCFF188HKC.bigWig
< Accept-Ranges: bytes
< Content-Range: bytes 0-131071/1176156755
< 
* Connection #1 to host download.encodeproject.org left intact
* Re-using existing connection! (#1) with host download.encodeproject.org
* Connected to download.encodeproject.org (171.67.205.70) port 443 (#1)
> GET /http://encode-files.s3.amazonaws.com/2017/03/05/d7be9e16-e742-4554-9e9b-347834665817/ENCFF188HKC.bigWig?Signature=h7TAvD1OvLohS282XhnUghbM2Ec%3D&Expires=1489917332&AWSAccessKeyId=ASIAIBGS2LKKZLYIOLVA&response-content-disposition=attachment%3B%20filename%3DENCFF188HKC.bigWig&x-amz-security-token=FQoDYXdzEB4aDKo3b1EtKQK0xzlrqCK3A8jMrctRMooXvbFhPZaBtN46iqYhdsIuZVnmCBYphXlMoRFfa%2B7dyVq1ICoFY7d6wrVj2sKHs4VfVMOYlRJOHonPlRj9BvF5DYR8EHZaItBq4ouDlkzOYcrCNbo36uR1IP%2BsDlX8vwqn7hw6ri/wtQYjReE35P8wyG7D3cN4cHZFm2bAmd4xfS6o7vsgh21LfSHjhKIg%2BoQqPoxZwdNB64qlUBrKYo%2BnhDQdKDceMc/0GB9NJqy1U1n0kaXitFHSwg88LzgXR/CY2Eyk/tQVcScceLERWAupB9nLyVpsVH1uSOumFhwcSf1FXEyqFCKWf4jgUqBHJ7T7kfUHmKcLP8VbJgQs0/TB7q8OY0fn7lzugK4kTXkF3GoGdI8aUwNBo2VuA7Z1S0ldUntTMHeh%2Bl9x9nLETbzmVSOB/qIlP%2BZimKVJTMszPoj57cqhxAUd/%2B6xfdiZbjjoh6NSY5rI%2BfMWfTqZS5my6aCIVXTjkg26Bfm4HGl04bNXhJp7uTiIeorwMex0yfSxFJnybunDaLv9fuFmqsOpY7tOXNzmbina6Tp38euVaLnHD/9rAKNpTu6C7pvyUWQohJ6xxgU%3D HTTP/1.1
Range: bytes=924251599-924382670
Host: download.encodeproject.org
Accept: */*

< HTTP/1.1 206 Partial Content
< Server: nginx/1.10.1
< Date: Fri, 17 Mar 2017 21:55:33 GMT
< Content-Type: binary/octet-stream
< Content-Length: 131072
< Connection: keep-alive
< x-amz-id-2: cj+yFdyady4mEY6r6M/8yibNhijgyvRkvZXP39pP+P9z2pOCsgIiraBfjO9LiK6QchvgoWuQFdE=
< x-amz-request-id: 0C7711F4484A095C
< Last-Modified: Sun, 05 Mar 2017 05:25:16 GMT
< ETag: "27a0a165fbaae852aafdb97860bce35a-141"
< x-amz-version-id: nJS91vm0UxKvoK9zaXN6by.IC9VgKm52
< Content-Disposition: attachment; filename=ENCFF188HKC.bigWig
< Accept-Ranges: bytes
< Content-Range: bytes 924251599-924382670/1176156755
< 
* Connection #1 to host download.encodeproject.org left intact
* Re-using existing connection! (#1) with host download.encodeproject.org
* Connected to download.encodeproject.org (171.67.205.70) port 443 (#1)
> GET /http://encode-files.s3.amazonaws.com/2017/03/05/d7be9e16-e742-4554-9e9b-347834665817/ENCFF188HKC.bigWig?Signature=h7TAvD1OvLohS282XhnUghbM2Ec%3D&Expires=1489917332&AWSAccessKeyId=ASIAIBGS2LKKZLYIOLVA&response-content-disposition=attachment%3B%20filename%3DENCFF188HKC.bigWig&x-amz-security-token=FQoDYXdzEB4aDKo3b1EtKQK0xzlrqCK3A8jMrctRMooXvbFhPZaBtN46iqYhdsIuZVnmCBYphXlMoRFfa%2B7dyVq1ICoFY7d6wrVj2sKHs4VfVMOYlRJOHonPlRj9BvF5DYR8EHZaItBq4ouDlkzOYcrCNbo36uR1IP%2BsDlX8vwqn7hw6ri/wtQYjReE35P8wyG7D3cN4cHZFm2bAmd4xfS6o7vsgh21LfSHjhKIg%2BoQqPoxZwdNB64qlUBrKYo%2BnhDQdKDceMc/0GB9NJqy1U1n0kaXitFHSwg88LzgXR/CY2Eyk/tQVcScceLERWAupB9nLyVpsVH1uSOumFhwcSf1FXEyqFCKWf4jgUqBHJ7T7kfUHmKcLP8VbJgQs0/TB7q8OY0fn7lzugK4kTXkF3GoGdI8aUwNBo2VuA7Z1S0ldUntTMHeh%2Bl9x9nLETbzmVSOB/qIlP%2BZimKVJTMszPoj57cqhxAUd/%2B6xfdiZbjjoh6NSY5rI%2BfMWfTqZS5my6aCIVXTjkg26Bfm4HGl04bNXhJp7uTiIeorwMex0yfSxFJnybunDaLv9fuFmqsOpY7tOXNzmbina6Tp38euVaLnHD/9rAKNpTu6C7pvyUWQohJ6xxgU%3D HTTP/1.1
Range: bytes=1160431630-1160562701
Host: download.encodeproject.org
Accept: */*

< HTTP/1.1 206 Partial Content
< Server: nginx/1.10.1
< Date: Fri, 17 Mar 2017 21:55:33 GMT
< Content-Type: binary/octet-stream
< Content-Length: 131072
< Connection: keep-alive
< x-amz-id-2: KVfKcEFlAu9gJmS5O0e12KaEolZAFlsy3j/5tGjJhnwRXYIUhQf3EklfMK0H2B92jMvNv00MmwQ=
< x-amz-request-id: 06204334E134A86A
< Last-Modified: Sun, 05 Mar 2017 05:25:16 GMT
< ETag: "27a0a165fbaae852aafdb97860bce35a-141"
< x-amz-version-id: nJS91vm0UxKvoK9zaXN6by.IC9VgKm52
< Content-Disposition: attachment; filename=ENCFF188HKC.bigWig
< Accept-Ranges: bytes
< Content-Range: bytes 1160431630-1160562701/1176156755
< 
* Connection #1 to host download.encodeproject.org left intact
* Re-using existing connection! (#1) with host download.encodeproject.org
* Connected to download.encodeproject.org (171.67.205.70) port 443 (#1)
> GET /http://encode-files.s3.amazonaws.com/2017/03/05/d7be9e16-e742-4554-9e9b-347834665817/ENCFF188HKC.bigWig?Signature=h7TAvD1OvLohS282XhnUghbM2Ec%3D&Expires=1489917332&AWSAccessKeyId=ASIAIBGS2LKKZLYIOLVA&response-content-disposition=attachment%3B%20filename%3DENCFF188HKC.bigWig&x-amz-security-token=FQoDYXdzEB4aDKo3b1EtKQK0xzlrqCK3A8jMrctRMooXvbFhPZaBtN46iqYhdsIuZVnmCBYphXlMoRFfa%2B7dyVq1ICoFY7d6wrVj2sKHs4VfVMOYlRJOHonPlRj9BvF5DYR8EHZaItBq4ouDlkzOYcrCNbo36uR1IP%2BsDlX8vwqn7hw6ri/wtQYjReE35P8wyG7D3cN4cHZFm2bAmd4xfS6o7vsgh21LfSHjhKIg%2BoQqPoxZwdNB64qlUBrKYo%2BnhDQdKDceMc/0GB9NJqy1U1n0kaXitFHSwg88LzgXR/CY2Eyk/tQVcScceLERWAupB9nLyVpsVH1uSOumFhwcSf1FXEyqFCKWf4jgUqBHJ7T7kfUHmKcLP8VbJgQs0/TB7q8OY0fn7lzugK4kTXkF3GoGdI8aUwNBo2VuA7Z1S0ldUntTMHeh%2Bl9x9nLETbzmVSOB/qIlP%2BZimKVJTMszPoj57cqhxAUd/%2B6xfdiZbjjoh6NSY5rI%2BfMWfTqZS5my6aCIVXTjkg26Bfm4HGl04bNXhJp7uTiIeorwMex0yfSxFJnybunDaLv9fuFmqsOpY7tOXNzmbina6Tp38euVaLnHD/9rAKNpTu6C7pvyUWQohJ6xxgU%3D HTTP/1.1
Range: bytes=1141013476-1141144547
Host: download.encodeproject.org
Accept: */*

< HTTP/1.1 206 Partial Content
< Server: nginx/1.10.1
< Date: Fri, 17 Mar 2017 21:55:34 GMT
< Content-Type: binary/octet-stream
< Content-Length: 131072
< Connection: keep-alive
< x-amz-id-2: oNjXzR17jQCdvrrznQ4Mid4aObOokUVp3a/awpyCQFZiiP9esEMhauW6JYMA1d96VQqVZjhWZ+g=
< x-amz-request-id: AC8B8ABBCDBB6905
< Last-Modified: Sun, 05 Mar 2017 05:25:16 GMT
< ETag: "27a0a165fbaae852aafdb97860bce35a-141"
< x-amz-version-id: nJS91vm0UxKvoK9zaXN6by.IC9VgKm52
< Content-Disposition: attachment; filename=ENCFF188HKC.bigWig
< Accept-Ranges: bytes
< Content-Range: bytes 1141013476-1141144547/1176156755
< 
* Connection #1 to host download.encodeproject.org left intact
got an error in bwStatsFromZoom in the range 27053199-27059606: Resource temporarily unavailable

[lidaof]

the testRemote is working after adding:
rv = curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
my problem is now with bwStatsFromZoom

[dpryan79]

@lidaof My guess is that you're not setting the zoom level. I don't export bwStatsFromZoom in the header, so I imagine you're getting some warnings indicating that. If you really really want to directly use that function, then its syntax is bwStatsFromZoom(bw, level, tid, start, end, nBins, type), where bw is the bigWigFile_t pointer, level is an int32_t specifying which of the zoom levels to use, tid is the numeric index of the chromosome/contig, and start/end/nBins/type are as specified for bwStats(). I never expected anyone to directly use that function, so I never documented or exported it.

[lidaof]

Thanks @dpryan79 . is the level parameter be 1 or 0, and 1 for full chromosome data fetch? could the tid be chr1 or chr2?

[dpryan79]

The level parameter is 0 through n-1, indicating the n zoom levels. There is no level to indicate full chromosome, since these zoom levels are intrinsic to how a given bigWig file was made. I'll just point you to the code for bwStats(), which is what I would recommend people use unless they explicitly need exact results and don't care about the extra overhead incurred by bwStatsFromFull().

As an aside, I mention some of the details about what zoom levels in the python wrapper.

[lidaof]

Thanks for the reply @dpryan79. If I understand you correct, are you suggesting me using bwStatsFromZoom instead of bwStats? I am confusing about how different they would be? I think I need also call determineZoomLevel and bwGetTid, basically like copy all lines from bwStats instead of call bwStats directly? Sorry...I am not quite understand bigWig or bigBed actually.

[lidaof]

Closed this as it's indeed working sometimes. Thanks.