ls only list file names below 1000 about goofys

[kahing]

could you describe your problem in better details? Such as what files are in the bucket and what files do you see from ls? I can ls directories with more than 1000 files fine.

[chuang0721]

OK, thank you for your reply.

I use goofys version 0.0.4. I have 100 thousands files in my swift cluster.

And what I did to mount our swift service(with amazon-s3 compatible API) is:
/root/bin/goofys --endpoint=http://xxx.com bucket /bucket_dir

I used ls /bucket_dir, it returns only 1000 files.

But with s3cmd, I can list all of the files.

Is there anything I did wrong with goofys to get the full file list?

[kahing]

sorry for the late reply, this is a busy couple weeks for me. I don't have a working swift3 setup right now, could you try to get the debug log from goofys?

$  /root/bin/goofys -f --debug_s3 --endpoint=http://xxx.com bucket /bucket_dir >& goofys.log
$ # in another shell
$ ls /bucket_dir

Also, what version of swift3 are you using?

[chuang0721]

Thank you for your reply.

The following is the debug log I got while I ls /bucket_dir.

We have been using a modified version of swift3 based on 1.7.0. We have made a lot modification to it. For now I am not very sure if these modification affects goofys.

But In the log I can see there is a IsTruncated: true attribute present. It seems like the response is correct. But goofys gave me only 1000 files.

Is it possible that goofys does not handle IsTruncated correctly?

2016/03/03 10:29:19.649707 s3.DEBUG DEBUG: Request s3/HeadBucket Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD http://xxx.vip.xxx.com/allfilecontainer HTTP/1.1
Host: xxx.vip.xxx.com
User-Agent: aws-sdk-go/1.1.6 (go1.5.1; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=xxxxx/20160303/us-west-2/s3/aws4_request, SignedHeaders=host;x-amz-date, Signature=xxxxx
X-Amz-Content-Sha256: xxxxx
X-Amz-Date: 20160303T022919Z


-----------------------------------------------------
2016/03/03 10:29:19.656233 s3.DEBUG DEBUG: Response s3/HeadBucket Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 400 Bad Request
Content-Length: 164
Connection: keep-alive
Content-Type: text/xml
Date: Thu, 03 Mar 2016 02:29:19 GMT
Server: Tengine
X-Trans-Id: tx5c5d11c64f1c4cd0acec2-0056d7a17f


-----------------------------------------------------
2016/03/03 10:29:19.656331 s3.DEBUG DEBUG: Validate Response s3/HeadBucket failed, not retrying, error :
status code: 400, request id:
2016/03/03 10:29:19.656349 s3.INFO Falling back to v2 signer
2016/03/03 10:29:19.656694 s3.DEBUG DEBUG: Request s3/HeadBucket Details:
---[ REQUEST POST-SIGN ]-----------------------------
HEAD /allfilecontainer HTTP/1.1
Host: xxx.vip.xxx.com
User-Agent: aws-sdk-go/1.1.6 (go1.5.1; linux; amd64)
Authorization: AWS xxxxx:xxxxx
X-Amz-Date: Thu, 3 Mar 2016 02:29:19 +0000


-----------------------------------------------------
2016/03/03 10:29:19.672345 s3.DEBUG DEBUG: Response s3/HeadBucket Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK

-----------------------------------------------------
2016/03/03 10:29:19.675434 main.INFO File system has been successfully mounted.
2016/03/03 10:29:26.648646 s3.DEBUG DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /allfilecontainer?delimiter=%2F&prefix= HTTP/1.1
Host: xxx.vip.xxx.com
User-Agent: aws-sdk-go/1.1.6 (go1.5.1; linux; amd64)
Authorization: AWS xxxxx:xxxxx
X-Amz-Date: Thu, 3 Mar 2016 02:29:26 +0000
Accept-Encoding: gzip


-----------------------------------------------------
2016/03/03 10:29:26.720938 s3.DEBUG DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Content-Length: 369469
Cache-Control: max-age=86400
Connection: keep-alive
Content-Type: application/xml
Date: Thu, 03 Mar 2016 02:29:26 GMT
Server: Tengine
X-Trans-Id: tx139aa8e839e844eea2d8d-0056d7a186


-----------------------------------------------------
2016/03/03 10:29:26.794436 s3.DEBUG {
  Contents: [
    {
      ETag: "34ddd6128cc4739519b6ba7b8da27d71",
      Key: "00001356-e5e1-44d1-b698-0bb91d1413f2_b6c065e6055e06fd012a2ad7d1cc60d0",
      LastModified: 2015-10-03 14:12:01.43402 +0000 UTC,
      Owner: {
        DisplayName: "fbf878000cb9405cb12589b39fc4965a",
        ID: "fbf878000cb9405cb12589b39fc4965a"
      },
      Size: 70,
      StorageClass: "STANDARD"
    },

………

    {
      ETag: "ab9ee1d898662acaa4d6b1cdfb083836",
      Key: "003e8039-69dd-4950-975b-7c8d5360fdfa_7a623c05a65d7550c69282fc257f6d68",
      LastModified: 2015-10-08 05:18:51.47499 +0000 UTC,
      Owner: {
        DisplayName: "fbf878000cb9405cb12589b39fc4965a",
        ID: "fbf878000cb9405cb12589b39fc4965a"
      },
      Size: 9071,
      StorageClass: "STANDARD"
    }
  ],
 Delimiter: "/",
  IsTruncated: true,
  Marker: "",
  MaxKeys: 1000,
  Name: "allfilecontainer",
  Prefix: ""
}

[kahing]

goofys does handle IsTruncated. Can you verify that the response does not contain NextMarker? According to aws docs the response is supposed to contain that field: https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html .

[kahing]

swift3 1.7.0 looks quite old and predated openstack-archive/swift3@c03a1b5 which added NextMarker. Is it possible for you to upgrade?

[chuang0721]

Sorry for my late reply. I have been busy week. We have solved the problem. Thank you very much.

[kahing]

Great! For future reference, how did you solve the problem?