Unable to read a parquet file from blob with Hierarchical namespace enabled about adlfs

[raybellwaves]

I just created my own blob via the Azure free account and it worked fine:

>>> dd.read_parquet('abfs://tmp/tmp.parquet', storage_options=storage_options)
Dask DataFrame Structure:
                col1   col2
npartitions=2              
0              int64  int64
2                ...    ...
3                ...    ...
Dask Name: read-parquet, 2 tasks

May have to see how the other blob is setup.

[raybellwaves]

Blob that worked:

Location: East US, West US (Primary: Available, Secondary: Available)
Replication: Read-access geo-redundant storage (RA-GRS)

Blob that does not work:

Location: East US 2
Replication: Locally-redundant storage (LRS)

[raybellwaves]

Closing as not related to adlfs. If am able to 'fix' the storage account i'll post it at https://stackoverflow.com/questions/61220615/dask-read-parquet-from-azure-blob-azurehttperror

[raybellwaves]

I believe I got to the bottom of it.

When creating a blob to match the one that wasn't working I switched on 'Hierarchical namespace' Data Lake Storage Gen2 and reproduced the error.

[raybellwaves]

[raybellwaves]

Not sure if related to
azure-datalake-store https://github.com/dask/adlfs/blob/master/requirements.txt#L1

[hayesgb]

Thanks for digging into this @raybellwaves. I just looked at the configuration on one of the accounts I'm using with adlfs daily. There, Heirarchical namespace is turned on, but Replication is set to Geo-redundant Storage. Can you confirm if that combination works in your test space?

[raybellwaves]

I believe the below matches your configuration and I was not able to read the parquet file.

[raybellwaves]

Can kind of see it in the blob logs but not in much detail

[raybellwaves]

Wanted to be a bit more thorough on the blob configuration. I am unable to read when I turn on the hierarchical namespace.

[hayesgb]

Thanks for sharing this @raybellwaves. Just to add to it, my current instance is:

• Region: East US 2
• Replication: GRS
• Performance: Standard
• Secure Transfer: Enabled
• Access Tier: Hot
• Hierarchical Namespace: Enabled
• NFSv3: Disabled

Any chance you can check the other parameters? Unfortunately, it will be at least another week before I can circle back to work on this.

[raybellwaves]

No problem. Thanks for sharing your config. Will do.

[raybellwaves]

[raybellwaves]

This has been working fine during the last couple of days. Closing.