We are aware of the issue with the badge emails resending to everyone, we apologise for the inconvenience - learn more here.
Forum Discussion
reading parquet file using python sdk
Hi, I am trying to read a parquet file using pandas and vaex. I can sucessfully read a .csv but I get the following error message when I try to download the parquet file with dbx.files_download :
dropbox.exceptions.ApiError: ApiError('4ce7ef4f93544a4fa18c29478e1869a8', DownloadError('path', LookupError('not_file', None)))
Full code:
ACCESS_TOKEN = "My_Token"
# Initialize the Dropbox API client
dbx = dropbox.Dropbox(ACCESS_TOKEN)
# download csv file from dropbox
metadata, f_csv = dbx.files_download('/County_test.csv')
# this works for csv and pandas
with io.BytesIO(f_csv.content) as stream:
df = pd.read_csv(stream, index_col=0)
print(df)
# this works for csv and vaex
with io.BytesIO(f_csv.content) as stream:
df = vaex.read_csv(stream, index_col=0)
print(df)
# download parquet file from dropbox FAILS
metadata, f_parquet = dbx.files_download('/County_test.parquet')
# this part NOT tested yet
with io.BytesIO(f_parquet.content) as stream:
df = pd.read_parquet(stream, index_col=0)
print(df)
Thanks for following up and sharing that. For reference, the URL parameters for shared links like that are documented here.
- Greg-DB
Dropbox Staff
I see you're getting a 'not_file' error, which means: "We were expecting a file, but the given path refers to something that isn’t a file."
It looks like the ".parquet" is not a file, but rather a sort of folder, possibly referred to as a "package" or "bundle" in some environments.
That being the case, to download that you would instead need to use files_download_zip (or files_download_zip_to_file) and then unzip the downloaded zip file, or walk through the contents using files_list_folder/files_list_folder_continue and then download each individual nested item using files_download (or files_download_to_file). The first option of using files_download_zip is probably better/faster.
Thanks for replying. I have tried with one of your solutions but it's really slow.
file_list = []
for entry in dbx.files_list_folder(path="/County.parquet").entries:
print(entry.path_lower)
_, dwnld_file = dbx.files_download(entry.path_lower)
with io.BytesIO(dwnld_file.content) as stream:
pd_df = pd.read_parquet(stream) #this works
vdf = vaex.from_pandas(pd_df)
del pd_df
file_list.append(vdf)
conc_df = vaex.concat(file_list)
print(conc_df)I have tried with dbx.files_download_zip but I can't find a way to read the data it returns 'utf-8' codec can't decode byte 0x82 in position 12: invalid start byte.
- Greg-DB
Yes, using files_list_folder/files_list_folder_continue and files_download (or files_download_to_file) requires more API calls so it would be less performant.
The files_download_zip method would return the requested data the same way files_download would, except that it would be zip data that you would need to unzip to access the original folder. Also, note that you can use files_download_zip_to_file if you want to save the zip data to a file. Can you share the code you're having trouble with for that, and indicate which line fails with that error?
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.
