We are aware of the issue with the badge emails resending to everyone, we apologise for the inconvenience - learn more here.

Forum Discussion

Levi2's avatar
Levi2
Explorer | Level 3
6 years ago

Internal server error when sharing folders

Hi, is there some kind of limitation with sharing folders? I'm trying to share 5.4k folders and I'm getting 

PollError('internal_error', None))

 

consistently. Sometimes the files share, sometimes they don't. I can manually share them no problem and usually I can share them as one offs in a python REPL using the dropbox sdk.

There are no error details.

My code is functionallt simialr to the snippet below:

import dropbox

#censored
access_token = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
dbx = dropbox.Dropbox(access_token)

def get_entries(folder_result_list):
    '''using this so we don't try and share the same folder twice'''
    entries = []
    for entry in folder_result_list.entries:
        if not hasattr(entry,'shared_folder_id') and not entry.shared_folder_id:
            entries.append(entry)
    return entries

folder_result_list = dbx.files_list_folder(parent_folder,include_mounted_folders=False,limit=2000)
entries = get_entries(folder_result_list)

while folder_result_list.has_more:
    folder_result_list = dbx.files_list_folder_continue(folder_result_list.cursor)
    entries.extend(get_entries(folder_result_list))

#filter for only FolderMetadata so we don't try and share DeletedFolder's or individual files    
entry_dict  = {entry.id:entry for entry in entries if isinstance(entry,dropbox.files.FolderMetadata)}

for index,keyval in enumerate(entry_dict.items()):
    entry_id = keyval[0]
    entry = keyval[1]
    try:
        #forcing async so we get consistent results. checking the job status and doing stuff with the shared_folder_id later on in the script
        entry_id_to_job_id[entry_id] = dbx.sharing_share_folder(entry.path_lower,force_async=True).get_async_job_id()
    except dropbox.exceptions.ApiError as e:
        #do some logging of errors here
        pass

The full error looks something like this:

ApiError('0ed9d04d8f01449b5dedea33b7445fee', PollError('internal_error', None))

 

 

  • Thanks for the post, and apologies that the error message isn't more helpful in this case. For this `internal_error` case, I looked it up and I can confirm that it's actually due to "lock contention". 

    That's a result of how Dropbox works on the backend. It's a technical inability to make multiple modifications, such as sharing a folder, at the same time. The simultaneous activity could be coming from your app itself, or elsewhere, e.g., from the user's desktop client. It can come from the same user, or another member of a shared folder. You can find more information about lock contention here:

    https://www.dropbox.com/developers/reference/data-ingress-guide

    In short, to avoid this error, you should avoid making multiple concurrent state modifications. E.g., don't issue multiple such requests at a time, and use batch endpoints whenever possible. That won't guarantee that you won't run in to this error though, as contention can still come from other sources, so you may also want to implement automatic retries in your app. For this particular scenario of sharing multiple folders, I also recommend adding a delay between the consecutive share operations, which can help in certain cases.

    I'll send this along as a request to the team to improve the error reporting.

  • Greg-DB's avatar
    Greg-DB
    Icon for Dropbox Staff rankDropbox Staff

    Thanks for the post, and apologies that the error message isn't more helpful in this case. For this `internal_error` case, I looked it up and I can confirm that it's actually due to "lock contention". 

    That's a result of how Dropbox works on the backend. It's a technical inability to make multiple modifications, such as sharing a folder, at the same time. The simultaneous activity could be coming from your app itself, or elsewhere, e.g., from the user's desktop client. It can come from the same user, or another member of a shared folder. You can find more information about lock contention here:

    https://www.dropbox.com/developers/reference/data-ingress-guide

    In short, to avoid this error, you should avoid making multiple concurrent state modifications. E.g., don't issue multiple such requests at a time, and use batch endpoints whenever possible. That won't guarantee that you won't run in to this error though, as contention can still come from other sources, so you may also want to implement automatic retries in your app. For this particular scenario of sharing multiple folders, I also recommend adding a delay between the consecutive share operations, which can help in certain cases.

    I'll send this along as a request to the team to improve the error reporting.

    • Levi2's avatar
      Levi2
      Explorer | Level 3

      Thanks for the links.

      This makes sense since I was trying to share thousands of folders. However, I avoided multi-threading and no one else is using this folder since it's a test folder.

      It seems that namespace locking is the culprit since all of the folders are in the same name space and I'm using force_asyn=True leaving when the folder actually gets shared up to dropbox.

      I know that there are batch endpoints for uploading data, but are there batch endpoints for sharing folders?

       

      • Greg-DB's avatar
        Greg-DB
        Icon for Dropbox Staff rankDropbox Staff

        No, unfortunatelty there isn't a batch option for sharing folders in particular, but I'll pass this along as a feature request. I can't promise if or when that might be implemented though.