We are aware of the issue with the badge emails resending to everyone, we apologise for the inconvenience - learn more here.
Forum Discussion
DavidM27
5 years agoExplorer | Level 4
Issue UploadSession and UploadSessionBatch with big files
Hello,
I'm working with the dropbox API in TypeScript. I do want to be able to able all kind of files (small, large, multiple files...)
For that I have made a method to uplaod a single file, and a method to uploadMultiple file. The fisrt one is using the uploadSessionFinish() method provided by the sdk, the other one is using uploadSessionFinishBatch(). To handle small and large file I also have made a method that handle the file and return a filesUploadSessionFinishArg so I can use it in my uploadFile and uploadMultipleFiles method.
This is working well for a single file either small or large. But as soon as I upload multiple files containing at least one big file it fails...
Here is what the code looks like :
Upload single file ->
async uploadFile( file: ArrayBuffer, path: string): Promise<MyFile> { const finishSessionArg : DropboxTypes.files.UploadSessionFinishArg = await this.prepareUploadFile({ file, path, }) const _file: DropboxTypes.files.FileMetadata = await this.dropBox.filesUploadSessionFinish(finishSessionArg) return dropBoxMapper.convertDropboxFileToFile(_file) }
Upload multiple file ->
async uploadMultipleFiles(files: Array<ArrayBuffer>, filesPath: Array<string>) : Promise<Array<MyFile>> { const finishUploadBatchArg: DropboxTypes.files.UploadSessionFinishBatchArg = { entries: [], } // For each file I get the finishArg for (const [index, file,] of files.entries()) { const finishSessionArg : DropboxTypes.files.UploadSessionFinishArg = await this.prepareUploadFile({ file:file, path:filesPath[index], }) finishUploadBatchArg.entries.push(finishSessionArg) } const response: DropboxTypes.files.UploadSessionFinishBatchLaunch = await this.dropBox.filesUploadSessionFinishBatch(finishUploadBatchArg) //Check the batch status from here until it is complete const checkArg : DropboxTypes.async.PollArg = { async_job_id: (<DropboxTypes.async.LaunchResultBaseAsyncJobId> response).async_job_id, } return this.checkUploadSessionBatch(checkArg) }
prepareUpload method ->
async prepareUploadFile(file: ArrayBuffer, path: string) : Promise<DropboxTypes.files.UploadSessionFinishArg> { const UPLOAD_FILE_SIZE_LIMIT: number = 8 * 1024 * 1024 //For testing purpose max size for a single upload is 8Mb const CHUNK_SIZE: number = 4 * 1024 * 1024 // Each chunk is 4Mb // If smaller than 8Mb single upload if (UPLOAD_FILE_SIZE_LIMIT > file.byteLength) { const uploadSessionStartArg : DropboxTypes.files.UploadSessionStartArg = { contents: file, close: true, } const response: DropboxTypes.files.UploadSessionStartResult = await this.dropBox.filesUploadSessionStart(uploadSessionStartArg) const sessionId: string = response.session_id const cursor: DropboxTypes.files.UploadSessionCursor = { contents: file, offset:file.byteLength, session_id: sessionId, } const commitInfo: DropboxTypes.files.CommitInfo = { contents: file, path: path, autorename: true, } const finishSessionArg : DropboxTypes.files.UploadSessionFinishArg= { contents : file, cursor: cursor, commit: commitInfo, } // Workaround to fix issues explained here : https://www.dropboxforum.com/t5/API-Support-Feedback/Uploading-multiple-files-with-UploadSession-with-TypeScript/m-p/399797/thread-id/21878#M21888 delete finishSessionArg.cursor.contents delete finishSessionArg.commit.contents delete finishSessionArg.contents return finishSessionArg } else { //If bigger than 8Mb divide the file in multiple 4Mb chunks const fileChunks: Array<ArrayBuffer> = [] let offset: number = 0 while (offset < file.byteLength) { var chunkSize = Math.min(CHUNK_SIZE, file.byteLength - offset) fileChunks.push(file.slice(offset, offset + chunkSize)) offset += chunkSize } let sessionId: string = '' const finishSessionArg : DropboxTypes.files.UploadSessionFinishArg= { cursor: { contents: {}, session_id: sessionId, offset: 0, }, commit: { contents: {}, path: path, }, contents: {}, } for (const [index,chunk,] of fileChunks.entries()) { if (0 === index) { const uploadSessionStartArg : DropboxTypes.files.UploadSessionStartArg = { contents: chunk, close: false, } const response = await this.dropBox.filesUploadSessionStart(uploadSessionStartArg) sessionId = response.session_id } else if (index < fileChunks.length-1) { const cursor: DropboxTypes.files.UploadSessionCursor = { contents: chunk, offset: index * CHUNK_SIZE, session_id: sessionId, } const uploadSessionAppendArg : DropboxTypes.files.UploadSessionAppendArg = { cursor: cursor, contents: chunk, close:false, } delete cursor.contents await this.dropBox.filesUploadSessionAppendV2(uploadSessionAppendArg) } else { const cursor: DropboxTypes.files.UploadSessionCursor = { contents: chunk, offset: index * CHUNK_SIZE, session_id: sessionId, } const commitInfo: DropboxTypes.files.CommitInfo = { contents: chunk, path: path, autorename: true, } finishSessionArg.commit = commitInfo finishSessionArg.cursor = cursor finishSessionArg.contents = chunk } } // Workaround to fix issues explained here : https://www.dropboxforum.com/t5/API-Support-Feedback/Uploading-multiple-files-with-UploadSession-with-TypeScript/m-p/399797/thread-id/21878#M21888 delete finishSessionArg.cursor.contents delete finishSessionArg.commit.contents delete finishSessionArg.contents return finishSessionArg } }
So to explained what is wrong when I use that code, it tries to upload every file properly, each chunks seems to be uplad with the appendV2 method. But once it is done all the small files are properly uploaded but the large ones are getting this status :
{".tag": "failure", "failure": {".tag": "lookup_failed", "lookup_failed": {".tag": "not_closed"}}}
Do you see any reason for this to happen ?
Thank you for your help.
When using filesUploadSessionFinishBatch, "UploadSessionStartArg.close or UploadSessionAppendArg.close needs to be true for the last upload_session/start or upload_session/append_v2 call". That is to say, since filesUploadSessionFinishBatch can't itself take any further file data, you need to make sure you've already finished sending all of the file data for each upload session, and have indicated that to the Dropbox API on the previous call for that upload session, by setting 'close' to 'true'.
The 'not_closed' error you're getting is indicating that you didn't close the upload session before calling filesUploadSessionFinishBatch. So, you'll need to update your code to set 'close:true' on UploadSessionAppendArg for the the last call to filesUploadSessionAppendV2 for each upload session that you're then sending to filesUploadSessionFinishBatch.
- Greg-DBDropbox Staff
When using filesUploadSessionFinishBatch, "UploadSessionStartArg.close or UploadSessionAppendArg.close needs to be true for the last upload_session/start or upload_session/append_v2 call". That is to say, since filesUploadSessionFinishBatch can't itself take any further file data, you need to make sure you've already finished sending all of the file data for each upload session, and have indicated that to the Dropbox API on the previous call for that upload session, by setting 'close' to 'true'.
The 'not_closed' error you're getting is indicating that you didn't close the upload session before calling filesUploadSessionFinishBatch. So, you'll need to update your code to set 'close:true' on UploadSessionAppendArg for the the last call to filesUploadSessionAppendV2 for each upload session that you're then sending to filesUploadSessionFinishBatch.
- DavidM27Explorer | Level 4
Greg, thank you for your reply !
So if I understand properly, correct me if I'm wrong, in case of uplaoding a big files within a batch session, I should not stop the AppendArgV2() at the chunkList-1 but until all the chunk are uploaded and then I build my finishArg. But in the other case I should stick with what I've done.
What if I always append all chunks and close on the last chunk ? Does it sound possible to you ?
- DavidM27Explorer | Level 4
I'm answering my own question, it seems to work when I always close the fileUploadSession with the appendArg on the last chunk event when I'm not using the batch.
Thank you for your help ! If you have any comments or suggestion on the code I sent you above please let me know I would be glad to have your feedback on it.
About Dropbox API Support & Feedback
Find help with the Dropbox API from other developers.
5,877 PostsLatest Activity: 12 months agoIf you need more help you can view your support options (expected response time for an email or ticket is 24 hours), or contact us on X or Facebook.
For more info on available support options for your Dropbox plan, see this article.
If you found the answer to your question in this Community thread, please 'like' the post to say thanks and to let us know it was useful!