Curious about A, B, C, and D drives? Learn what they mean and how to use them effectively with Dropbox in this handy guide! - check it out now!
Forum Discussion
Solved
Dropbox uploads TOO fast
Hi All,
This may sound like a strange glitch. But, I use dropbox to send important files to paying clients, so I want to try to figure out what is going on to make sure that I don't send people...
- If you add a file to your Dropbox file, which is identical to another file on your account, then the Dropbox desktop application recognizes they are the same, so it creates a duplicate of the original file on the site and instantly 'syncs' the file.
This works even if the new file has a different name. I've tested on on my device with a 1 GB video.
I didn't know that. Super useful. Thanks!
I have noticed something similar with audio files. I will upload 20GB of audio files which takes a while. I then make a tiny change to the audio in those audio files.
This audio is a bunch of 30 minute wave files of a tv program split into its various components, dialogue, music, effects, full mix etc. I change one tiny 5 second segment. That is the only thing that has changed.
I erase the old files in DB, I recreate the files, give them different names and they have different timestamps. They are almost identical to the old files as far as content goes except for the changed 5 secs of audio across all the wave files, the names and the time stamps.
DB seems to upload these very very quickly. Is DB so clever that it compares the content of the new and old files and updates the original uploaded files only altering the changed 5 secs in the whole file, plus the name and the time stamp?
- Rich
Super User II
KMT333 wrote:
Is DB so clever that it compares the content of the new and old files and updates the original uploaded files only altering the changed 5 secs in the whole file, plus the name and the time stamp?
Yes. Before every file is uploaded, it's split into 4MB chunks and then hashed. Before uploading the chunks, Dropbox compares the hash of each chunk to the hashes of all the chunks in your account. If there's a match, Dropbox doesn't upload that chunk. It already exists online, so no need to upload it.
Now the next part is dependent on the applications writing your files. If, when a file is modified, an application rewrites the entire file or just writes data to the beginning of the file, it's likely that every chunk of that file (and every hash) will be different, resulting in Dropbox having to re-upload the entire file. But if the application only rewrites a portion of the file, or just adds new data to the end of the file, the majority of your file remains the same, so most of the hashes remain the same, and Dropbox only needs to upload the chunks that have changed. In such a case, it will appear as though Dropbox has re-uploaded the entire 20GB file when in reality it's only uploading a few 4MB chunks.
Thank you for your reply!
That's impressive. I did actually totally recreate of all the files using exactly he same method that I used to create the original files so they are the same except for the tiny changed section, name and time info.
The software I use must be incredibly / perfectly accurate in its creation / recreation to the files, which is comforting :)!
- This is true, I’ve had to do a lot of digging around for a somewhat similar problem with this method of uploading/downloading in ‘chunks’ adopted by Dropbox is often referred to as block mode.
This works great for most users which I expect are working with everyday small and uncomplicated programs, word, excel, one note etc. and seems to make good sense to me however be careful and seek advice from someone appropriate if you use apps that rely on a database of some kind for whatever reason. Lots of 3d modelling / GIS programs work this way and I’ve tested uploading the same data to the major players in this space. I found any service that adopts block storage as Dropbox does had issues uploading files or rather, the software that opens the files doesn’t see what it expects and then problems occur. Usually like Rich said, the only thing that’s changed is a small line of text/code in a file at the end, this will often not be uploaded but sometimes I found it was. Downloads didn’t appear to have the same issues from the experiments I completed anyway.
I have created some power apps, shell code etc to try and work around and the convulsion I came to is this: for programs like I’ve mentioned, use databases which reference other files and point to various locations within the projects it’s ‘root’ folder the safest was is to use a service that doesn’t adopt block storage which will take longer of course, but I would suspect the difference in minutes is negligible compared to missing or corrupt data. Programs like this really need a check in and out system where the pc or the user can specify dependants which would normally be, the project tile or root folder it self. Ie anytime I check out Project.xyz, every file found as of that moment in time (a new list needs to be created) needs to also I’ve checked and then uploaded once finished making the changes. The only way I’ve come close with Dropbox and almost all the other provides I’ve used is to pause the syncing of the cloud app while working on the project so to speak. When you’re finished, saved and closed down the what you need, then resume syncing. It shouldn’t make too much of a difference but it seems to. I suspect some kind of indexing process happens when a services resumes after being in a period of paused/inactive. That had been my experience anyway.
