Learn how to make the most out of the Dropbox Community here 💙.
Forum Discussion
Solved
How to make Dropbox ignore node_modules folder with symbolic links (aliases)
PROBLEM
Dropbox does not currently provide an ignore file/folder feature that enables a locally-unique instance to exist on each machine. One important use of this is the “node_moduels" folder used for node javascript development. The node_modules folder sees thousands of file changes during the development cycle. If this folder is allowed to sync, the important files you need (outside of it), are so far down in the sync queue that you never have the important files when you need them on your other systems.
SOLUTION
A better way to sync projects with Dropbox while ignoring the node_modules folder is to use a “properly” formed symlink (alias). At this time npm and yarn support a property formed symlink. HOWEVER, IT IS CRITICAL THAT THE LINK (ALIAS) AND TARGET HAVE THE SAME NAME!
From shell prompt (terminal):
- cd YOUR_PROJECT_NAME
- mkdir -p ~/Downloads/node_modules_for_YOUR_PROJECT_NAME
-
mv node_modules ~/Downloads/node_modules_for_YOUR_PROJECT_NAME
-
ln -s ~/Downloads/node_modules_for_YOUR_PROJECT_NAME/node_modules
Now create the same target node_moduels folder on each of your other systems **AFTER** Dropbox syncs the ENTIRE project direcotry. The old node_modules folder should be removed and the symlink (alias) added before running these steps.
- cd YOUR_PROJECT_NAME
- mkdir -p ~/Downloads/node_modules_for_YOUR_PROJECT_NAME/node_modules
- yarn install
SUCCESS
Dropbox WILL sync the symlink/alias file itself, but NOT the contents it points to. This seems to work well for file syncing across several of your own development systems, and then use git to push to the repo for others on the team.
CAVEAT
The tiny symlink (alias) file itself IS sync’d on all systems, but will point to a non-existent / empty folder on the other systems. This is most applicable for a single user, but can also be useful for teams if an individual variant of the ignored data is appropriate, such as for node_moduels.
Hope this helps.
🙂
..sebastian
NOTE: The Initial Post on this thread IS the Solution. While the technique specifically targets the node_modules folder in Javascript / TypeScript projects, it can also be applied to any "folders" that need to be ignored (see bottom of this post for a drag-n-drop technique).
HOW IT WORKS
Because Dropbox does not synchronize folders referened by symlinks (aliases), you can move those folders into a NON-DROPBOX path (outside of dropbox), then create a symlink (alias) in the original Dropbox location.
RESULT
The ignored folder in Dropbox looks and behaves as normal folder, but in reality the data lives in a non-synchronized location on the storage device.
HOW TO IGNORE ANY FOLDER (THE DRAG-N-DROP WAY)
- Move the folder to ignore "Outside of Dropbox" into another location (e.g. ~/Documents)
- In Finder (Mac OS), Command+Option Drag the folder back into Dropbox to create an alias. The cursor should turn into a little curved arrow while dragging if you have the right keys pressed. Once dropped, an alias will be created in the original Dropbox location pointing to the moved folder.
NOTE: Using symlinks/aliases on Windows and Linux should behave the same way, but I have not verified it. Also, I've updated the original post to state that the tiny symlink (alias) file itself "IS" copied to all systems, but not the content of the folder.
..sebastian
Thank you for posting this idea!
The only thing is... where is this working? In OSX Dropbox is still syncing the symlinks.
NOTE: The Initial Post on this thread IS the Solution. While the technique specifically targets the node_modules folder in Javascript / TypeScript projects, it can also be applied to any "folders" that need to be ignored (see bottom of this post for a drag-n-drop technique).
HOW IT WORKS
Because Dropbox does not synchronize folders referened by symlinks (aliases), you can move those folders into a NON-DROPBOX path (outside of dropbox), then create a symlink (alias) in the original Dropbox location.
RESULT
The ignored folder in Dropbox looks and behaves as normal folder, but in reality the data lives in a non-synchronized location on the storage device.
HOW TO IGNORE ANY FOLDER (THE DRAG-N-DROP WAY)
- Move the folder to ignore "Outside of Dropbox" into another location (e.g. ~/Documents)
- In Finder (Mac OS), Command+Option Drag the folder back into Dropbox to create an alias. The cursor should turn into a little curved arrow while dragging if you have the right keys pressed. Once dropped, an alias will be created in the original Dropbox location pointing to the moved folder.
NOTE: Using symlinks/aliases on Windows and Linux should behave the same way, but I have not verified it. Also, I've updated the original post to state that the tiny symlink (alias) file itself "IS" copied to all systems, but not the content of the folder.
..sebastian
I received a response from Tech Support explaining how to use the xattr command in terminal to ignore a file/folder (link below).
There is a an upside and a downside to each approach (xattr and symlink). The xattr method ignores the target file/folder content and does not upload it, HOWEVER, upon EVERY change within node_modules, Dropbox reindexes and pegs the processor for long periods. The symlink method avoids indexing entirely, so it has a zero-impact when changes within node_modules occur. The symlink/alias file may point to a non-existent folder on a system that doesn't need it, so it's ideal for node_modules, but may not for all others.You can find the xattr ignore Dropbox article here, plus ongoing conversation about the ignore topic here (pages 87, 88, etc...). Hopefully the upcoming xattr ignore method (currently in beta) will support a hard-ignore without reindexing before it's released.
Are you a god?
I've been searching for a solution for several years! I wouldn't expect it to be as simple as a symlink. Kudos!
It does not work for me (on Windows). The first npm install is fine, the second run will basically delete the symlink, recreate the directory, add some packages and then fail with a file not found error :(
- lkljsjjjsjj, Glad to hear it works for you!! I too pulled my hair out for years before figuring this configuration out.Arne1, Sorry to hear it isn't easily translating to Windows. Is the "second run" failure happening on the same system, or on a secondary system? I no longer have access to a Window platform or I'd be happy to troubleshoot it.
- Windows Secondary System: “Theoretically", Dropbox should sync the symlink/alias file itself to all clients subscribed to the same Dropbox/share. Once the symlink/alias file itself is sync'd to your other system(s), you need to create the exact same target folder path (e.g. "C:\users\your_username\downloads\node_modules_for_projectname\node_modules").
- Windows Secondary Run (on same system): "Theoretically" NPM should behave the same, regardless of platform, but maybe not. Have you tried yarn? What do the commands do after creating the node_modules symlink=>downloads_folder?
- yarn install
- yarn upgrade --latest
Unfortunately, I do not know the behavior of symlinks in cross-platform implementations (e.g. Mac Dropbox syncing w/ Windows and/or Linux Dropboxes).** Can someone working with NPM on Windows help with a “Working and Tested” set of steps for Windows and post back? **Hi,
Thanks for the reply! It is the same machine.
It seems to work with yarn though, but I can't switch easily since other people work with the code (and lock file) too.
Symlinks are probably the best general solution. For developers using Git, you have another option. If you are fanatic about creating great git history/commit log for your changes, then you should try using `git worktree add ../projectA-dupe` which will create a duplicate folder that's a sibling to your current git repo. It's actually another "working directory" in proper git terms. It's the same git repo, all the same branches, same stashes, all the git info is the same. Actually the .git dir for "projectA-dupe" basically just points to your original git repo's .git folder. The advantage here, the workflow, is to have your main working directory outside of dropbox entirely, and then add another "working directory" or "worktree" inside your dropbox folder, which will sync. Or, vice versa. You could have the main git repo be inside dropbox, but, setup an npm pre-install hook to fail installation if it sees "Dropbox" is in your current folder path. All developers are instructed to cd to the main git repo inside Dropbox, and then run `git worktree add ../../path/to/your/projectA`. Furthermore, you could have everyone keep their worktree inside dropbox, which means they could all collaborate without ever doing a git push/pull/fetch, by simply allowing dropbox to sync all the git objects. One developer could help another simply by cd'ing to their worktree inside of dropbox. They see the exact same thing, no need to stash changes, commit anything, etc. The symlink might as well be committed into the git repo, and point to a globally reliable path, like ~/universal_node_modules/projectA. Better than all that, probably use yarn v2, and inside .yarnrc.yml set cacheDirectory to be something like ~/Dropbox/.yarn-cache/projectA, so all worktree's can share the same cache.
I've recently implemented dropboxignore which is a simple shell script which facilitates you to generate .dropboxignore files based on your file patterns or even based on existing .gitignore files and ignore matched files from dropbox. Hope to find it useful for your case. Any feedback is more than welcome. dropboxignore is currently available only for Mac OS and Linux.
This is just dumb. No system should ever back up node_modules. This makes me want to rethink my Dropbox subscription.
It's 2021 and Dropbox still has no ignore functionality. I'm wondering how difficult would that be, really. Just before you sync a folder, check for existence of a .dropboxignore file, and then make sure you don't sync the files listed there. Like, seriously. How long can this take to implement? A week? A month? What the f are you waiting for?
Thank you so much! After so much time wasted fiddling with xattr andthe Dropbox selective sync!!
