Looking for a self-hosted file share tool
-
@jaredbusch Cool! 120 users connected and yet it reports about 800MB RAM. Mine is on PHP 7.0.28, MySQL 5.7.21, Ubuntu 16.04. That makes me feel a little better.
As many file sync tools I've tried over the years, the top most important feature becomes stability. As long as the thing keeps syncing, keeps accurate, then I'll deal with the rest.
You'd be surprised how quickly I get myself into trouble.
I found this app for "Group Shared Folders" so that I could have a master share that everybody uses, seemed like a good idea at the time. But then I found out the deleted folder (trash can) doesn't work on group shares.
I created another admin user to act as a master account that I could put a share in for the whole office. Then I used the NC web interface to move all the files from the group share into the new share. This seemed to work fine.
On my test workstations, I had turned off sync so I could move the files locally as well, I want to avoid using a ton of bandwidth if I can help it.
When I turned sync back on, it seemed to go just fine until I noticed in the web interface that most every folder was duplicated! Lots of files appear duplicated as well. But this doesn't translate to Windows, I only see one copy of everything.
When I open the sync client, the duplicates are seen in the folder selection tree.Now it gets weirder. When in the web UI, if I click the share icon on one of the duplicate folders/files, the duplicate magically disappears, just by viewing share info in the sidebar. But if I refresh the folder or click on the folder name, once again all the folders/files reappear as duplicates. This is freaking bizarre. Kinda goes against my need for a stable and robust system.
-
@jaredbusch said in Looking for a self-hosted file share tool:
@stacksofplates said in Looking for a self-hosted file share tool:
Using Remi to get PHP 7.
Just as a heads up, that’s in the CentOS SCLo repos so you don’t need any of the outside repos any longer.
Doesn’t that required the named PHP install though?
Umm, I don't remember off hand. You could be right.
-
@guyinpv said in Looking for a self-hosted file share tool:
@jaredbusch Cool! 120 users connected and yet it reports about 800MB RAM. Mine is on PHP 7.0.28, MySQL 5.7.21, Ubuntu 16.04. That makes me feel a little better.
As many file sync tools I've tried over the years, the top most important feature becomes stability. As long as the thing keeps syncing, keeps accurate, then I'll deal with the rest.
You'd be surprised how quickly I get myself into trouble.
I found this app for "Group Shared Folders" so that I could have a master share that everybody uses, seemed like a good idea at the time. But then I found out the deleted folder (trash can) doesn't work on group shares.
I created another admin user to act as a master account that I could put a share in for the whole office. Then I used the NC web interface to move all the files from the group share into the new share. This seemed to work fine.
On my test workstations, I had turned off sync so I could move the files locally as well, I want to avoid using a ton of bandwidth if I can help it.
When I turned sync back on, it seemed to go just fine until I noticed in the web interface that most every folder was duplicated! Lots of files appear duplicated as well. But this doesn't translate to Windows, I only see one copy of everything.
When I open the sync client, the duplicates are seen in the folder selection tree.Now it gets weirder. When in the web UI, if I click the share icon on one of the duplicate folders/files, the duplicate magically disappears, just by viewing share info in the sidebar. But if I refresh the folder or click on the folder name, once again all the folders/files reappear as duplicates. This is freaking bizarre. Kinda goes against my need for a stable and robust system.
I have no clue WTF you are doing.
But it sounds like your browser shit.
Look at your data store directly.
If it is all good and it still looks bad in a new porn mode browser, then have it rescan the files from the command line.
-
@jaredbusch said in Looking for a self-hosted file share tool:
@stacksofplates said in Looking for a self-hosted file share tool:
Using Remi to get PHP 7.
Just as a heads up, that’s in the CentOS SCLo repos so you don’t need any of the outside repos any longer.
Doesn’t that required the named PHP install though?
Yes, it gives you all versions so you do have to specify.
-
@jaredbusch said in Looking for a self-hosted file share tool:
@guyinpv said in Looking for a self-hosted file share tool:
@jaredbusch Cool! 120 users connected and yet it reports about 800MB RAM. Mine is on PHP 7.0.28, MySQL 5.7.21, Ubuntu 16.04. That makes me feel a little better.
As many file sync tools I've tried over the years, the top most important feature becomes stability. As long as the thing keeps syncing, keeps accurate, then I'll deal with the rest.
You'd be surprised how quickly I get myself into trouble.
I found this app for "Group Shared Folders" so that I could have a master share that everybody uses, seemed like a good idea at the time. But then I found out the deleted folder (trash can) doesn't work on group shares.
I created another admin user to act as a master account that I could put a share in for the whole office. Then I used the NC web interface to move all the files from the group share into the new share. This seemed to work fine.
On my test workstations, I had turned off sync so I could move the files locally as well, I want to avoid using a ton of bandwidth if I can help it.
When I turned sync back on, it seemed to go just fine until I noticed in the web interface that most every folder was duplicated! Lots of files appear duplicated as well. But this doesn't translate to Windows, I only see one copy of everything.
When I open the sync client, the duplicates are seen in the folder selection tree.Now it gets weirder. When in the web UI, if I click the share icon on one of the duplicate folders/files, the duplicate magically disappears, just by viewing share info in the sidebar. But if I refresh the folder or click on the folder name, once again all the folders/files reappear as duplicates. This is freaking bizarre. Kinda goes against my need for a stable and robust system.
I have no clue WTF you are doing.
But it sounds like your browser shit.
Look at your data store directly.
If it is all good and it still looks bad in a new porn mode browser, then have it rescan the files from the command line.
haha, what does that mean?
Like I said, I originally did a "group share", this is a feature via a plugin.
When the features of a group share didn't work out, I went back to just having a regular share owned by a user.
In the web interface, I selected all the files in the group share and used the "move" link to move them into the new share.
Maybe I'm going mad but this seems like a pretty standard operation, move some files and folders from one share into another??But now all those folders/files appear duplicated.
What are the magic commands to force things to rescan at the command line?
-
@guyinpv said in Looking for a self-hosted file share tool:
@jaredbusch said in Looking for a self-hosted file share tool:
@guyinpv said in Looking for a self-hosted file share tool:
@jaredbusch Cool! 120 users connected and yet it reports about 800MB RAM. Mine is on PHP 7.0.28, MySQL 5.7.21, Ubuntu 16.04. That makes me feel a little better.
As many file sync tools I've tried over the years, the top most important feature becomes stability. As long as the thing keeps syncing, keeps accurate, then I'll deal with the rest.
You'd be surprised how quickly I get myself into trouble.
I found this app for "Group Shared Folders" so that I could have a master share that everybody uses, seemed like a good idea at the time. But then I found out the deleted folder (trash can) doesn't work on group shares.
I created another admin user to act as a master account that I could put a share in for the whole office. Then I used the NC web interface to move all the files from the group share into the new share. This seemed to work fine.
On my test workstations, I had turned off sync so I could move the files locally as well, I want to avoid using a ton of bandwidth if I can help it.
When I turned sync back on, it seemed to go just fine until I noticed in the web interface that most every folder was duplicated! Lots of files appear duplicated as well. But this doesn't translate to Windows, I only see one copy of everything.
When I open the sync client, the duplicates are seen in the folder selection tree.Now it gets weirder. When in the web UI, if I click the share icon on one of the duplicate folders/files, the duplicate magically disappears, just by viewing share info in the sidebar. But if I refresh the folder or click on the folder name, once again all the folders/files reappear as duplicates. This is freaking bizarre. Kinda goes against my need for a stable and robust system.
I have no clue WTF you are doing.
But it sounds like your browser shit.
Look at your data store directly.
If it is all good and it still looks bad in a new porn mode browser, then have it rescan the files from the command line.
haha, what does that mean?
Like I said, I originally did a "group share", this is a feature via a plugin.
When the features of a group share didn't work out, I went back to just having a regular share owned by a user.
In the web interface, I selected all the files in the group share and used the "move" link to move them into the new share.
Maybe I'm going mad but this seems like a pretty standard operation, move some files and folders from one share into another??But now all those folders/files appear duplicated.
What are the magic commands to force things to rescan at the command line?
Did you close all browsers tabs and then start a new private browsing session? To ensure your browser was not just showing cached data.
-
I am not at a computer to get you examples.
But first just ssh in and look in the data folder. Look at wha is here. Are there two copies of the files?
If yes. There is the problem.
If no, then run the rescan.
-
@guyinpv Just create a share with an admin account that will never be deleted, then you can create a group for all of your users, and add the group to the share as shown in the pic. The folder will automatically appear for every one in that group.
You can also use the normal link sharing rules with this, so every user shares the same link.
-
@bnrstnr said in Looking for a self-hosted file share tool:
@guyinpv Just create a share with an admin account that will never be deleted, then you can create a group for all of your users, and add the group to the share as shown in the pic. The folder will automatically appear for every one in that group.
You can also use the normal link sharing rules with this, so every user shares the same link.
This.
Whatever that long-winded post is up there, I have no idea. That's an incredibly convoluted way to do that. Curious if you actually duplicated the files or not. Follow JB's advice and check the data directory to verify.
This is the kind of stuff I do and get yelled at. It's like looking in a mirror.
-
@guyinpv said in Looking for a self-hosted file share tool:
@jaredbusch Cool! 120 users connected and yet it reports about 800MB RAM.
"Connected" on a web connection (which is what the sync client is, too) isn't what connected means in a VPN sense. Think about the memory use of 120 users on a website, you'd not even notice. Now if all 120 were actively clicking links and doing things at once, you might notice.
-
@scottalanmiller said in Looking for a self-hosted file share tool:
@guyinpv said in Looking for a self-hosted file share tool:
@jaredbusch Cool! 120 users connected and yet it reports about 800MB RAM.
"Connected" on a web connection (which is what the sync client is, too) isn't what connected means in a VPN sense. Think about the memory use of 120 users on a website, you'd not even notice. Now if all 120 were actively clicking links and doing things at once, you might notice.
Or updating files at the same time because a new globally shared set of documents get uploaded. Mostly network though.
-
@bnrstnr said in Looking for a self-hosted file share tool:
@guyinpv Just create a share with an admin account that will never be deleted, then you can create a group for all of your users, and add the group to the share as shown in the pic. The folder will automatically appear for every one in that group.
You can also use the normal link sharing rules with this, so every user shares the same link.
This is exactly what I did, right out of the box.
But then I found this: https://apps.nextcloud.com/apps/groupfoldersSo I created one of those. Then I found the features of group folders didn't work for me. For example the trash bin doesn't work at all for group folders.
So I went back to method #1. Created a share linked to a group.
Really guys, nothing weird going on here. Then all I did was select all the files in the one folder (group share), and used the button called "move" right in the interface to move them to the normal share, which seemed to work. And then I see all the duplicates. Here is a screenshot of how the web interface looks.
@wirestyle22 said in Looking for a self-hosted file share tool:
Whatever that long-winded post is up there, I have no idea. That's an incredibly convoluted way to do that.
I don't see how it's convoluted to want to move files and folders from one folder into another. You'd think a program meant for file management can do such trivial tasks without screwing up it's own cache or indexes or whatever went wrong.
Anyway, so I looked in the file system and there are definitely no duplicates there. It looks as it should.
I ran the occ commandsfiles:scanfor all users, no errors reported. Then I ranfiles:cleanupto clean the filecache.I didn't see any other commands that seemed relevant.
Anyway, I logged in and duplicates still showing. Grrr.
Then I went in and deleted the previous group share folder (it is empty anyway) and then removed the group share plugin entirely.
At this point there are no clients connected, I'm only working in the web interface.
Others have seen this problem too: https://help.nextcloud.com/t/files-and-folders-shown-twice-in-web-ui/12416/2
https://help.nextcloud.com/t/duplicate-files-delete-not-possible/8869
https://github.com/nextcloud/server/issues/3502
Looks like OwnCloud was dealing with same issue as well: https://github.com/owncloud/core/issues/28018
-
I think I fixed it, but who knows.
When I look at entries in the filecache table, it still references the original group folder, here is a sample query:
That group folder doesn't exist any more, it was deleted through normal means in the web interface, the group folder UI, and removed the plugin itself. So this definitely seems like a bug since I didn't remove the folder in any strange way, it was all done in the web interface.
I took a chance and simply deleted all rows that referenced "groupfolders", a nice 13,631 rows deleted. Now the same query doesn't show any entries for "groupfolders".
A quick refresh of the web interface and all the duplicates disappeared, now looks normal.
You'll notice in my earlier screenshot, the interface was outputting the entire path, rather than just the relative folder name, which was weird. But now it's only showing the folder names, not the whole paths.
I don't know if I want to trust such buggy software, it can't even keep its own filecache in order when moving folders between shares. OwnCloud and NextCloud both seem to be dealing with this inconsistency, and the filecache cleaning command line tool doesn't fix it.
-
@guyinpv I have never seen this issue.
I have been using ownCloud since version 7 and switched to Nextcloud with version 10/11.
-
@jaredbusch said in Looking for a self-hosted file share tool:
@guyinpv I have never seen this issue.
I have been using ownCloud since version 7 and switched to Nextcloud with version 10/11.
The steps to reproduce it are at the top of this bug report: https://github.com/nextcloud/server/issues/3502
I'd be curious if you tried it. The very last comment on that bug report is a person on version 13 who experienced it. And I did too, on 13.0.1.
The main difference with me is that I used the group folders plugin which, I believe, makes use of the NextCloud ability to connect to external storage. So I'm not sure if this is a bug with normal shares, or only when involving external storage shares.
Whatever the case, if anybody reads any of this, I would definitely stay away from the group folder plugin.
-
@guyinpv I don't know if this could be contributing at all, but for your "background jobs" ajax is the default, I would definitely setup a cronjob to do this.
EDIT: It almost certainly looks like this could be contributing, as the database cleanup relies on this.
-
@guyinpv seafile
been using it here for quite a while.
each user here has an upload link i store in active directory and it is added to each user's signature.
-
@guyinpv said in Looking for a self-hosted file share tool:
@jaredbusch said in Looking for a self-hosted file share tool:
@guyinpv I have never seen this issue.
I have been using ownCloud since version 7 and switched to Nextcloud with version 10/11.
The steps to reproduce it are at the top of this bug report: https://github.com/nextcloud/server/issues/3502
I'd be curious if you tried it. The very last comment on that bug report is a person on version 13 who experienced it. And I did too, on 13.0.1.
The main difference with me is that I used the group folders plugin which, I believe, makes use of the NextCloud ability to connect to external storage. So I'm not sure if this is a bug with normal shares, or only when involving external storage shares.
Whatever the case, if anybody reads any of this, I would definitely stay away from the group folder plugin.
Reading that thread, Group folders are a problem in general.
So blow it out and start over without that app.
It looks like manual clean up was possible by moving the files around while subsequently clicking on things in the GUI.
-
What I did by deleting rows in the filecache table referencing the group share seemed to work. I just can't be sure there aren't any other pieces or references anywhere else in the DB. Hoping it doesn't lead to bugs later if there are still leftovers.
@jame_s said in Looking for a self-hosted file share tool:
@guyinpv seafile
been using it here for quite a while.
each user here has an upload link i store in active directory and it is added to each user's signature.
I tested it for a while on our internal server. I had a heck of a time getting clients to connect though, lots of troubleshooting routing, DNS, ports, etc, was never really stable. Perhaps it would be better on some VPS instead of internal. But there are other reasons I didn't go with it, like pricing model. Some info on that here: https://mangolassi.it/topic/9882/why-would-you-chose-nextcloud-over-seafile/17
Seafile had some great concepts, the desktop tool was pretty nice, fast sync. But it's not quite what I was looking for.
-
Speaking of deleting cache entries. This seems to be what the occ command is for
occ files:cleanup.
It even reports that way:
0 orphaned file cache entries deleted"orphaned file cache entries" seems like exactly what I found, it had orphaned entries for the group share folder. When I deleted them, that seemed to me to be "deleting orphaned file cache entries."
So I guess, maybe the
files:cleanuptool is not functional or is where the bug really lies if it's not working?It wouldn't help to put this in a cron job if it doesn't work anyway.
