Looking for a self-hosted file share tool

[scottalanmiller]

@guyinpv said in Looking for a self-hosted file share tool:

@jaredbusch Cool! 120 users connected and yet it reports about 800MB RAM.

"Connected" on a web connection (which is what the sync client is, too) isn't what connected means in a VPN sense. Think about the memory use of 120 users on a website, you'd not even notice. Now if all 120 were actively clicking links and doing things at once, you might notice.

[JaredBusch]

@scottalanmiller said in Looking for a self-hosted file share tool:

@guyinpv said in Looking for a self-hosted file share tool:

@jaredbusch Cool! 120 users connected and yet it reports about 800MB RAM.

"Connected" on a web connection (which is what the sync client is, too) isn't what connected means in a VPN sense. Think about the memory use of 120 users on a website, you'd not even notice. Now if all 120 were actively clicking links and doing things at once, you might notice.

Or updating files at the same time because a new globally shared set of documents get uploaded. Mostly network though.

[guyinpv]

@bnrstnr said in Looking for a self-hosted file share tool:

@guyinpv Just create a share with an admin account that will never be deleted, then you can create a group for all of your users, and add the group to the share as shown in the pic. The folder will automatically appear for every one in that group.

You can also use the normal link sharing rules with this, so every user shares the same link.

https://i.imgur.com/Q69nHOi.png

This is exactly what I did, right out of the box.
But then I found this: https://apps.nextcloud.com/apps/groupfolders

So I created one of those. Then I found the features of group folders didn't work for me. For example the trash bin doesn't work at all for group folders.

So I went back to method #1. Created a share linked to a group.

Really guys, nothing weird going on here. Then all I did was select all the files in the one folder (group share), and used the button called "move" right in the interface to move them to the normal share, which seemed to work. And then I see all the duplicates. Here is a screenshot of how the web interface looks.

@wirestyle22 said in Looking for a self-hosted file share tool:

Whatever that long-winded post is up there, I have no idea. That's an incredibly convoluted way to do that.

I don't see how it's convoluted to want to move files and folders from one folder into another. You'd think a program meant for file management can do such trivial tasks without screwing up it's own cache or indexes or whatever went wrong.

Anyway, so I looked in the file system and there are definitely no duplicates there. It looks as it should.
I ran the occ commands files:scan for all users, no errors reported. Then I ran files:cleanup to clean the filecache.

I didn't see any other commands that seemed relevant.

Anyway, I logged in and duplicates still showing. Grrr.

Then I went in and deleted the previous group share folder (it is empty anyway) and then removed the group share plugin entirely.

At this point there are no clients connected, I'm only working in the web interface.

Others have seen this problem too: https://help.nextcloud.com/t/files-and-folders-shown-twice-in-web-ui/12416/2

https://help.nextcloud.com/t/duplicate-files-delete-not-possible/8869

https://github.com/nextcloud/server/issues/3502

Looks like OwnCloud was dealing with same issue as well: https://github.com/owncloud/core/issues/28018

[guyinpv]

I think I fixed it, but who knows.

When I look at entries in the filecache table, it still references the original group folder, here is a sample query:

That group folder doesn't exist any more, it was deleted through normal means in the web interface, the group folder UI, and removed the plugin itself. So this definitely seems like a bug since I didn't remove the folder in any strange way, it was all done in the web interface.

I took a chance and simply deleted all rows that referenced "groupfolders", a nice 13,631 rows deleted. Now the same query doesn't show any entries for "groupfolders".

A quick refresh of the web interface and all the duplicates disappeared, now looks normal.

You'll notice in my earlier screenshot, the interface was outputting the entire path, rather than just the relative folder name, which was weird. But now it's only showing the folder names, not the whole paths.

I don't know if I want to trust such buggy software, it can't even keep its own filecache in order when moving folders between shares. OwnCloud and NextCloud both seem to be dealing with this inconsistency, and the filecache cleaning command line tool doesn't fix it.

[JaredBusch]

@guyinpv I have never seen this issue.

I have been using ownCloud since version 7 and switched to Nextcloud with version 10/11.

[guyinpv]

@jaredbusch said in Looking for a self-hosted file share tool:

@guyinpv I have never seen this issue.

I have been using ownCloud since version 7 and switched to Nextcloud with version 10/11.

The steps to reproduce it are at the top of this bug report: https://github.com/nextcloud/server/issues/3502

I'd be curious if you tried it. The very last comment on that bug report is a person on version 13 who experienced it. And I did too, on 13.0.1.

The main difference with me is that I used the group folders plugin which, I believe, makes use of the NextCloud ability to connect to external storage. So I'm not sure if this is a bug with normal shares, or only when involving external storage shares.

Whatever the case, if anybody reads any of this, I would definitely stay away from the group folder plugin.

[bnrstnr]

@guyinpv I don't know if this could be contributing at all, but for your "background jobs" ajax is the default, I would definitely setup a cronjob to do this.

https://docs.nextcloud.com/server/13/admin_manual/configuration_server/background_jobs_configuration.html

EDIT: It almost certainly looks like this could be contributing, as the database cleanup relies on this.

[Jame_s]

@guyinpv seafile

https://www.seafile.com/

been using it here for quite a while.

each user here has an upload link i store in active directory and it is added to each user's signature.

[JaredBusch]

@guyinpv said in Looking for a self-hosted file share tool:

@jaredbusch said in Looking for a self-hosted file share tool:

@guyinpv I have never seen this issue.

I have been using ownCloud since version 7 and switched to Nextcloud with version 10/11.

The steps to reproduce it are at the top of this bug report: https://github.com/nextcloud/server/issues/3502

I'd be curious if you tried it. The very last comment on that bug report is a person on version 13 who experienced it. And I did too, on 13.0.1.

The main difference with me is that I used the group folders plugin which, I believe, makes use of the NextCloud ability to connect to external storage. So I'm not sure if this is a bug with normal shares, or only when involving external storage shares.

Whatever the case, if anybody reads any of this, I would definitely stay away from the group folder plugin.

Reading that thread, Group folders are a problem in general.

So blow it out and start over without that app.

It looks like manual clean up was possible by moving the files around while subsequently clicking on things in the GUI.

[guyinpv]

What I did by deleting rows in the filecache table referencing the group share seemed to work. I just can't be sure there aren't any other pieces or references anywhere else in the DB. Hoping it doesn't lead to bugs later if there are still leftovers.

@jame_s said in Looking for a self-hosted file share tool:

@guyinpv seafile

https://www.seafile.com/

been using it here for quite a while.

each user here has an upload link i store in active directory and it is added to each user's signature.

I tested it for a while on our internal server. I had a heck of a time getting clients to connect though, lots of troubleshooting routing, DNS, ports, etc, was never really stable. Perhaps it would be better on some VPS instead of internal. But there are other reasons I didn't go with it, like pricing model. Some info on that here: https://mangolassi.it/topic/9882/why-would-you-chose-nextcloud-over-seafile/17

Seafile had some great concepts, the desktop tool was pretty nice, fast sync. But it's not quite what I was looking for.

[guyinpv]

Speaking of deleting cache entries. This seems to be what the occ command is for occ files:cleanup.
It even reports that way:
0 orphaned file cache entries deleted

"orphaned file cache entries" seems like exactly what I found, it had orphaned entries for the group share folder. When I deleted them, that seemed to me to be "deleting orphaned file cache entries."

So I guess, maybe the files:cleanup tool is not functional or is where the bug really lies if it's not working?

It wouldn't help to put this in a cron job if it doesn't work anyway.

[Jame_s]

@guyinpv odd, i have had no issues with clients, everything is done via https. as far as pricing goes, i use the free version and have about 30 users. no stability issues whatsoever.

[guyinpv]

@jame_s said in Looking for a self-hosted file share tool:

@guyinpv odd, i have had no issues with clients, everything is done via https. as far as pricing goes, i use the free version and have about 30 users. no stability issues whatsoever.

Isn't the free version limited to 3 users?

I'm sure a lot of my issues were from trying to run it on an internal server. Doing https and dealing with virtual networking and routing and everything didn't pan out. Probably could do it better on a cloud VPS.

Regardless, NextCloud does have a couple advantages over SeaFile, but that goes the other way around too so..... pick your poison!

[Jame_s]

@guyinpv community edition is unlimited and we had no use for the extra pro features, but yes, horses for courses.