ML
    • Recent
    • Categories
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    B2 Uploading large files - Post - File already closed

    Scheduled Pinned Locked Moved IT Discussion
    backblaze b2apisyncrclonelinuxappleexternal storage
    10 Posts 1 Posters 1.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • DustinB3403D
      DustinB3403
      last edited by

      @BackblazeNathan Hoping you can help me to understand what's going on. I have rclone performing a PoC sync from an external volume that is backing up into our B2 account and bucket.

      On a few files I'll get an ERROR: File/Name Failed to copy: Post long url: file already closed

      Based on the documentation here the error code means the "method is not allowed".

      What does this actually mean, and is there anyways to quickly resolve said issue(s)?

      1 Reply Last reply Reply Quote -2
      • DustinB3403D
        DustinB3403
        last edited by

        This doesn't occur on any pattern of a file, large files or small files seem to be randomly affected or not. Running the sync again some times seems to work. Is there a better approach to syncing that I should be looking at?

        1 Reply Last reply Reply Quote -2
        • DustinB3403D
          DustinB3403
          last edited by

          Worth nothing that a least a few of the files that error'd with the Post code are being sync'd now, so is the Post error something to simply ignore?

          1 Reply Last reply Reply Quote -2
          • DustinB3403D
            DustinB3403
            last edited by

            So with doing some more digging and running rclone in verbose mode, this appears to be an issue with the workstation closing the SSH connection.

            2019-09-10 10:23:46 ERROR : sftp://user@ip:22//Volumes/G-SPEED Shuttle TB3/Client/: Discarding closed SSH connection: EOF
2019-09-10 10:23:46 ERROR : sftp://user@ip:22//Volumes/G-SPEED Shuttle TB3/Client/: Discarding closed SSH connection: EOF
2019-09-10 10:23:46 ERROR : sftp://user@ip:22//Volumes/G-SPEED Shuttle TB3/Client/: Discarding closed SSH connection: EOF
2019-09-10 10:23:46 ERROR : sftp://user@ip:22//Volumes/G-SPEED Shuttle TB3/Client/: Discarding closed SSH connection: EOF
2019-09-10 10:23:46 ERROR : sftp://user@ip:22//Volumes/G-SPEED Shuttle TB3/Client/: Discarding closed SSH connection: EOF
2019-09-10 10:23:46 ERROR : sftp://user@ip:22//Volumes/G-SPEED Shuttle TB3/Client/: Discarding closed SSH connection: EOF
2019-09-10 10:23:46 ERROR : sftp://user@ip:22//Volumes/G-SPEED Shuttle TB3/Client/: Discarding closed SSH connection: EOF
2019-09-10 10:23:47 ERROR : Becoming Client/Renders/first4.mov: Failed to copy: failed to open source object: Open: couldn't connect SSH: dial tcp ip:22: connect: connection refused
2019-09-10 10:23:47 ERROR : B2 bucket G-Tech: not deleting files as there were IO errors
2019-09-10 10:23:47 ERROR : B2 bucket G-Tech: not deleting directories as there were IO errors
2019-09-10 10:23:47 ERROR : Attempt 3/3 failed with 3 errors and: failed to open source object: Open: couldn't connect SSH: dial tcp ip:22: connect: connection refused
Transferred:       15.329G / 15.329 GBytes, 100%, 3.126 MBytes/s, ETA 0s
Errors:                 3 (retrying may help)
Checks:               637 / 637, 100%
Transferred:            1 / 1, 100%
Elapsed time:  1h23m41.8s
2019/09/10 10:23:47 Failed to sync with 3 errors: last error was: failed to open source object: Open: couldn't connect SSH: dial tcp ip:22: connect: connection refused

            Anyone have any idea on how to better handle this?

            1 Reply Last reply Reply Quote -2
            • DustinB3403D
              DustinB3403
              last edited by

              Looks like I'll need to make 1 small edit to each workstation's ssh_config file.

              ServerAliveInterval 120 as an addition should do. ConnectionTimeout 0 is a default that's disabled as well, so that may have some issues.

              Will test and report back.

              1 Reply Last reply Reply Quote -2
              • DustinB3403D
                DustinB3403
                last edited by

                So adjusting the ServerAliveInterval seems to have done the trick, it's a simple and small change; at least as far as I've tested since this morning.

                But!

                I'm having a hell of a time with rclone and this 1 damn file, it keeps failing to sync this 1 file to B2 for some god forsaken reason and is really ticking me off and ruining my PoC.

                It'll copy the file to B2 to 100% and then start over again for some reason. . . . the file is almost 4GB (but this is smaller than other files that have finished) and the logs don't really point to a specific reason this 1 file is not syncing.

                ffs!

                1 Reply Last reply Reply Quote -2
                • DustinB3403D
                  DustinB3403
                  last edited by

                  Manually uploading the file with Cyberduck did work, so I'm not sure where to go with this. . .

                  1 Reply Last reply Reply Quote -2
                  • DustinB3403D
                    DustinB3403
                    last edited by

                    Maybe just drafting policy to have the backup checked would make the most sense, but I'm not sure how / where that would work as a lot of these things are just going to be a dumping ground of files.

                    1 Reply Last reply Reply Quote -2
                    • DustinB3403D
                      DustinB3403
                      last edited by

                      Turns out this may be a bug in rclone that potentially is fixed in https://beta.rclone.org/branch/v1.49.0-041-g6ade4a26-fix-already-closed-beta/

                      Testing now approximately 2 minutes to find out. (of course it could just have worked this time who honestly knows).

                      1 Reply Last reply Reply Quote -2
                      • DustinB3403D
                        DustinB3403
                        last edited by

                        No dice just yet, finishing a beer and headed home. Going to see if this can get resolved tomorrow.

                        1 Reply Last reply Reply Quote -2
                        • 1 / 1
                        • First post
                          Last post