How to let only customers download files with wget/curl?
-
@Pete-S said in How to let only customers download files with wget/curl?:
@stacksofplates said in How to let only customers download files with wget/curl?:
@Obsolesce said in How to let only customers download files with wget/curl?:
Some sort of authentication is the only way if the app doesn't support built in updating of some kind.
Yeah as @Obsolesce said auth is prob best. Basic auth would be the easiest.
Just
curl -u 'user:pass' https://mything.com/filesWget is
--userand--passwordDoes using digest auth make it better? Perhaps I don't have to put the password in clear text that way.
I would say digest isn't any better. With basic auth, yes you send the creds across the wire, but I believe digest still requires MD5 hashing.
This is going to depend a lot on what your server allows (if you're not writing an API from scratch) and the pain for your clients.
A JWT is better (or some other type of bearer token) but it's extra work for your clients if they're using just curl or wget.
-
@Pete-S said in How to let only customers download files with wget/curl?:
@stacksofplates said in How to let only customers download files with wget/curl?:
@Obsolesce said in How to let only customers download files with wget/curl?:
Some sort of authentication is the only way if the app doesn't support built in updating of some kind.
Yeah as @Obsolesce said auth is prob best. Basic auth would be the easiest.
Just
curl -u 'user:pass' https://mything.com/filesWget is
--userand--passwordDoes using digest auth make it better? Perhaps I don't have to put the password in clear text that way.
Do you mean clear text on the wire or clear text on the cli?
-
@stacksofplates said in How to let only customers download files with wget/curl?:
@Pete-S said in How to let only customers download files with wget/curl?:
@stacksofplates said in How to let only customers download files with wget/curl?:
@Obsolesce said in How to let only customers download files with wget/curl?:
Some sort of authentication is the only way if the app doesn't support built in updating of some kind.
Yeah as @Obsolesce said auth is prob best. Basic auth would be the easiest.
Just
curl -u 'user:pass' https://mything.com/filesWget is
--userand--passwordDoes using digest auth make it better? Perhaps I don't have to put the password in clear text that way.
I would say digest isn't any better. With basic auth, yes you send the creds across the wire, but I believe digest still requires MD5 hashing.
This is going to depend a lot on what your server allows (if you're not writing an API from scratch) and the pain for your clients.
A JWT is better (or some other type of bearer token) but it's extra work for your clients if they're using just curl or wget.
My idea on the client side is that it has to be done with standard tools available on basically every linux distro such as wget, curl etc. Basically a couple of lines that an admin could copy & paste into ssh from instructions.
If you have to enter a password / username somewhere it should only need to be done once.
Besides basic and digest authentication I also had a look at SSL/TLS client certificates. I didn't know about that but it's been around for a long time. Basically the server is authenticated by the server certificate and in the same manner the client can be authenticated by the client certificate.
The webserver would then have to verify that the client has a valid SSL certificate and is allowed to download whatever file it is.
-
@stacksofplates said in How to let only customers download files with wget/curl?:
@Pete-S said in How to let only customers download files with wget/curl?:
@stacksofplates said in How to let only customers download files with wget/curl?:
@Obsolesce said in How to let only customers download files with wget/curl?:
Some sort of authentication is the only way if the app doesn't support built in updating of some kind.
Yeah as @Obsolesce said auth is prob best. Basic auth would be the easiest.
Just
curl -u 'user:pass' https://mything.com/filesWget is
--userand--passwordDoes using digest auth make it better? Perhaps I don't have to put the password in clear text that way.
Do you mean clear text on the wire or clear text on the cli?
CLI. Using https I assume every thing is encrypted on the line.
-
@DustinB3403 said in How to let only customers download files with wget/curl?:
@Pete-S using SSL is the first step before you start looking at measures for passing credentials with web traffic
Yes, I mentioned https in my post so that was what I had in mind.
-
Not sure if this works for you as this falls under the "not solving your problem but just proposing something else" category, but we deal with this commonly and solve it by just using SSH/SFTP. It's not wget/curl, but it's super simple and standard.
-
@Pete-S said in How to let only customers download files with wget/curl?:
@stacksofplates said in How to let only customers download files with wget/curl?:
@Pete-S said in How to let only customers download files with wget/curl?:
@stacksofplates said in How to let only customers download files with wget/curl?:
@Obsolesce said in How to let only customers download files with wget/curl?:
Some sort of authentication is the only way if the app doesn't support built in updating of some kind.
Yeah as @Obsolesce said auth is prob best. Basic auth would be the easiest.
Just
curl -u 'user:pass' https://mything.com/filesWget is
--userand--passwordDoes using digest auth make it better? Perhaps I don't have to put the password in clear text that way.
I would say digest isn't any better. With basic auth, yes you send the creds across the wire, but I believe digest still requires MD5 hashing.
This is going to depend a lot on what your server allows (if you're not writing an API from scratch) and the pain for your clients.
A JWT is better (or some other type of bearer token) but it's extra work for your clients if they're using just curl or wget.
My idea on the client side is that it has to be done with standard tools available on basically every linux distro such as wget, curl etc. Basically a couple of lines that an admin could copy & paste into ssh from instructions.
If you have to enter a password / username somewhere it should only need to be done once.
Besides basic and digest authentication I also had a look at SSL/TLS client certificates. I didn't know about that but it's been around for a long time. Basically the server is authenticated by the server certificate and in the same manner the client can be authenticated by the client certificate.
The webserver would then have to verify that the client has a valid SSL certificate and is allowed to download whatever file it is.
A common thing to do is export your creds as env vars. You can even store them in a script with the export commands in the script and just source that script. Then just use
-u $user:$passand that means you have to only do it once. -
@scottalanmiller said in How to let only customers download files with wget/curl?:
Not sure if this works for you as this falls under the "not solving your problem but just proposing something else" category, but we deal with this commonly and solve it by just using SSH/SFTP. It's not wget/curl, but it's super simple and standard.
Yes, that could also work but I'm looking for a https solution so you do partial downloads, have scripts serve the data and what not.
For some reason it also seems like file transfer over an ssh session is always slow at least when were talking lots of data.
-
@Pete-S said in How to let only customers download files with wget/curl?:
For some reason it also seems like file transfer over an ssh session is always slow at least when were talking lots of data.
Even with SFTP or RSYNC? It might not be HTTP fast, but it's usually pretty decent.
-
Why not force https, then allow them to wget/curl with basic authentication using something they already know like Custer number and company name (for example). That can always be logged and tracked, alerted on if it's abused.
-
If you did something like host the files in an Azure Storage Account then they could download using an SAS token that's passed in as a request parameter
-
@Obsolesce said in How to let only customers download files with wget/curl?:
Why not force https, then allow them to wget/curl with basic authentication using something they already know like Custer number and company name (for example). That can always be logged and tracked, alerted on if it's abused.
This is what I have been doing in my password manager. Each request is logged with the username and the endpoint.
-
@Pete-S said in How to let only customers download files with wget/curl?:
have scripts serve the data and what not.
Wait are you talking about CGI scripts?
-
@Obsolesce said in How to let only customers download files with wget/curl?:
Why not force https, then allow them to wget/curl with basic authentication using something they already know like Custer number and company name (for example). That can always be logged and tracked, alerted on if it's abused.
It would work but I prefer if I could get SSL client certificates up and running. You'd have to install the certificate once but then you're authorized to access "your" files on the web server. The webserver can log and track you and decide what you are allowed to access because of the client certificate.
-
@stacksofplates said in How to let only customers download files with wget/curl?:
@Pete-S said in How to let only customers download files with wget/curl?:
have scripts serve the data and what not.
Wait are you talking about CGI scripts?
Yes, that's a possibility when you are using a webserver, instead of ssh.
If you access a file over ssh, AFAIK the file is a static file and it is what it is.
If you however access a file over https, you can have a script on the webserver delivering you the file and you can send parameters to it. For instance :
wget -o install.sh "https://xyz.com/my_special_install_script.py?os=CentOS7&special=2&customer=2432"You just have a gazilion options when you connect over a webserver.
-
@Pete-S said in How to let only customers download files with wget/curl?:
@stacksofplates said in How to let only customers download files with wget/curl?:
@Pete-S said in How to let only customers download files with wget/curl?:
have scripts serve the data and what not.
Wait are you talking about CGI scripts?
Yes, that's a possibility when you are using a webserver, instead of ssh.
If you access a file over ssh, AFAIK the file is a static file and it is what it is.
If you however access a file over https, you can have a script on the webserver delivering you the file and you can send parameters to it. For instance :
wget -o install.sh "https://xyz.com/my_special_install_script.py?os=CentOS7&special=2&customer=2432"You just have a gazilion options when you connect over a webserver.
Yeah I thought you were providing files through just a default webserver. So while CGI isn't insecure by itself, you have a ton of work in securing the scripts you create. You might be better off just writing a small API to hand off the info instead of trying to properly secure CGI scripts.
Another option is a serverless function leveraging the providers authentication to serve the files up.
Here's an example from GCP where you can just check require authentication using their IAM.
-
@stacksofplates said in How to let only customers download files with wget/curl?:
@Pete-S said in How to let only customers download files with wget/curl?:
@stacksofplates said in How to let only customers download files with wget/curl?:
@Pete-S said in How to let only customers download files with wget/curl?:
have scripts serve the data and what not.
Wait are you talking about CGI scripts?
Yes, that's a possibility when you are using a webserver, instead of ssh.
If you access a file over ssh, AFAIK the file is a static file and it is what it is.
If you however access a file over https, you can have a script on the webserver delivering you the file and you can send parameters to it. For instance :
wget -o install.sh "https://xyz.com/my_special_install_script.py?os=CentOS7&special=2&customer=2432"You just have a gazilion options when you connect over a webserver.
Yeah I thought you were providing files through just a default webserver. So while CGI isn't insecure by itself, you have a ton of work in securing the scripts you create. You might be better off just writing a small API to hand off the info instead of trying to properly secure CGI scripts.
Another option is a serverless function leveraging the providers authentication to serve the files up.
Here's an example from GCP where you can just check require authentication using their IAM.
Thanks, I had the intention of letting the webserver authenticate and in most cases provide a static file directly or when needed invoke a script that will provide dynamic content.
I haven't checked nginx yet but apache can check client SSL certificates easily.
