Timeout in .sh file

Lakshmana

@lakshmana said in Timeout in .sh file:

@scottalanmiller said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller If i am giving the .sh file to be runned with ny collegues system i cannot share that the executable file since it may lead Infosec Issue

How will that work regardless of the tool used?

@scottalanmiller said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller If i am giving the .sh file to be runned with ny collegues system i cannot share that the executable file since it may lead Infosec Issue

How will that work regardless of the tool used?

That .sh file should download the data by terminal or by Browser inside the machine without using credentials!!!

Ok, full stop. What happens here is that your web server simply tells you: "Hey, dude, I would love to give you that file, but your f'ing unauthorized, so just leave me alone. I won't give you the file unless you can present me a valid auth".

In other words: HTTP 401 Unauthorized.

So what needs to be done is simple: we need to authenticate against the server. This does not relate to the client side in any way. You will see the same behavior in Internet Explorer, Edge, Chrome, misc. PowerShell cmdlets or whatever else you may use to access that file.

Like SAM said, let us stop talking about client side tools for now. Let us instead discuss very basic authentication mechanisms.

I understood but in Browser the file will download from the cache credentials since the cookies make to download the data Right?????

thwr

@lakshmana said in Timeout in .sh file:

@thwr said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller If i am giving the .sh file to be runned with ny collegues system i cannot share that the executable file since it may lead Infosec Issue

How will that work regardless of the tool used?

@scottalanmiller said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller If i am giving the .sh file to be runned with ny collegues system i cannot share that the executable file since it may lead Infosec Issue

How will that work regardless of the tool used?

That .sh file should download the data by terminal or by Browser inside the machine without using credentials!!!

Ok, full stop. What happens here is that your web server simply tells you: "Hey, dude, I would love to give you that file, but your f'ing unauthorized, so just leave me alone. I won't give you the file unless you can present me a valid auth".

In other words: HTTP 401 Unauthorized.

So what needs to be done is simple: we need to authenticate against the server. This does not relate to the client side in any way. You will see the same behavior in Internet Explorer, Edge, Chrome, misc. PowerShell cmdlets or whatever else you may use to access that file.

Like SAM said, let us stop talking about client side tools for now. Let us instead discuss very basic authentication mechanisms.

I understood but in Browser the file will download from the cache credentials since the cookies make to download the data Right?????

Yep. And it's a very bad idea to rely on something you can't control: a users cached cookies.

scottalanmiller

@lakshmana said in Timeout in .sh file:

@thwr said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller If i am giving the .sh file to be runned with ny collegues system i cannot share that the executable file since it may lead Infosec Issue

How will that work regardless of the tool used?

@scottalanmiller said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller If i am giving the .sh file to be runned with ny collegues system i cannot share that the executable file since it may lead Infosec Issue

How will that work regardless of the tool used?

That .sh file should download the data by terminal or by Browser inside the machine without using credentials!!!

Ok, full stop. What happens here is that your web server simply tells you: "Hey, dude, I would love to give you that file, but your f'ing unauthorized, so just leave me alone. I won't give you the file unless you can present me a valid auth".

In other words: HTTP 401 Unauthorized.

So what needs to be done is simple: we need to authenticate against the server. This does not relate to the client side in any way. You will see the same behavior in Internet Explorer, Edge, Chrome, misc. PowerShell cmdlets or whatever else you may use to access that file.

Like SAM said, let us stop talking about client side tools for now. Let us instead discuss very basic authentication mechanisms.

I understood but in Browser the file will download from the cache credentials since the cookies make to download the data Right?????

Right, so that is a reason to avoid that. you need to get the cookie cache and manage it. It's specifically for this kind of reason that cURL is the right tool. I've done exactly this in the past for automation where we needed a login AND a cookie cache and cURL handles this very well.

Lakshmana

@thwr said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@thwr said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller If i am giving the .sh file to be runned with ny collegues system i cannot share that the executable file since it may lead Infosec Issue

How will that work regardless of the tool used?

@scottalanmiller said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@scottalanmiller If i am giving the .sh file to be runned with ny collegues system i cannot share that the executable file since it may lead Infosec Issue

How will that work regardless of the tool used?

That .sh file should download the data by terminal or by Browser inside the machine without using credentials!!!

Ok, full stop. What happens here is that your web server simply tells you: "Hey, dude, I would love to give you that file, but your f'ing unauthorized, so just leave me alone. I won't give you the file unless you can present me a valid auth".

In other words: HTTP 401 Unauthorized.

So what needs to be done is simple: we need to authenticate against the server. This does not relate to the client side in any way. You will see the same behavior in Internet Explorer, Edge, Chrome, misc. PowerShell cmdlets or whatever else you may use to access that file.

Like SAM said, let us stop talking about client side tools for now. Let us instead discuss very basic authentication mechanisms.

I understood but in Browser the file will download from the cache credentials since the cookies make to download the data Right?????

Yep. And it's a very bad idea to rely on something you can't control: a users cached cookies.

This script will not run on users .only on System Administrator machines only

thwr

That doesn't matter at all. There are only a few factors:

Do you need any form of authentication - if so, what exactly? A web based form, a HTTP BASIC AUTH (your browser shows a popup for credentials) or something else (some auth protocols can be tricky)
Can the machine running the script reach the webserver?
Is there any script language available? not required, just a plus
Any download tool available? curl, powershell?

You see, it's pretty simple. Most questions are already answered. We now need to know what kind of authentication your webserver requires. Simply start your favorite browser in incognito mode (shift + ctrl + p in Chrome) and navigate to your URL. There are no cookies or caches involved in that moment. Tell us what you see, maybe create a screenshot.

Lakshmana

@thwr said in Timeout in .sh file:

That doesn't matter at all. There are only a few factors:

Do you need any form of authentication - if so, what exactly? A web based form, a HTTP BASIC AUTH (your browser shows a popup for credentials) or something else (some auth protocols can be tricky)

Can the machine running the script reach the webserver?

Is there any script language available? not required, just a plus

Any download tool available? curl, powershell?

You see, it's pretty simple. Most questions are already answered. We now need to know what kind of authentication your webserver requires. Simply start your favorite browser in incognito mode (shift + ctrl + p in Chrome) and navigate to your URL. There are no cookies or caches involved in that moment. Tell us what you see, maybe create a screenshot.

As checked the link in the browser by incognito window there is not Login Credentials was asked where the file get downloaded without any issues.

1)As of now i dont need any authorization to download the file
2)The machine which is running is able to reach the Webserver
3)No script language.Just making .sh file to download the files consecutively
4)I am trying the process in Ubuntu and same needs to be worked in Mac but tried the Curl which is creating the file which have the same link inside that file once i click on that link the file will be downloading

I need to download the Files automatically in Downloads Test Folder in the .csv format first

thwr

@lakshmana said in Timeout in .sh file:

@thwr said in Timeout in .sh file:

That doesn't matter at all. There are only a few factors:

Do you need any form of authentication - if so, what exactly? A web based form, a HTTP BASIC AUTH (your browser shows a popup for credentials) or something else (some auth protocols can be tricky)

Can the machine running the script reach the webserver?

Is there any script language available? not required, just a plus

Any download tool available? curl, powershell?

You see, it's pretty simple. Most questions are already answered. We now need to know what kind of authentication your webserver requires. Simply start your favorite browser in incognito mode (shift + ctrl + p in Chrome) and navigate to your URL. There are no cookies or caches involved in that moment. Tell us what you see, maybe create a screenshot.

As checked the link in the browser by incognito window there is not Login Credentials was asked where the file get downloaded without any issues.

1)As of now i dont need any authorization to download the file
2)The machine which is running is able to reach the Webserver
3)No script language.Just making .sh file to download the files consecutively
4)I am trying the process in Ubuntu and same needs to be worked in Mac but tried the Curl which is creating the file which have the same link inside that file once i click on that link the file will be downloading

I need to download the Files automatically in Downloads Test Folder in the .csv format first

Ah, very good. So we can safely ignore any authentication. Mostly. It's still possible that there's some kind of authentication you can't see this way, like IP source or user agent checks for example. A user agent string is basically a way to tell a webserver which browser is asking for files. BTW: sh / bash is a language on its own But let's skip that for now.

So what exactly happens when you just enter

wget -O /home/user/download/file1.csv https://your.webserver/folder/file1.csv

Is there any error message?

Lakshmana

@thwr another doubt is whether tiny url of the internal site will be working in Wget?

thwr

@lakshmana said in Timeout in .sh file:

@thwr another doubt is whether tiny url of the internal site will be working in Wget?

why not? Oh, wait. Please try:

curl -JLO https://your.webserver/folder/file1.csv

Lakshmana

@thwr said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:

@thwr another doubt is whether tiny url of the internal site will be working in Wget?

why not? Oh, wait. Please try:
curl -JLO https://your.webserver/folder/file1.csv

The data is downloading now.How to open that .csv from the terminal ?
The machine is GUI only but I accessing the same in putty ,if i need to confirm tomorrrow once present at Office

thwr

@lakshmana said in Timeout in .sh file:

@thwr said in Timeout in .sh file:
@lakshmana said in Timeout in .sh file:

@thwr another doubt is whether tiny url of the internal site will be working in Wget?

why not? Oh, wait. Please try:
curl -JLO https://your.webserver/folder/file1.csv
The data is downloading now.How to open that .csv from the terminal ?
The machine is GUI only but I accessing the same in putty ,if i need to confirm tomorrrow once present at Office

Great. Erm, cat file.csv for example?

Don't you want to know what caused you so much trouble?

Lakshmana

@thwr said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:
@thwr said in Timeout in .sh file:
@lakshmana said in Timeout in .sh file:

@thwr another doubt is whether tiny url of the internal site will be working in Wget?

why not? Oh, wait. Please try:
curl -JLO https://your.webserver/folder/file1.csv
The data is downloading now.How to open that .csv from the terminal ?
The machine is GUI only but I accessing the same in putty ,if i need to confirm tomorrrow once present at Office
Great. Erm, cat file.csv for example?

Don't you want to know what caused you so much trouble?

The .csv file is there but the file is having another link to download the original .csv file

thwr

@lakshmana said in Timeout in .sh file:

@thwr said in Timeout in .sh file:
@lakshmana said in Timeout in .sh file:
@thwr said in Timeout in .sh file:
@lakshmana said in Timeout in .sh file:

@thwr another doubt is whether tiny url of the internal site will be working in Wget?

why not? Oh, wait. Please try:
curl -JLO https://your.webserver/folder/file1.csv
The data is downloading now.How to open that .csv from the terminal ?
The machine is GUI only but I accessing the same in putty ,if i need to confirm tomorrrow once present at Office
Great. Erm, cat file.csv for example?

Don't you want to know what caused you so much trouble?
The .csv file is there but the file is having another link to download the original .csv file

what do you mean by "another link"? Is the link the content of the file you've downloaded?

Lakshmana

@thwr said in Timeout in .sh file:

@lakshmana said in Timeout in .sh file:
@thwr said in Timeout in .sh file:
@lakshmana said in Timeout in .sh file:
@thwr said in Timeout in .sh file:
@lakshmana said in Timeout in .sh file:

@thwr another doubt is whether tiny url of the internal site will be working in Wget?

why not? Oh, wait. Please try:
curl -JLO https://your.webserver/folder/file1.csv
The data is downloading now.How to open that .csv from the terminal ?
The machine is GUI only but I accessing the same in putty ,if i need to confirm tomorrrow once present at Office
Great. Erm, cat file.csv for example?

Don't you want to know what caused you so much trouble?
The .csv file is there but the file is having another link to download the original .csv file
what do you mean by "another link"? Is the link the content of the file you've downloaded?

The example is "facebook,com/example.mp4" that example.mp4 is the file which needs to be downloaded but the file download as example and inside that example file Click here Link(facebook.com/example.mp4) is present and i need to click that to download that file

thwr

@lakshmana said in Timeout in .sh file:

@thwr said in Timeout in .sh file:
@lakshmana said in Timeout in .sh file:
@thwr said in Timeout in .sh file:
@lakshmana said in Timeout in .sh file:
@thwr said in Timeout in .sh file:
@lakshmana said in Timeout in .sh file:

@thwr another doubt is whether tiny url of the internal site will be working in Wget?

why not? Oh, wait. Please try:
curl -JLO https://your.webserver/folder/file1.csv
The data is downloading now.How to open that .csv from the terminal ?
The machine is GUI only but I accessing the same in putty ,if i need to confirm tomorrrow once present at Office
Great. Erm, cat file.csv for example?

Don't you want to know what caused you so much trouble?
The .csv file is there but the file is having another link to download the original .csv file
what do you mean by "another link"? Is the link the content of the file you've downloaded?
The example is "facebook,com/example.mp4" that example.mp4 is the file which needs to be downloaded but the file download as example and inside that example file Click here Link(facebook.com/example.mp4) is present and i need to click that to download that file

So what you've downloaded is actually a HTML page containing a button with a link, for example?

Lakshmana

Yes

thwr

@lakshmana said in Timeout in .sh file:

Yes

Are you able to parse that page? Regex maybe? Just hope that the page never changes.

Lakshmana

@thwr Sorry I cant understand

thwr

@lakshmana said in Timeout in .sh file:

@thwr Sorry I cant understand

From what I understand, you've downloaded a HTML formated file. That's what your browser reads whenever you visit a website with a browser. It contains information about the content and structure of a web page.

You could read that file yourself and grab the link that's contained. Regular expression pattern matching is a perfectly valid, easy and straight forward way to do so. You'll find tons of information with your search engine of choice.

Please understand that people here at ML for example will be happy to help you in many ways, like pointing you into the right direction or reviewing your ideas. But I simply can't do your job. It's you who gets the paycheck for solving this problem, not me or anyone else.

Let me say that again: I would be happy to provide you with directions, but it's you who needs to get the work done.

Lakshmana

@thwr Hereafter i should not update any doubts in this post only Right !!!!