Need live network monitoring

JaredBusch

I have a customer with an Ubiquiti ER-4 that is running into bandwidth issues

We will be expanding their internet service, but that takes time as always.

So until that comes through, what would be the best solution I can implement to get details on what is going through the router?

Prior to this, the built in stats have been enough detail to handle minor issues.

But now, this is the problem. I reset the stats about 4 hours ago.
Since then, this computer has put all this "other" through the router.
I do assume that TX means sent from the router to the client. So it is download. But what/how/where from?

The site is also consistently hitting 50mbps upload on the WAN for unknown reasons.

Nothing has really changed when I look at the router historically in UNMS.
But the last two days it has been a huge problem.

JaredBusch

So the problem is ZeroTier 1.6.0
It was pushed to chocolately on November 22.

All of the client machines updated on the 22 & 23 automatically via chocolatey.

A branch location called today reporting issues and their only have 6 computers. It was much easier to analyze.

Only 2 computers were causing problems. The only thing the same on those computers but different from the rest was ZeroTier. Stopped the ZeroTier service and poof, network normal.

So I hopped in to ScreenConnect and globally stopped the service at the main office.

#!ps
#maxlength=200000
#timeout=90000
Stop-Service -Name ZeroTierOneService

Magic. Things are normalized. Now to troubleshoot ZeroTier.

JaredBusch

Historic view, 24 hours:

WTF is this 100mbps on the LAN? What would be hitting the LAN interface of the router that is not coming through the WAN? Sure local traffic hits high speed, but none of that should hit the router, it is all within the /23 subnet.

JaredBusch

month view. Barely touched "normally", until yesterday.

Dashrender

Transmit is higher than receive? I'm betting someone is infected... possibly with a ransomware, and the baddies might be stealing your data right now. Just a thought at least.

Can you narrow it down to what client is generating that traffic?

JaredBusch

@Dashrender said in Need live network monitoring:

Transmit is higher than receive? I'm betting someone is infected... possibly with a ransomware, and the baddies might be stealing your data right now. Just a thought at least.

Can you narrow it down to what client is generating that traffic?

The long views are smoothed averages..

once afternoon hit and people start leaving it dropped off.

now at 5pm, there are still little spokes. I'm hoping to narrow it down now that the network is less busy.

jt1001001

Saw this behavior when a user installed a Canon printer at home and put the software on their work laptop. The software kept scanning for a printer. The software slammed the router but the traffic never left it.

JaredBusch

@Dashrender said in Need live network monitoring:

Transmit is higher than receive?

If you wwere commenting about the TX > RX on the device view, recall that it is TX from the router to the endpoint. aka the endpoint downloading something.

DustinB3403

Is it possibly torrent traffic? Do you have anything like zabbix on the end points and servers to see what they are doing?

JaredBusch

@JaredBusch said in Need live network monitoring:

@Dashrender said in Need live network monitoring:

Transmit is higher than receive?

If you wwere commenting about the TX > RX on the device view, recall that it is TX from the router to the endpoint. aka the endpoint downloading something.

This assumption was wrong. I jsut tested by downloading the current fedora desktop ISO to the user machine..

So user is sending data.

JaredBusch

@DustinB3403 said in Need live network monitoring:

Do you have anything like zabbix on the end points and servers to see what they are doing?

No, because they are not paying for type of ongoing work.

That said I could set it up if that is the easiest option for a hopefully one off.

DustinB3403

@JaredBusch it would be pretty quick to get setup, the servers and endpoints are going to be the most time consuming for getting each setup.

Edit installing each agent

dafyre

@JaredBusch said in Need live network monitoring:

@DustinB3403 said in Need live network monitoring:

Do you have anything like zabbix on the end points and servers to see what they are doing?

No, because they are not paying for type of ongoing work.

That said I could set it up if that is the easiest option for a hopefully one off.

If you can identify which device it is, just block it from the internet until you can get your remote tools installed on it and check it?

JaredBusch

Killed the most likely suspect machine and things calmed.

Now nothing abnormal. The big upload spike is offsite backups sending.

krzykat

@JaredBusch Sure would be nice if the Edgerouter DPI would know about off-site backups. Would have allowed you to isolate it real quick.

JaredBusch

@krzykat said in Need live network monitoring:

@JaredBusch Sure would be nice if the Edgerouter DPI would know about off-site backups. Would have allowed you to isolate it real quick.

That has nothing to do with the problem.

The problem occurred all day long, while no backups were running.
Here is the DPI for the offsite backup.
Exactly as I expect, never knew why it was classified as LE. But it always has been.

JaredBusch