File Parsing Magic

anthonyh

I have a log that I need to pull some data from. The entries look like this:

2016-04-21 07:11:34,512 INFO [qtp509886383-547489:https://10.39.6.4:443/service/soap/SyncRequest] [name=user@domain.org;mid=66;ip=10.39.248.191;ua=ZCO/8.6.0.1320 (6.2.9200 en-US) P1248 T25c0;] soap - SyncRequest elapsed=3

What I need to do is pull the text between name= and ip= and ; so that I have the following:

user@domain.org;10.39.248.191

These log entries are variable lengths (various URLs), but the desire is to import the user and IP into a MySQL DB so I can pull distinct results.

Once I can get it into a delimited format I can take it from there.

Oh ML magicians, what do you suggest?

scottalanmiller

Put the file that you want to process into file2parse and this will do the rest...

#!/bin/bash

while read line; do
  echo $(echo $line | cut -d'=' -f2 | cut -d';' -f1)";"$(echo $line | cut -d'=' -f4 | cut -d';' -f1)
done < file2parse

scottalanmiller

Using the line that you supplied, here is a sample run...

$ ./parsingtool 
user@domain.org;10.39.248.191

scottalanmiller

You can process the output with uniq to make it easier to get unique results without needing to use MySQL.

Jason

Is this on windows or linux you didn't specify.

On windows use Powershell, on linux use Bash.

scottalanmiller

@Jason said in File Parsing Magic:

Is this on windows or linux you didn't specify.

On windows use Powershell, on linux use Bash.

You've got BASH on Windows now, right?

scottalanmiller

I assumed Linux since he is using MySQL which you would only run on Linux normally.

Jason

@scottalanmiller said in File Parsing Magic:

You've got BASH on Windows now, right?

You mean that pointless thing that want interact with anything else.. yeah. It's pointless.

scottalanmiller

@Jason said in File Parsing Magic:

@scottalanmiller said in File Parsing Magic:

You've got BASH on Windows now, right?

You mean that pointless thing that want interact with anything else.. yeah. It's pointless.

It should still parse text, though. In theory. Maybe.

travisdh1

@scottalanmiller said in File Parsing Magic:

@Jason said in File Parsing Magic:

@scottalanmiller said in File Parsing Magic:

You've got BASH on Windows now, right?

You mean that pointless thing that want interact with anything else.. yeah. It's pointless.

It should still parse text, though. In theory. Maybe.

Assuming it has access to the base system and not just it's own container. From everything I've read so far it's more like a Docker container than actually BASH on Windows... if you want that you're still stuck with cygwin or the like.

Jason

@travisdh1 said in File Parsing Magic:

@scottalanmiller said in File Parsing Magic:

@Jason said in File Parsing Magic:

@scottalanmiller said in File Parsing Magic:

You've got BASH on Windows now, right?

You mean that pointless thing that want interact with anything else.. yeah. It's pointless.

It should still parse text, though. In theory. Maybe.

Assuming it has access to the base system and not just it's own container. From everything I've read so far it's more like a Docker container than actually BASH on Windows... if you want that you're still stuck with cygwin or the like.

Parse text sure.. getting the text he wants with a script into it in the first place, not so sure.

Romo

@travisdh1 You have access to all the files in Windows from /mnt/c , so yeah you can easily parse the text with the script provided by @scottalanmiller

travisdh1

@Romo said in File Parsing Magic:

@travisdh1 You have access to all the files in Windows from /mnt/c , so yeah you can easily parse the text with the script provided by @scottalanmiller

Ok, so it's more like cygwin than Docker. Thanks for the correction/confirmation.

jyates

If windows, powershell has split and trim functions.

$this = $this.ToString().Split("name=",2)[1].Split(";",4)
$name = $this[0].split("=",2)[1]
$ip = $this[2].Trim("ip=")

anthonyh

@scottalanmiller said in File Parsing Magic:

Put the file that you want to process into file2parse and this will do the rest...
#!/bin/bash

while read line; do
  echo $(echo $line | cut -d'=' -f2 | cut -d';' -f1)";"$(echo $line | cut -d'=' -f4 | cut -d';' -f1)
done < file2parse

OMG SAM you are the best!

Sorry for not being clear. This is all under Linux VMs on-prem in my own environment (XenServer).

anthonyh

@scottalanmiller said in File Parsing Magic:

Put the file that you want to process into file2parse and this will do the rest...
#!/bin/bash

while read line; do
  echo $(echo $line | cut -d'=' -f2 | cut -d';' -f1)";"$(echo $line | cut -d'=' -f4 | cut -d';' -f1)
done < file2parse

This works 75% of the time, but it looks like some log entries show when a user is syncing an item shared by another user, which does not result in the desired output.

mailbox.log.2016-04-19:2016-04-19 01:27:53,338 INFO [qtp509886383-480009:https://10.39.6.4:443/service/soap/SyncRequest] [name=user@domain.org;aname=shareditem@domain.org;mid=14;ip=10.39.253.62;ua=ZCO/8.6.0.1320 (6.1.7601 SP1 en-US) P9b4 T1404;] soap - SyncRequest elapsed=4

What happens here is you get the following:

user@domain.org;14

Desired output is:

user@domain.org;10.39.253.62

scottalanmiller

That's because your log format changed. That second one has more fields in it.

RamblingBiped

@scottalanmiller said in File Parsing Magic:

Put the file that you want to process into file2parse and this will do the rest...
#!/bin/bash

while read line; do
  echo $(echo $line | cut -d'=' -f2 | cut -d';' -f1)";"$(echo $line | cut -d'=' -f4 | cut -d';' -f1)
done < file2parse

Wait, I think there is a more important question that needs to be answered now. If you echo an echo, do you get an echoed echo's echo, or do they just cancel each other out and build a strange uncomfortable silence?

RamblingBiped

@RamblingBiped said in File Parsing Magic:

@scottalanmiller said in File Parsing Magic:
Put the file that you want to process into file2parse and this will do the rest...
#!/bin/bash

while read line; do
  echo $(echo $line | cut -d'=' -f2 | cut -d';' -f1)";"$(echo $line | cut -d'=' -f4 | cut -d';' -f1)
done < file2parse
Wait, I think there is a more important question that needs to be answered now. If you echo an echo, do you get an echoed echo's echo, or do they just cancel each other out and build a strange uncomfortable silence?

And to follow up, if you simultaneously echo two echos from a single echo, will your head explode or somehow magically stay intact?

anthonyh

@scottalanmiller

Understood. I need to figure out a way to parse the file so that the process finds "user=" and pulls everything after it until it hits the following ";", then finds "ip=" and pulls everything after it until it hits the following ";"