PDA

View Full Version : Awk and sed bash script help



Annonymous
05-07-2012, 09:13 PM
I posted another thread on linuxquestion forums. LINK: ---> if else then trouble(bash) (http://www.linuxquestions.org/questions/programming-9/if-else-then-trouble-bash-943615/) I am having scripting troubles. I am not knocking that forum because they are seriously awesome but cboard is so much more active!! Probably one of, if not the most active forum i have been on! So i decided to post my question here as well.

I am trying to use awk, sed, and bash commands to parse netstat output and then if ports other than 443 or 80 are open display a UI i wrote. otherwise exit.

UI code:


#include <gtk/gtk.h>

static void destroy_event(GtkWidget *widget, gpointer data) {
gtk_main_quit();
}

static gboolean delete_event(GtkWidget *widget, GdkEvent *event, gpointer data) {
gtk_main_quit();
return FALSE;
}

static void callback( GtkWidget *widget, gpointer data ) {
gtk_main_quit();
}

//To compile append `pkg-config --cflags --libs gtk+-2.0`
int main(int argc, char *argv[]) {

GtkWidget *window;
GtkWidget *table;
GtkWidget *button;
GtkWidget *frame;

gtk_init(&argc, &argv);

window = gtk_window_new(GTK_WINDOW_TOPLEVEL);
gtk_window_set_title(GTK_WINDOW(window), "SECURITY WARNING");
gtk_window_set_position(GTK_WINDOW(window), GTK_WIN_POS_CENTER);
gtk_window_set_default_size(GTK_WINDOW(window), 300, 200);
gtk_widget_show(window);

table = gtk_table_new(10, 10, TRUE);
gtk_container_add(GTK_CONTAINER(window), table);
gtk_widget_show(table);

frame = gtk_frame_new("SUSPICIOUS PORT IS OPEN");
gtk_table_attach_defaults(GTK_TABLE(table), frame, 1, 9, 2, 8);
gtk_frame_set_shadow_type(GTK_FRAME(frame), GTK_SHADOW_IN);
gtk_widget_show(frame);

button = gtk_button_new_with_label("OK");
gtk_table_attach_defaults(GTK_TABLE(table), button, 2, 8, 3, 7);
gtk_widget_show(button);

g_signal_connect_swapped(G_OBJECT(window), "destroy-event",
G_CALLBACK(destroy_event), NULL);

g_signal_connect_swapped(G_OBJECT(window), "delete-event",
G_CALLBACK(delete_event), NULL);

g_signal_connect (button, "clicked",
G_CALLBACK (callback), (gpointer) "cool button");

gtk_main();

return 0;
}


The bash script:


#!/bin/bash

netstat -ant |
awk -F" " '{print $5}' |
awk -F":" '{print $2}' > netstat.txt;
sed '/^$/d' netstat.txt;

PORTS="$(egrep -i "^80|443" /path/to/netstat.txt)"

if [ "$PORTS" ]; then
exit 0;

else
sudo /path/to/ui_executable

fi


What i would like to achieve with the if statement is if the only ports open are 443 and 80, then exit. If "MORE" than 443 and 80 are open, display my UI.

I cannot figure out how to achieve this. Ideas?

Thanks!

oogabooga
05-07-2012, 10:38 PM
With grep you're using -i (case insensitive) when you want -v (invert match).


PORTS=$(netstat -ant | awk -F" " '{print $5}' | awk -F":" '{print $2}' | sed '/^$/d' | grep -Ev "80|443")
if [ "$PORTS" ] ; then echo yep ; else echo nope ; fi


Or a little simpler


if [ "$(netstat -ant | awk '{print $5}' | grep -Eo ":[0-9]+" | grep -Ev ":80|:443")" ] ;
then echo yep ;
else echo nope ;
fi


Actually, that's not really any simpler.

Annonymous
05-07-2012, 10:59 PM
With grep you're using -i (case insensitive) when you want -v (invert match).


PORTS=$(netstat -ant | awk -F" " '{print $5}' | awk -F":" '{print $2}' | sed '/^$/d' | grep -Ev "80|443")
if [ "$PORTS" ] ; then echo yep ; else echo nope ; fi



WOW, so much cleaner than the slop i put together. thanks.

See the problem i am still having is that ports will always evaluate as true. As long as the net is open. Even if i have a torrent site up with high ports open, it will still be true.

I am trying to launch the UI if an open port other than 443 or 80 is open.

I figured, remove all duplicate ports. Remove the asterisk and blank lines. Then count the lines and if there are more than 2, then launch the UI.

oogabooga
05-07-2012, 11:08 PM
Can you give an example of your netstat -ant output?

I'm not sure what you mean that PORTS will always be true?

Annonymous
05-07-2012, 11:29 PM
Well what i am saying is that if i use this:


root@blah:/path# PORTS=$(netstat -ant | awk -F" " '{print $5}' | awk -F":" '{print $2}' | sed '/^$/d' | grep -Ev "0|1|2|3|4|5|6|7|8|9"); if [ "$PORTS" ] ; then echo yep ; else echo nope ; fi
yep
root@blah:/path#


i will always get the yep answer. PORT will always evaluate to true. Well unless the internet is not open.

So what i am trying to accomplish is that if anything other than port 443 and/or 80 is open then display the UI warning. Otherwise, just exit. Ill have the script ran at start up and run every minute im guessing.


I figured i could do something like this:


#!/bin/bash

netstat -ant | awk -F" " '{print $5}' | awk -F":" '{print $2}' | sed '/^$/d' | egrep -E "[0..9]" > netstat.txt;
#-v switch w/grep fam will remove specified.

awk '!x[$0]++' netstat.txt > netstat2.txt && cat netstat2.txt | sed '/^$/d' netstat2.txt;
LINES=`awk 'END {print NR}' netstat2.txt`;
echo "Number of lines = $LINES"

if [ $LINES > 3 ]; then
echo "Greater than 3"

else #if [ $LINES < 3 ]; then
echo "Less than 3"

#fi

fi



But for some reason the script returns greater than 3 every time. Even if only port 80 is open.

I HOPE I AM BEING CLEAR! If not please say so and i will try to better clarify my intention/statement. Thanks.

oogabooga
05-07-2012, 11:34 PM
Try this


CNT=$(netstat -ant | awk '{print $5}' | grep -Eo ":[0-9]+" | grep -Evc ":80|:443")
if [ $CNT == 0 ] ; then echo nope ; else echo run program ; fi

Annonymous
05-07-2012, 11:41 PM
@oogabooga, that works. Thanks. If you would be so kind, could you explain your code? More specifically

this
-Eo ":[0-9]+" and
-Evc ":80|:443" Thanks again.

oogabooga
05-07-2012, 11:45 PM
grep -E is the same as egrep (but I believe it's the preferred modern form).
The o flag means only output the matching text instead of the whole line.
v means invert output (output non-matching lines).
c means count lines.

anduril462
05-07-2012, 11:52 PM
Are you sure you want to print column 5? That's the foreign address/port. If you're looking for what's open on your machine, you should print $4, at least, that's the way it appears on my netstat output.

Also, try dropping the quotes from your test, or explicitly using the -n option in your test:


if [ $PORTS ]; ...
# or
if [ -n "$PORTS" ]; ...

You can read up on subtleties of bash quoting, and in particular, with the test command, here: Quotes and escaping [Bash Hackers Wiki] (http://wiki.bash-hackers.org/syntax/quoting).

EDIT: Another awesome bash reference: Advanced Bash-Scripting Guide (http://tldp.org/LDP/abs/html/).

Annonymous
05-08-2012, 12:03 AM
@anduril462, with 4, yes I will see local ports. The problem with that is, they are all high ports and i wouldnt know the difference. Where as, if i use 5, i can see what ports i am connecting to. Like if i open facebook only, it will show port 443 open in the foreign field. Let's say I were to use 4, how would i know which to parse? Again, they are after all, high random ports.

Correct me if I am wrong though.

@oogabooga, i could just open a terminal and type in the command with the --help switch and it will explain all of that. I was actually talking about the extras that the man pages wouldn't explain. Like The : or the + sign. I will just google it. I appreciate the help you have already provided. I a grateful!

anduril462
05-08-2012, 12:11 AM
@anduril462, with 4, yes I will see local ports. The problem with that is, they are all high ports and i wouldnt know the difference. Where as, if i use 5, i can see what ports i am connecting to. Like if i open facebook only, it will show port 443 open in the foreign field. Let's say I were to use 4, how would i know which to parse? Again, they are after all, high random ports.

Correct me if I am wrong though.

No, you're right. I was playing catch-up, and I guess I skimmed the thread a little too quickly :).

Annonymous
05-08-2012, 12:35 AM
So now could anyone tell me how i could get my script to produce the same results as @oogabooga's?



#!/bin/bash

netstat -ant | awk -F" " '{print $5}' | awk -F":" '{print $2}' | sed '/^$/d' | egrep "0|1|2|3|4|5|6|7|8|9" > netstat.txt;
#CNT=$(netstat -ant | awk '{print $5}' | grep -Eo ":[0-9]+" | grep -Evc ":80|:443")
#-v switch w/grep fam will remove specified.

LINES=$(awk '!x[$0]++' netstat.txt > netstat2.txt && cat netstat2.txt | sed '/^$/d' netstat2.txt | awk 'END {print NR}' netstat2.txt)
NLINES=`echo $LINES`;

if [ $NLINES < 3 ]; then
echo "Your safe"

else if [ $NLINES > 3 ]; then
/home/codecaine21/Documents/warning

fi

fi


No matter how i re-arrange the script, the result "always" prints your safe. Never runs the UI warning. Even if i have 73 different ports open.

MY PROBLEM SEEMS TO COME FROM MY COMPARISON in my if statement. If i echo $LINES or $NLINES they will output the same number but when try to compare either variables to the number 3, it seems to make no difference.
I even switched the conditions to the if statement around just to see if its behavior would change, it didn't. Which leads me believe that...well idk, I'm STUMPED!

anduril462
05-08-2012, 09:29 AM
Try reading the links I gave you. The bash-hackers.org has a link to the test command near the bottom of the quoting page I liked you to that will explain your problem with using <. The Advanced Bash-Scripting guide has a whole section on tests that might explain why your code doesn't work. Heck, to save you time, it's section 7.3 of the ABS guide.

Also, you may want to make sure you cover the case where $NLINES is exactly 3, and you should look up the elif command for a better "else if" construct that doesn't nest.

oogabooga
05-08-2012, 10:37 AM
i could just open a terminal and type in the command with the --help switch and it will explain all of that. I was actually talking about the extras that the man pages wouldn't explain. Like The : or the + sign. I will just google it.
I know. I felt kind of silly posting that but I wasn't sure what you didn't understand.

The : is simply a : (i.e., match a colon).
The + means to match "one or more" of the previous item (it's a basic regex thing, which you could also google).

Here's a GNU Grep 2.12 (http://www.gnu.org/software/grep/manual/grep.html) reference. It describes the switches and the regex's.

Annonymous
05-08-2012, 03:50 PM
I know. I felt kind of silly posting that but I wasn't sure what you didn't understand.

The : is simply a : (i.e., match a colon).
The + means to match "one or more" of the previous item (it's a basic regex thing, which you could also google).

Here's a GNU Grep 2.12 (http://www.gnu.org/software/grep/manual/grep.html) reference. It describes the switches and the regex's.

@oogabooga, Don't feel silly. I have a tendency to post ambiguous questions! Just ask anyone, or better yet read some of the posts i have made here. I appreciate the responses and help you have provided though. Thank you!

@Anduril, i will revisit the code in a little. I am downloading a movie from thepiratebay, so i will most likely have 20+ ports open. I have an idea as to why it is not properly working. I figure i need to use the -gt or -lt switches instead of the string comparison operator(> or <) i am using.

We will see in a little while though. Thanks!

Annonymous
05-08-2012, 09:40 PM
Thanks Anduril for a push in the right direction for the numerical comparison. Got it working. It's fully functional and running.

Final code:


#!/bin/bash

netstat -ant | awk -F" " '{print $5}' | awk -F":" '{print $2}' | sed '/^$/d' | grep -E "0|1|2|3|4|5|6|7|8|9" > netstat.txt;

awk '!x[$0]++' netstat.txt > netstat2.txt && cat netstat2.txt | sed '/^$/d' netstat2.txt;
LINES=$(awk 'END {print NR}' netstat2.txt)
NLINES=`echo $LINES`;

THREE=3;

if [ $NLINES -lt $THREE ]; then
echo "Your safe"

else
echo "[danger]";
/path/to/ui_executable;

fi

oogabooga
05-09-2012, 05:53 PM
You don't need the sed calls to remove empty lines since the grep will do that (since it only matches lines with at least one digit). And you don't need the files, either. Also, the variables THREE and NLINES are both unnecessary. If the idea with THREE was to name a "magic number" then it really doesn't accomplish much! Maybe LIMIT, but I think a plain 3 is best.


#!/bin/bash

LINES=$(
netstat -ant |
awk '{print $5}' |
awk -F":" '{print $2}' |
grep -E "[0-9]" |
awk '!x[$0]++' |
awk 'END {print NR}' )

if [ $LINES -lt 3 ]
then
echo "You're safe"
else
echo "[danger]"
/path/to/ui_executable
fi


It's still pretty inefficient, 6 program invocations (4 awks!).

A possibility that doesn't use awk is this (but I don't know what the cut columns should be since I'm not running Linux at the moment; so you'll have to adjust the 50-70 to cut the correct columns).


LINES=$(
netstat -ant |
cut -c 50-70 |
grep -Eo ":[0-9]+" |
sort -u |
wc -l )

Annonymous
05-09-2012, 09:25 PM
You don't need the sed calls to remove empty lines since the grep will do that (since it only matches lines with at least one digit). And you don't need the files, either. Also, the variables THREE and NLINES are both unnecessary. If the idea with THREE was to name a "magic number" then it really doesn't accomplish much! Maybe LIMIT, but I think a plain 3 is best.


#!/bin/bash

LINES=$(
netstat -ant |
awk '{print $5}' |
awk -F":" '{print $2}' |
grep -E "[0-9]" |
awk '!x[$0]++' |
awk 'END {print NR}' )

if [ $LINES -lt 3 ]
then
echo "You're safe"
else
echo "[danger]"
/path/to/ui_executable
fi


It's still pretty inefficient, 6 program invocations (4 awks!).

A possibility that doesn't use awk is this (but I don't know what the cut columns should be since I'm not running Linux at the moment; so you'll have to adjust the 50-70 to cut the correct columns).


LINES=$(
netstat -ant |
cut -c 50-70 |
grep -Eo ":[0-9]+" |
sort -u |
wc -l )



Yeah I see that now. I removed the sed command, and removed the field separator in the first awk command.

I am taking a different approach too. I was thinking, what if the internet something other than 443 and 80 are running. For instance, 443 and 1337? That fits the safe criteria of 2 lines. Though, it is actually quite the opposite! So, instead of searching for the number of lines, I used an inverted egrep search to parse out 443 and 80. Then if there are any remaining ports, launch the warning UI.



#!/bin/bash

netstat -ant | awk '{print $5}' | awk -F":" '{print $2}' | awk '!x[$0]++' | grep -E "[0-9]" > netstat.txt; egrep -v "^(443|80)" netstat.txt > netstat2.txt;

echo "[OPEN PORTS]"

LINES=$(awk 'END {print NR}' netstat2.txt)

THREE=1;

if [ $LINES -lt $THREE ]; then
cat netstat.txt;
exit 0;

else
cat netstat.txt;
sudo /path/to/ui_executable

fi


I also removed the unecissary $NLINES variable as well.

I WOULD LIKE TO ADD MORE TO THE GUI! Maybe a button to turn it off. Lets say I am downloading a torrent or using an ftp site; I wouldn't want an annoying pop up every minute. So I would make a button to act as a kill switch! What do you think?

anduril462
05-10-2012, 12:03 AM
I like the idea of adding to the GUI and the functionality overall, but it's going to get complicated, at least if you want it to be reasonably useful. Since this is to alert you of unexpected/unauthorized network usage, you wont want to shut it all the way off while downloading a big torrent, which may take hours. Thus you would only want to ignore traffic to that port for that process. But then, what if that process forks for some reason (and ends up with a new pid). I understand this is just for personal use, but I'd say it's worth thinking it through a bit and coming up with a solid plan before doing much more coding. It will get much harder to do via a simple shell script as you add functionality, think about alternatives. Consider too checking more often than 60 seconds. I can send a lot of malicious data in under 60 seconds, that you wont detect.

Here's a list of some common stuff that I could think of that you might not want to flag.


Torrents
Streaming music/media
SSH/SCP
SFTP, NFS
SMTP/POP
NetBIOS/Samba if you connect to Windows shares with your Linux machine
Instant message
VMWare
Video games

Annonymous
05-10-2012, 12:40 AM
I like the idea of adding to the GUI and the functionality overall, but it's going to get complicated, at least if you want it to be reasonably useful. Since this is to alert you of unexpected/unauthorized network usage, you wont want to shut it all the way off while downloading a big torrent, which may take hours.
Yeah, that makes sense. See, my reasoning was that if I were to download a movie from thepiratebay and have all kinds of high random ports open; i would be aware of this and not want a pesky pop up annoying the crap out of me every minute!

Reading the very valid points you have made, have changed my outlook completely. I did just add a kill button using the execl function. Which i am going to keep.


Thus you would only want to ignore traffic to that port for that process. But then, what if that process forks for some reason (and ends up with a new pid). I understand this is just for personal use, but I'd say it's worth thinking it through a bit and coming up with a solid plan before doing much more coding. It will get much harder to do via a simple shell script as you add functionality, think about alternatives. Consider too checking more often than 60 seconds. I can send a lot of malicious data in under 60 seconds, that you wont detect.
You are absolutely right. I need to think it through if i am going to add anything useful!

I actually wanted to add a great deal of functionality though. Like for starters, the ability to manually enter the ports that i want to flag or something along those lines. As of the moment, I am only monitoring 2 ports; 443 and 80. Which is minimal. So, that's just one idea for starters. I can definitely see the bash script becoming an issue in the future. Probably quite messy! There has to be a better way to do it. Python maybe??




Here's a list of some common stuff that I could think of that you might not want to flag.





Torrents
Streaming music/media
SSH/SCP
SFTP, NFS
SMTP/POP
NetBIOS/Samba if you connect to Windows shares with your Linux machine
Instant message
VMWare
Video games


Those are great things to flag. All to common exploitable services. Thanks!