oneliner-cmds

Useful one liner commands for data parsing

Test input:

┌──(natraj㉿natraj-kali)-[~]
└─$ ifconfig eth0                                                     
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.195.128  netmask 255.255.255.0  broadcast 192.168.195.255
        inet6 fe80::20c:29ff:fe34:5c1c  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:34:5c:1c  txqueuelen 1000  (Ethernet)
        RX packets 124  bytes 8243 (8.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 34  bytes 3927 (3.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

To extract IP address

┌──(natraj㉿natraj-kali)-[~]
└─$ ifconfig eth0 | grep -Po "(?i)\binet (\d+\.){3}\d+\b" | cut -d" " -f2
192.168.195.128

To extract netmask (similar to above)

┌──(natraj㉿natraj-kali)-[~]
└─$ ifconfig eth0 | grep -Po "(?i)\bnetmak (\d+\.){3}\d+\b" | cut -d" " -f2
255.255.255.0

To extract mac address

┌──(natraj㉿natraj-kali)-[~]
└─$ ifconfig eth0 | grep -Po "(?i)\bether ([\da-f]{2}:){5}[\da-f]{2}\b" | cut -d" " -f2
00:0c:29:34:5c:1c

To extract IPv6 address

┌──(natraj㉿natraj-kali)-[~]
└─$ ifconfig eth0 | grep -Po "(?i)\binet6 [\da-f:]+\b" | cut -d" " -f2
fe80::20c:29ff:fe34:5c1c

Simple oneliner to fetch all unique URLs from html file.

┌──(natraj㉿natraj-kali)-[~/Desktop]
└─$ cat test.html | grep -Po "https?://[^\");]+" | sort -u
https://api.search.yahoo.com/data/v3/search?appid=4d234a9d&market=
https://cc.bingj.com/cache.aspx?q=Latest+Indian+news&amp
https://consent.cmp.oath.com/cmp.js
https://geo.yahoo.com/t
https://guce.yahoo.com/privacy-dashboard?locale=en-IN&amp
https://help.yahoo.com/kb/search-for-desktop/yahoo-search-suggestions-sln26943.html
https://in.help.yahoo.com/kb/search-for-desktop
https://in.help.yahoo.com/kb/search-for-desktop/yahoo-search-suggestions-sln26943.html
https://in.images.search.yahoo.com/search/images
https://in.mail.yahoo.com/?.intl=in&amp
...
...

Simple one liner to fetch unique domain names from html file.

┌──(natraj㉿natraj-kali)-[~/Desktop]
└─$ cat test.html | grep -Po "https?://[^/\"]+" | sort -u
https://api.search.yahoo.com
https://cc.bingj.com
https://consent.cmp.oath.com
https://geo.yahoo.com
https://guce.yahoo.com
https://help.yahoo.com
https://in.help.yahoo.com
https://in.images.search.yahoo.com
https://in.mail.yahoo.com
...
...

Simple oneliner to count each domain name called from the html file.

┌──(natraj㉿natraj-kali)-[~/Desktop]
└─$ cat test.html | grep -Po "https?://[^/\"]+" | sort | uniq -c | sort -r
     41 https://in.search.yahoo.com
     30 https://s.yimg.com
      7 https://timesofindia.indiatimes.com
      6 https://cc.bingj.com
      3 https://www.myfonts.com
      3 https://in.news.search.yahoo.com
      3 https://in.help.yahoo.com
      2 https://sports.ndtv.com
      2 https://legal.yahoo.com
      2 https://geo.yahoo.com
      1 https://www.ndtv.com
      1 https://www.indiatoday.in
      1 https://login.yahoo.com
      1 https://indianexpress.com
      1 https://in.video.search.yahoo.com
      1 https://in.mail.yahoo.com
...
...

To count HTML tags count from given html file.

┌──(natraj㉿natraj-kali)-[~/Desktop]
└─$ cat test.html | grep -Po "<\w+>" | sort | uniq -c | sort -r 
     86 <b>
     25 <span>
     15 <tr>
     14 <li>
     13 <strong>
      6 <div>
      4 <style>
      3 <p>
      2 <ul>
      2 <tbody>
      2 <script>
      1 <title>
      1 <noscript>
      1 <ins>
      1 <head>
      1 <form>
      1 <footer>
      1 <button>

Collect and append a string("test") to all collected domain names.

┌──(natraj㉿natraj-kali)-[~/Desktop]
└─$ cat test.html | grep -Po "https?://[^/\"]+" | sort -u | sed -re 's#(https://)#\1test.#'g                  
https://test.api.search.yahoo.com
https://test.cc.bingj.com
https://test.consent.cmp.oath.com
https://test.geo.yahoo.com
https://test.guce.yahoo.com
https://test.help.yahoo.com
https://test.in.help.yahoo.com
https://test.in.images.search.yahoo.com
https://test.in.mail.yahoo.com
https://test.in.news.search.yahoo.com
https://test.in.search.yahoo.com
https://test.in.video.search.yahoo.com
https://test.indianexpress.com
https://test.legal.yahoo.com
https://test.login.yahoo.com
https://test.s.yimg.com
https://test.sports.ndtv.com
https://test.timesofindia.indiatimes.com
https://test.www.indiatoday.in
https://test.www.myfonts.com
https://test.www.ndtv.com

To remove unwanted strings/data from output.(removed "www." from the above output)

┌──(natraj㉿natraj-kali)-[~/Desktop]
└─$ cat test.html | grep -Po "https?://[^/\"]+" | sort -u | sed -re 's#(https://)#\1test.#'g -e 's#www.##g'   
https://test.api.search.yahoo.com
https://test.cc.bingj.com
https://test.consent.cmp.oath.com
https://test.geo.yahoo.com
https://test.guce.yahoo.com
https://test.help.yahoo.com
https://test.in.help.yahoo.com
https://test.in.images.search.yahoo.com
https://test.in.mail.yahoo.com
https://test.in.news.search.yahoo.com
https://test.in.search.yahoo.com
https://test.in.video.search.yahoo.com
https://test.indianexpress.com
https://test.legal.yahoo.com
https://test.login.yahoo.com
https://test.s.yimg.com
https://test.sports.ndtv.com
https://test.timesofindia.indiatimes.com
https://test.indiatoday.in
https://test.myfonts.com
https://test.ndtv.com

To extract particular fields from the output(extracted 2nd and 3rd strings, delimiter='.(dot)')

┌──(natraj㉿natraj-kali)-[~/Desktop]
└─$ cat test.html | grep -Po "https?://[^/\"]+" | sort -u | sed -re 's#(https://)#\1test.#'g -e 's#www.##g' | awk -F"." '{print $2,$3}'
api search
cc bingj
consent cmp
geo yahoo
guce yahoo
help yahoo
in help
in images
in mail
in news
in search
in video
indianexpress com
legal yahoo
login yahoo
s yimg
sports ndtv
timesofindia indiatimes
indiatoday in
myfonts com
ndtv com

To keep the newline between extracted elements

┌──(natraj㉿natraj-kali)-[~/Desktop]
└─$ cat test.html | grep -Po "https?://[^/\"]+" | sort -u | sed -re 's#(https://)#\1test.#'g -e 's#www.##g' | awk -F"." '{print $2 "\n" $3}'
api
search
cc
bingj
consent
cmp
geo
yahoo
guce
yahoo
help
yahoo
in
help
in
images
in
mail
in
news
...
...

Remove blank lines from a file (sed):

sed '/^$/d' file.txt

Extract specific lines between two patterns in a file (awk):

awk '/start_pattern/,/end_pattern/' file.txt

Name		Name	Last commit message	Last commit date
Latest commit History 3 Commits
README.md		README.md

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

oneliner-cmds

Test input:

To extract IP address

To extract netmask (similar to above)

To extract mac address

To extract IPv6 address

Simple oneliner to fetch all unique URLs from html file.

Simple one liner to fetch unique domain names from html file.

Simple oneliner to count each domain name called from the html file.

To count HTML tags count from given html file.

Collect and append a string("test") to all collected domain names.

To remove unwanted strings/data from output.(removed "www." from the above output)

To extract particular fields from the output(extracted 2nd and 3rd strings, delimiter='.(dot)')

To keep the newline between extracted elements

Remove blank lines from a file (sed):

Extract specific lines between two patterns in a file (awk):

About

Releases

Packages

x00itachi/oneliner-cmds

Folders and files

Latest commit

History

Repository files navigation

oneliner-cmds

Test input:

To extract IP address

To extract netmask (similar to above)

To extract mac address

To extract IPv6 address

Simple oneliner to fetch all unique URLs from html file.

Simple one liner to fetch unique domain names from html file.

Simple oneliner to count each domain name called from the html file.

To count HTML tags count from given html file.

Collect and append a string("test") to all collected domain names.

To remove unwanted strings/data from output.(removed "www." from the above output)

To extract particular fields from the output(extracted 2nd and 3rd strings, delimiter='.(dot)')

To keep the newline between extracted elements

Remove blank lines from a file (sed):

Extract specific lines between two patterns in a file (awk):

About

Resources

Stars

Watchers

Forks

Releases

Packages 0

Packages