Tuesday, May 17, 2016

Phantom App Development Tips

I was recently introduced to the Phantom security orchestration framework (version 1.2.113).  The product is very interesting and provides an extensible incident investigation and response automation framework.  The level of documentation and extensibility is reminiscent of Splunk's approach to allowing development and customization.

The ample documentation, novel take on security automation, and a $10,000 app development challenge motivated me to try my hand at developing an app.

I ended up building the Shodan.io API connector for Phantom (github).  It was a great excuse to expand my Python scripting experience while contributing something to the community.  There were some speed bumps along the way.  I thought I'd share my experience since the Phantom community is still growing.

Please leave a comment if you find this useful.

Issue #1: Console & SSH Timeouts 

After working on this app for quite a while, I noticed that my sessions were closing.  I had multiple Putty windows open and the one I left for a while would be gone when I went back to it.  This would even happen with byobu sessions.  The active session would usually stay available.  But the others sessions would close seemingly at random.

My first inclination was that this was due to the SSH timeout as described in this page.  However, changing these settings had no effect.  And, they did not explain the byobu sessions disappearing.


Found here. The closing sessions was due to the TMOUT variable being set.
This can be fixed in the Phantom OVA by deleting or commenting out the lines in the following file:
$ cat /etc/profile.d/timeout.sh
#export TMOUT

I would recommend leaving this setting as-is in your production environment. But this change eliminates a lot of frustrations working via SSH in your dev instance.

The TMOUT value is set in seconds. So you could also change this to a higher value to extend the time it takes to auto-logout.

Issue #2: "Failed to load the app json"

While following the app tutorial (here) I came across the syntax to run/debug the app from the command line.

I remembered the command running correctly when I first ran it.  But for some reason my follow-up attempts kept resulting in the following error:

$ python2.7  samplewhois_connector.py ../test_jsons/sample_whois_domain.json
   ... uninteresting JSON from sample_whois_domain.json ...
Loaded action execution configuration
    "status": "failed",
    "result_summary": {},
    "action_cancelled": false,
    "message": "Failed to load the app json",
    "identifier": "whois_domain",
    "exception_occured": false,
    "result_data": []
Exception AttributeError: "'NoneType' object has no attribute 'path'" in <function 0x7f4a356e2758 _remove at> ignored

Kind of confusing right?  The real error is not the "Exception AttributeError...".  It is "Failed to load the app json".

The issue has to do with how the command is formed on the command line.  The tutorial lists the following command to run/debug the app:

$ python2.7 ./samplewhois_connector.py ../test_jsons/sample_whois_domain.json

The command I was running was:
$ python2.7 samplewhois_connector.py ../test_jsons/sample_whois_domain.json

Do you see the difference?  It took me more time than I'm comfortable admitting to realize that I had omitted the './'  in front of the python file.  

For the uninitiated, specifying the relative path of a file is usually unnecessary when you're referencing a file from the command line so long as you're working directory is the same directory as the file.  In fact, the file executes as you would expect it to, until it exits with the error.  If anyone knows what's causing this and can cure my ignorance I'd greatly appreciate a tip in the comments section.


Always include the relative path prefix when you're running your Phantom apps from the command line.  Even if you're in the same working directory.

$ python2.7 ./samplewhois_connector.py ../test_jsons/sample_whois_domain.json

Issue #3: Environment Variables

The tutorial lists two essential environment variables to be set for development.  This can easily be set from the command-line as suggested.  However, this is necessary for every new session.  I highly suggest adding these environment variables to the ~/.bash_profile file for the phantom user.


Append these two lines to the bottom of ~/.bash_profile

export PYTHONPATH=/opt/phantom/lib/:/opt/phantom/www/
export REQUESTS_CA_BUNDLE=/opt/phantom/etc/cacerts.pem 

Friday, August 23, 2013

Network Traffic Visualization

Back Story

     So there I was with a pcap in front of me while I sat there converting Oxygen to Carbon dioxide while I scoured the internet looking for ways to visualize bandwidth consumption and protocol throughout the duration of a vulnerability scan.

      I was trying to have a simple visualization of Transmit and Receive bandwidth over the time of the scan.  (Evidently, this is not as popular as I thought, or I lacked the Google-Fu to find a utility to help me in this endeavor.)  This was all in an effort to validate that the vulnerability (operated by a third party) was honoring the bandwidth limitations we had outlined.  Given the performance issues observed at the remote site, I had suspicions that the bandwidth threshold was exceeded.
     The following tools are available, but either did not provide what I was looking for or did not run well on my VM.   There are many more, some of which probably do exactly what I was able to accomplish. 


     Below is how everything was setup.  Most notably, our capture box was on location with the vulnerability scanner.  We can see everything the scanner is sending out, but anything dropped by the remote site WAN connection obviously won't make it to our scanner.


     Not one to give up easily, I resorted to using what I had available.... tshark and R.


     Below is the command I used to squeeze the pertinent information out of the pcap. ( is the remote site and is the scanner.)

     (I'll leave it to you to look at the tshark man page and display filter reference)

     This command outputs a tilde (~) delimited file containing the pertinent information.  Why a tilde?  Tildes work well for delimiting text fields since they are much rarer than commas. The frame.time field uses a comma in the date format.  Using a tilde is just a good habit when delimiting anything with text.

Sample Output

Aug 17, 2013 01:54:09.555897000~6~40~
Aug 17, 2013 01:54:09.555916000~6~40~
Aug 17, 2013 01:54:09.555928000~6~40~
Aug 17, 2013 01:54:09.555986000~6~40~
Aug 17, 2013 01:54:09.555995000~6~40~
Aug 17, 2013 01:54:09.556009000~6~40~
Aug 17, 2013 01:54:09.556012000~6~40~
Aug 17, 2013 01:54:09.556068000~6~40~
Aug 17, 2013 01:54:09.556077000~6~40~

     There were over 1.8 million rows in this data set (one for each packet).  This immediately ruled Excel out of the picture since there's a hard limit to the number of rows available.

How to work with > 1.8 Million line .csv

     I'm a fan of Perl, but looping through a CSV and doing calculations takes a lot of control structures and logic (something in short supply these days).  Perl has some great graphing modules available (GDGraph is one), I'm sure <insert your favorite language here> does as well.  I'm not knocking any language, but R has excellent built in functions that make graphing a snap.
     Teaching the syntax of R is out of the scope of this post, but it is well worth the time spent getting familiar.  Looking at the code below, you should be able to decipher the long and short of what's going on.  If not, then start Googling!  I'm sure R gurus will have some criticisms here.  I am by no means an expert in R, just a big fan.  I welcome any suggestions.

R loading the CSV

R Script and Output

      Now we can do some great things with our zagg dataframe ... like graphing.

     You can see very clearly that our bandwidth limitations were not honored.  Time to chase down the vendor and make them correct their settings.

     Obviously, this does not have to be limited to the packet length.  We can also get a protocol distribution, graph the number of ICMP error codes, etc.  Let me know what you think of my first blog post below.  I look forward to your constructive criticism, suggestions, and questions.  Be sure to leave some feedback in the comments.