My GSoC Adventure

Noseyparker - 7 August

This morning I worked more on user documentation. I spent some time thinking about a logical document structure, although I think the structure will only become clearer after I've written most of it. I also spent time updating the README and how-to/about pages for the PII functions. Here is the documentation so far:

This afternoon I need to update the presentation for the conference on Monday. I'm off to Bali tonight (where the conference is) it will great to get away and unwind for the weekend.

Noseyparker - 6 August

I collected a couple more application samples containing PII entries and tested these with the new reporting code. Two minor bugs were found and fixed and I also improved the efficiency of the code. After experimenting I discovered for Python string comparisons "smaller-string in larger-string" is much faster than "larger-string.startswith(smaller-string") ... who knew!

After a few hours testing I felt confident-enough in the code and merged it into the "dev" branch.

For the remainder of the afternoon I started on user documentation.

Noseyparker - 5 August

I collected a couple of application log samples from the server and processed them using the updated report code. There were a few bugs which I fixed. The reporting code seems to be reasonably robust - however more testing is still required.

While collecting the application logs I checked the performance of the MitM server. With just the PII handlers running the CPU usage peaked below 15%, which is pleasing given the large amount of processing occurring and the modest server specifications. Even with a low handler frequency (running handlers every 1 in 5 requests) there is still occasional timeouts. This is most likely caused by the latency of my cloud based setup.

Noseyparker - 4 August

I finished making updating the PII data report JSON report to handle recent application log chaanges. The report is looking good now and and for each online domain shows: - PII disclosed over unencrypted (non-HTTPS) connections - PII disclosed over encrypted (HTTPS) connections - Query-string key/value pairs occuring for multiple requests over unencrypted (non-HTTPS) connections

Note. although some pairs are anonymous when they persist across multiple requests that could allow user tracking.

Here is a sample of the PII data report - https://github.com/mkenne11/nogotofail-pii/blob/57b24763c9f99f75a892093c778734f929db8333/nogotofail/mitm/report/samples/pii_data_report.json

I ironed out a few bugs in the report. Tomorrow I will process more application log samples and fix issues I found.

Noseyparker - 3 August

Today I started on changes to the JSON report code that are needed to handle the new HTTPS PII log alerts and other alert name changes. I am still writing the code and haven't had a chance to test it yet.

Unfortunately I didn't have alot of time to code today. I was working in the PowerPoint presentation for the conference early next week - the deadline for this is very soon.

Noseyparker – 31 July

I completed the PII detection code in the TLS proxy handler today. During testing, I ran about 20 mobile applications to check on the PII logged - it was very surprising (and concerning) how much of my personal information apps sent and the number of different services! I found one app which sent my information to 8 different services, including advertising, analytics and performance services.

I started on a new base PII connection handler that will be inherited by the HTTP and HTTPS PII handlers. Unfortunately the current HTTP PII handler isn't aware if the connection uses SSL, and alerts are firing for HTTPS as well. However, the new base PII handler will contain alot of code that will be inherited - and make the HTTP and HTTPS PII handlers much simpler.

29 July - Noseyparker

I added PII detection logic to my TLS proxy connection handler - so far I'm parsing HTTPS request query strings and headers for PII. I compared a small number of alerts against PII I found in clear-text (unencrypted) traffic logs for the same apps (traffic logs were collected using the mitmproxy tool). The alerts raised look correct and I'm more confident the handler is working properly.

Now that I'm inspecting HTTPS traffic I noticed there is a conflict with the HTTP traffic handler. It doesn't look too difficult to fix but will require some thought to get the design right.

I started a new subject this week, and need will need to do some admin and get started on the research project over the next 2 days - I still plan to do work on the SoC project just less than usual.

Noseyparker - 28 July

Success I think! Well mostly. I ran tests to verify my TLS proxy handler is working - and from what I can tell it is ... mostly. Why mostly u ask? - because I'm not seeing a HTTPS response for every HTTPS request. I reduced the frequency with which nogotofail attempts to proxy each request (from every 2nd to every 5th) and I saw more HTTPS responses come back.

I added debug code to make sure unencrypted attributes could be inspected for requests and responses, and they can! Still more testing required but I am more confident than yesterday.

It doesn't seem realistic to inspect every HTTPS request/response - proxying appears to add latency causing timeouts if the sampling rate is too high. My setup also contributes to latency - requests to the server are over a VPN to a GCE server instance in East Asia (Taiwan I believe). A lower sampling rate is fine for my objective, it's enough to sample some requests to identify PII disclosure by applications.

Here is my TLS proxy handler, it's bare bones ATM: https://github.com/mkenne11/nogotofail-pii/blob/04a60b60537076e476bbb4c4f67191109874d771/nogotofail/mitm/connection/handlers/connection/httpspii.py

Also, started on project documentation today. I wrote up the procedure for generating the TLS proxy certificate chain files - it's a little complicated and would have probably forgot if I didn't start on it (it's still draft): https://github.com/mkenne11/nogotofail-pii/wiki/Create-certificate-chain-to-performing-MitM-proxying

Noseyparker - 24 July

I submitted pull-requests to the root nogotofail project for two recent features I developed. The requests were for 2 recent TLS connection handlers:

This evening I studied the TLS connection and traffic code further to investigate how SSL proxy functionality could be added.

I also discovered an installation issue with the project on Debian 8.0+ and Ubuntu 14.04+. Installs are working fine on Debian 7.8, and were recently on Ubuntu 14.04. I raised this as an issue to look into.

(Out of order post. Publishing wasn't working when I tried Friday)

Noseyparker - 27 July

Today and over the weekend was very productive. I didn't have lots of time to code, but did lots of reading - looked at the OpenSSL library and the source code for TLS connections. I have a much better understanding now.

I appear to have had some success setting up a man-in-the-middle TLS proxy. After some experimenting my connection handler successfully terminates HTTPS connections between the client and itself (on_request method), and creates a new HTTPS connection between the server and itself. However the reply from the server (on_response method) isn't firing for all replies from the server. I'll keep experimenting to confirm but it's looking promising!

Syndicate content