Success I think! Well mostly. I ran tests to verify my TLS proxy handler is working - and from what I can tell it is ... mostly. Why mostly u ask? - because I'm not seeing a HTTPS response for every HTTPS request. I reduced the frequency with which nogotofail attempts to proxy each request (from every 2nd to every 5th) and I saw more HTTPS responses come back.
I added debug code to make sure unencrypted attributes could be inspected for requests and responses, and they can! Still more testing required but I am more confident than yesterday.
It doesn't seem realistic to inspect every HTTPS request/response - proxying appears to add latency causing timeouts if the sampling rate is too high. My setup also contributes to latency - requests to the server are over a VPN to a GCE server instance in East Asia (Taiwan I believe). A lower sampling rate is fine for my objective, it's enough to sample some requests to identify PII disclosure by applications.
Here is my TLS proxy handler, it's bare bones ATM: https://github.com/mkenne11/nogotofail-pii/blob/04a60b60537076e476bbb4c4f67191109874d771/nogotofail/mitm/connection/handlers/connection/httpspii.py
Also, started on project documentation today. I wrote up the procedure for generating the TLS proxy certificate chain files - it's a little complicated and would have probably forgot if I didn't start on it (it's still draft): https://github.com/mkenne11/nogotofail-pii/wiki/Create-certificate-chain-to-performing-MitM-proxying
I submitted pull-requests to the root nogotofail project for two recent features I developed. The requests were for 2 recent TLS connection handlers:
This evening I studied the TLS connection and traffic code further to investigate how SSL proxy functionality could be added.
I also discovered an installation issue with the project on Debian 8.0+ and Ubuntu 14.04+. Installs are working fine on Debian 7.8, and were recently on Ubuntu 14.04. I raised this as an issue to look into.
(Out of order post. Publishing wasn't working when I tried Friday)
This evening I started looking over Kalman filters and trying to figure out how to apply those to my system. Hopefully I'll be able to get something demonstrable up and running in the next couple of days.
My computer had a hard drive failure that kept me offline for a little bit. It took longer than expected for me to get it back because my working computer is an older model that uses a rare/ obsolete connection for the drive. I had an SSD drive but the shop told me I needed to replace it with a PATA/IDE SSD drive. The Locating and delivery of the part was driving me crazy impatient. I did do offline work with the main author which was good. On one day, he reviewed my code, and he played "devil's advocate" regarding the front-end refactoring. He quizzed me on the decision I made with regards to the frontend. He asked me if I've designed the frontend to "fail gracefully". I hadn't.
Tomorrow: the bug above mentioned was holding up the getting started guide. Now that the bug is fixed, I'll be working on that part of the documentation again.
Today and over the weekend was very productive. I didn't have lots of time to code, but did lots of reading - looked at the OpenSSL library and the source code for TLS connections. I have a much better understanding now.
I appear to have had some success setting up a man-in-the-middle TLS proxy. After some experimenting my connection handler successfully terminates HTTPS connections between the client and itself (on_request method), and creates a new HTTPS connection between the server and itself. However the reply from the server (on_response method) isn't firing for all replies from the server. I'll keep experimenting to confirm but it's looking promising!
I am working on the layout feature, I think I am halfway done. I also thought about the last two components of my project which will be highlighting diagram elements related to browsing history and making connections among diagram elements and Drupal nodes.
Some more testing today, the accuracy improved after applying PCA to input features. Now the accuracy i am getting is 45% on IMFDB dataset which is much better than 15-18% on videos for 6 classes. Although one problem that i faced while testing due to increase in the number of testing vectors is the increase in time that the classifier takes to produce results. It takes approximately 10 minutes for one run of the classifier for testing of 230 features.
I didn't write code on Friday, so I had to compensate on Saturday.
I started to implement MIDI Actions the way I described it before.
Now we have 2 places to store MIDI Actions:
Using midi-actions.xml file for storing global actions solves the problem I described in the previous post: now we will be able to share user-defined events across several scores.
I implemented saving selected MIDI Actions and rendering them to midi, so playback works. Also I wrote several predefined actions, so now MIDI Actions in MuseScore work almost like before, but with different UI and architecture.
Today collected some results using IMFDB dataset and solved some of the errors that were caused by difference in decision function of different classifiers of scikit-learn library. Today's results were a bit better, the new method of preprocessing images and then applying PCA is giving better results.