aingerson's blog

13 August

Still working on the last revisions to the code in order to allow it to run very large data sets since it should all be for the most part in place, not requiring huge buffers. I should be putting the final touches on it tomorrow so I can run tests and clean everything over the weekend. The goal is to be able to run it on the 10Gb file David provided to me this week.

5 August

Have been running tests with VTune this week to try to understand how my code is interacting with my CPU and my cache. I have a meeting with David set up tomorrow to discuss the results and talk about where I can head from here.

22 July

Very productive day today. I finally found the bug and fixed it (along with the other 3 bugs I found after that) and finished up the OR code. Tomorrow, I'm going to polish the OR code and implement the compression threads.

21 July

I think I found the bug which is super exciting! I can finally move on to finishing the OR code and investigating implementation of p_threads.

20 July

Met with David today and discussed the final steps to polish my code so it's ready for testing with VTune. This should be the last week left for coding before testing.

17 July

Today I finally pinpointed the bug in the query code and got it to run smoothly. It also reads a lot better which is great for me and anyone looking at my code. I think the time I spent recoding all of this will really pay off, especially during testing next week. While it runs as expected on most of the data sets, my program seems to crash any time I query the largest data set. I'm not exactly sure why, though. It could be because of insufficient memory or, more likely, a memory leak which I will try to find this weekend. This weekend I'm also going to try to get the OR code working as well which should be fairly similar to the AND code with a few exceptions.

16 July

My reformatted code is better but I'm still running into the same bug that I had before (which isn't a surprise as I pretty much recoded the same logic sequence into a easier to understand format). That said, it is going to be A LOT easier to debug now and that's what I'm hoping to do tomorrow.

15 July

I think backtracking was a good idea. This new way is going to not only be clearer for me to code but also easier for anyone else to follow when reading my code. I will hopefully be fully done with querying by the end of the week.

14 July

Found yet another bug in my query code. At this point, I think I'm going to backtrack a bit and retry from scratch, this time splitting the AND code into various different methods to make it easier to debug because it's ending up being a headache and a half trying to pinpoint where each bug is coming from. And if I start form scratch, I can add in some code to load each file into memory before querying it in order to save query time.

13 July

Met with David today and discussed my current query code. Then worked on a bug in my query code and got it almost figure out. Also figured out my exit code issue from earlier (turns our fread returns 0 if unsuccessful which is nice). Tomorrow, I'm going to work on loading the compressed bitmaps in to memory before querying so that load time isn't included in our tests later on.

Syndicate content