Changes 2/19/14

NSF:

• The array storing the xml documents wasn't clearing after completion of a scrape, fixed that
• Correctly implemented checking for local files before downloading, the only problem now is if you break before downloading a file, next time it will stop the program due to a bad zip file, I'll have to fix that
• run.py is getting extremely long and hard to navigate, and I am thinking of splitting it into a utilities file, a main file, and an xml parsing file
• Attempted to fix up console output a bit

Changes 2/18/14

NSF:

• Program now finds the xml anywhere within the directory tree of the zip file
• Began adding checks to see if xml files have already been downloaded.  If so, the local copy will be used
• Cleaned up console output by using sys.stdout.write() with sys.stdout.flush() to overwrite one line instead of filling the entire console with output on new lines.
• One problem I am having is that many of the full xml files I am working with do not mention NSF in the entire document, so sometimes it is hard to tell if things work correctly.
• In order to fix the problem of tags differing as time goes on, I will have to find and look at the DTD changelogs.

Changes 2/13/14 and 2/14/14

I was updating ubuntu studio, but the update didn't work right and /dev got corrupted or something like that.  I booted into windows and installed a program that allowed me to mount the linux partition, and copied my /home/ folder out.  I had to reinstall from a flash drive, and it took me a lot of tries because I didn't put the bootloader on the right partition.

For future reference, for me: The fat32 partition isn't for the bootloader, it's a dell thing. You also should format the ubuntu partition just in case. Put the bootloader on /dev/sda and it will work.

NSF:

• The program now scrapes http://patents.reedtech.com/parbft.php for all the patent urls, and should iterate through them.  It also gives a readout of the amount downloaded for each file.
• I realized that the newer patent applications don't use <patent-application-publication> to denote the xml, but instead <us-patent-application>, which breaks the program as of now.
• Also, not all the zip files have the xml in their root directory.  I am trying to figure out a way to iterate through the tree to find the xml file, but os.walk doesn't seem to work and I don't really want to unzip the entire zip directory tree (though I might have to)
• Added more convenience methods to clean up the code a bit

Changes 02/12/14

NSF:

• Started a new python file that will scrape http://patents.reedtech.com/parbft.php for all the patent zip file urls.  These will eventually be compiled into a list that the main program can iterate through

Visualize Algorithms:

• Downloaded and set up VAP
• Messed around with the demo.py program

Changes 2/11/14

NSF:

• Removed lxml from the Federal Research Statement check
• This eliminates the need for any xml file that does not pertain to the NSF to have its xml parsed and a memory structure created (I am assuming lxml does this)
• Instead, my program reads any text between strings '<paragraph-federal-research-statement>' and '</paragraph-federal-research-statement>' (if they are found).
• It then runs a regex on the strings between these two tags, converting everything to lower case and removing all spaces and numbers
• If the strings 'nsf' or 'nationalsciencefoundation' are found, it logs to file.  Otherwise, it skips to the next one with no xml parsing ever occuring.
• This results in a speedup of 80 times (Before it took 26.42 seconds to parse the first 100 items, now it takes 0.33 to parse the first 100 and only 11.55 seconds to run the entire 2,396 patent file)
• This incredible speedup is largely due to the fact that the file I chose has no patent applications mentioning NSF, so no xml is ever parsed by lxml, and all the checks are done by string.find().
• Started adding support for downloading the files directly from the web

Changes 2/10/14

NSF:

• Created github page
• Parser now checks for NSF clause right away, and skips any documents that don't have it (which is 99% of them)
• Cleaned up console output.  Example: (Scraping 932 of 2396 - (3212 lines). No NSF reference, skipping.)
• Fixed inefficient memory use by calling unicode() on the parsed strings.  From the bs4 wiki:
  If you want to use a NavigableString outside of Beautiful Soup, you should call unicode() on it to turn it into a normal Python Unicode string. If you don’t, your string will carry around a reference to the entire Beautiful Soup parse tree, even when you’re done using Beautiful Soup. This is a big waste of memory.
  
• I am not the best at git, so I may sometimes upload the wrong thing or not upload at all.  Apologies in advance.
• I will add in a timer to see how long the parse takes, it seems to take about 20 minutes.  I'll try running it on my desktop at home which has a much better processor

Changes 2/6/14

NSF:

• Fixed up csv data separation in the output
• Started implementing support for the real world, bulk datasets