NSF:
- Figured out the problem with finding Government interest. It was looking for the string '<paragraph-federal-research-statement>', but in practice, these tags are always <paragraph-federal-research-statement id=xx...>, so an opening tag was never found.
- With this fixed, however, new problems showed themselves. Since the parser code hadn't been run since I split patparser.py into its own file, there are now a ton of missing methods. I am trying to figure out the class system, but it seems really counter-intuitive.
- I need a way to call methods from the instance python file (run.py) within patparser.py, but there does not seem to be a way to do this. In java, you would just make those methods static or pass the imported class the instance class's object with the this keyword.
- In python, there is no this, and seemingly no way to get the object of the file that instantiated another file
- I tried putting stuff into classes, but then you have to add an additional argument self to every single method. Really? That's a dumb thing to force on anyone who wants to use objects.
- No github push because the code is in pieces
NSF:
- Verified that reading .breakpoint behaves correctly
- Realized that I was probably getting low download speeds due to being on Wi-Fi rather than Ethernet
- Added a check for <patent-application-publication>, the program will now force stop if this is not found anywhere
- This is a temporary behavior which will allow me to pinpoint the date at which it is changed.
- Added a sort to the list of urls. It now sorts from oldest to newest, whereas before the order was based on the page layout. It does this by using regex to remove all non-numbers, then sorting the strings alphabetically (since 10 comes before 010 and so on)
- I still haven't gotten anything logged to the csv file though.
- I changed the NSF check slightly: when checking for the string 'nsf', it now does not remove spaces, so that if a word happens to have nsf in it, such as 'transfer', a false positive is not triggered. When checking 'nationalsciencefoundation', it still ignores spaces.
- Added output for when the government interest statement is found, but it never outputs, so it is possible that I somehow messed up the tag. I will look into this tomorrow.
NSF:
- The download speeds were really slow today, so I didn't get a whole lot done
- Probably implemented forcing inclusion of the filename within .breakpoint, but I couldn't test it because I couldn't download any of the zip files.
NSF:
- Fixed concurrent modification
- Added a handler for Ctrl+C (break) to delay breaking until after a scrape has completed if one is currently in progress
- Added a function that writes the current filename to a file called .breakpoint if the program breaks while scraping.
- This will prevent the program from skipping a file that has been downloaded but not yet scraped.
- I haven't yet implemented reading of .breakpoint, but I plan to have it remove the filename it contains from the remove array in removeDownloaded() thus keeping the filename in the queue to scrape.
Javascript:
NSF:
- Switched to google for downloading the zip files (I downloaded a file from both and got 0.26 MB/s from reedtech and 1.33 MB/s from google)
- Since google doesn't have the links in tables, I set the parser to find all links, then only keep the ones that started with 'ipa' or 'pa'.
- Program now removes urls of any file already in the temp folder (after testing if it is a properly written zip file), causing it to no longer waste resources parsing already parsed files.
- I will have to do a bit more testing to make sure it is actually working correctly. I set it to print out all skipped files, and only prints that it is skipping every other file, even though it appears to correctly skip all of them
- I realized that I am parsing through a list while removing elements from it. In java, this gives a ConcurrentModificationException, but I guess in python it is technically allowed. I will fix this tomorrow by adding all elements found in both lists to be added to a new list, and once the scan is complete, that list of elements will be removed from the urls list
NSF:
- Added a method to parse year, month, and day out of the filename
- Learned more regex
- Set the program to download and scrape everything from 2001, but it didn't finish
- So far, I've looked through 32 full xml files and found no NSF documents. I have tested the program against sample files mentioning NSF, but it is still kind of discouraging to not get any results.
- Also, I need to find copies of the DTD from various years in order to find how the tags changed
NSF:
- Learned more about how python modules, classes, and namespaces work
- Learned how to do the Python equivalent of Java's try{}catch{}
- Moved all parsing code into patparser.py to clean up a bit
- Moved unzipping code from the main() method into a separate method
- Added a check for badzipfile exception, and the zip will be redownloaded if this is thrown
