For a while now I been thinking about the best ways to remember and associate things. I started working on Learnt.io, which was a platform to just do that. It was an easy way to "store" your notes, and be able to make associations between them. For example, if you learnt about Quantum Mechanics today you can easily relate that to the classical mechanics notes that you've made, and other notes that have helped you moved to learning quantum mechanics from classical mechanics.
I started having this issue when I started taking more and more classes in Computer Science, and also increasing my philosophical, and entrepreneurial reading. I wasn't able to make all the associations because my notes were usually not trying to look for patterns and similarities between them. To me, it seemed obvious that a platform should exist that should use knowledge that exists on the internet to help you make associations, but this is actually a very difficult problem to solve. This is why I started from making a notes app, and making simple associations on top of it.
However, recently I've begun using Wikipedia and the text containing on each topic to make a graph of associations between ideas, and then using that to help tell users how to make associations and what associations to make. For example if you were to make notes about Mahatma Gandhi - it would know the articles that link to it, and give you suggestions on what to take notes on, but also to associate articles you've written before on those topics. Each article you write will be given a tag that corresponds to how it exists on Wikipedia, and then it becomes a little simpler to make these associations.
So - my work now is trying to build a graph of associations of Wikipedia articles, and to find the correct weights between nodes so I'm able to give the user the best associations between articles. Another feature I've worked on is being able to tell the user what to read next. If he's currently reading about Mahatma Gandhi then we can suggest the user to read about the Independence of India, which is a natural successor to her reading. This becomes simple because we are able to take nodes from the graph that the user hasn't made notes on, and simply return to the user the next node with the highest weight.
My ultimate goal would be to create a simple map of things I know, and things the user knows and help him visualize it and understand it. Then he's able to learn and make associations, and actually visualize where they are placed on a graph. This would also help him understand where things link together, and see the different connections that exists that he hadn't before.
The graph above is a representation of test data that I'm performing to find links between different categories. For example - here you can see a couple nodes that are linked using a ton of edges. These are just topics on Wikipedia they I've used to demonstrate the type of links between knowledge I'm trying to find. They are also weighed edges using a couple different techniques, and using this we can construct a couple of things. We can see how related two things are, or things like "What to read next", which are simple applications of this technology.
To Do:
- Consider writing algorithms to switch between Neo4j and Mongo
- Neo4j for analysis, and Mongo for store
- Search algorithms to traverse the tree efficiently
- Upgrade instance of offline wikipedia to support newer information
- Think about how to include updates as data on Wikipedia changes
- Already: Monitoring the changes API through Wikimedia
- Analysis of tables on Wikipedia
- Beginning to understand how to involve media (images, videos) in the project
Progression
October 25 2014 - hit one million nodes in my wikigraph.
October 30 2014 - hit two million nodes in my wikigraph.