This project was my attempt at building a GIF search engine that could understanding natural language search. I used a variety of methods to tag and classify images (such as object detection, scenery detection, etc), and used a couple APIs to validate and generate more tags. I built primarily in Python and C++, and this was built during HackWeek at BuzzFeed.
I used Caffe for the image classification module. You can see image classification example from Caffe here. I used Clarifai, and many other APIs to initially classify GIFs, and then used them to train my own instance of Caffe. It ran on one of the AWS EC2 GPU instances. Training Caffe instance was the most difficult since the basic support built-in was for images rather than GIFs. I overwrote some of the methods to turn a GIF into a series of images, and then used the normal machine learning algorithms built in to classify them.
I've open sourced this here.