“Scaling Big Data Mining Infrastructure at Twitter,” is a nice slide show that Alex Popescu posted on his blog, myNoSQL. The slide show is a broad overview the of the data engineering Twitter is doing to make life easier for their data scientists.
Facebook has long been a leading force in the development of big data. The recent release of Graph Search is seen as a move by the company to flex their big data muscles and put their engineering bona fides to the test.
A couple of good articles came out this week discussing the engineering challenges Facebook is taking on with this project. Zach Miners, at IT World, explains the work the new tool does searching large graphs with this article. Harpreet, at Tools Journal, wrote a good piece explaining how Natural Language Processing is used by Graph Search.
NYU has announced a university wide Data Science initiative and the creation of the Center for Data Science (CDS). Columbia University got a head start with The Institute for Data Science and Engineering (IDSE), but NYU is offering a MS degree with plans to offer a PhD, something Columbia’s IDSE is not doing at this time.
This news comes on the heels of Massachusetts’ efforts to make Boston a big data capital. Massachusetts has long been a leading center for academic research in data mining and big data through MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL).
There are many institutions across the US that have added Data Science degrees to existing programs. However, NYU is one of the first schools to create a center with the purpose of offering a degree, as opposed to only conducting research, in this field.
Python and big data have recently been in the news. Continuum Analytics just received $3M from DARPA (I hope they cash their check before the possible sequester) to develop big data capabilities for Python with projects Blaze and Bokeh. This is promising news for those of us that are not proficient in multiple programming languages. At this point, Java has been the lingua franca for most big data applications. This project won’t address all the performance issues with Python, hence the common use of Java in most development, but hopefully it’ll allow us non-polylingual programmers do to some heavy lifting without all the curly braces.
Once again GigaOm’s Derrick Harris gives us a great report in, “DARPA puts $3M into startup pushing big data in Python.”