Monthly Archives: February 2013

Posted on by

This is great checklist of topics to become proficient in for all aspiring Data Scientists. This list of skills can reasonably be attained within a master’s degree program in Statistics, Computer Science or Operations Research.

Ryan Swanstrom

A while back James Kobielus wrote the article, Data Scientist: Consider the Curriculum. It contains one of the best descriptions of a data science curriculum I have seen.  Also the article includes a list of algorithms/modeling techniques that should be known by a data scientist. Below is the list from the article.

  • linear algebra
  • basic statistics
  • linear and logistic regression
  • data mining
  • predictive modeling
  • cluster analysis
  • association rules
  • market basket analysis
  • decision trees
  • time-series analysis
  • forecasting
  • machine learning
  • Bayesian and Monte Carlo Statistics
  • matrix operations
  • sampling
  • text analytics
  • summarization
  • classification
  • primary components analysis
  • experimental design
  • unsupervised learning
  • constrained optimization

The list almost looks overwhelming.
Do you think anything is missing from the list?

View original post

Big Data at Facebook

Posted on by

Facebook has long been a leading force in the development of big data. The recent release of Graph Search is seen as a move by the company to flex their big data muscles and put their engineering bona fides to the test.

A couple of good articles came out this week discussing the engineering challenges Facebook is taking on with this project. Zach Miners, at IT World, explains the work the new tool does searching large graphs with this article. Harpreet, at Tools Journal, wrote a good piece explaining how  Natural Language Processing is used by Graph Search.

One last article I would like to highlight is a glossary of big data at Facebook that was put out by Wade Roush at Xconomy. 

NYU’s Data Science initiative pushes North East to forefront

Posted on by

NYU has announced a university wide Data Science initiative and the creation of the Center for Data Science (CDS). Columbia University got a head start with The Institute for Data Science and Engineering (IDSE), but NYU is offering a MS degree with plans to offer a PhD, something Columbia’s IDSE is not doing at this time.

This news comes on the heels of Massachusetts’ efforts to make Boston a big data capital.   Massachusetts has long been a leading center for academic research in data mining and big data through MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL).

There are many institutions across the US that have added Data Science degrees to existing programs. However, NYU is one of the first schools to create a center with the purpose of offering a degree, as opposed to only conducting research, in this field.

Python wraps itself around big data

Posted on by

Python and big data have recently been in the news. Continuum Analytics just received $3M from DARPA (I hope they cash their check before the possible sequester) to develop big data capabilities for Python with projects Blaze and Bokeh. This is promising news for those of us that are not proficient in multiple programming languages. At this point, Java has been the lingua franca for most big data applications. This project won’t address all the performance issues with Python, hence the common use of Java in most development, but hopefully it’ll allow us non-polylingual programmers do to some heavy lifting without all the curly braces.

Once again GigaOm’s Derrick Harris gives us a great report in, “DARPA puts $3M into startup pushing big data in Python.”

InformationWeek Government also has a nice article, “DARPA Funds Python Big Data Effort,” by J. Nicholas Hoover.

Also check out Continuum Analytics’ blog announcement about the project and another post detailing Blaze.

Ayasdi: Topological Data Analysis

Posted on by

These are older stories, but never the less, I wanted to post them as a reminder to myself and a heads up to others. Ayasdi has developed a data mining platform that is based on topological data analysis.  The company was co-founded my Dr. Gunnar Carlsson of Standford University and is based on Carlsson’s pioneering research in the field of Applied and Computational Algebraic Topology.

Why is this interesting? Because this is a completely new approach to data analysis. A lot of the methodologies used in data mining/machine learning/data science is based on statistics and probability. The use of algebraic topology sort of comes out of left field (abstract algebra is extensively used in cryptology and computer security). Ayasdi’s approach shows they are doing some very novel and innovative research.

Enough said, below are two great articles that do a better job than I can explaining what exactly Ayasdi does.

This first article comes to us from Derrick Harris who writes for GigaOm. Check out his great article, “Has Ayasdi turned machine learning into a magic bullet?“

The last article is from the New York Time’s blog Bits. The post, “Ayasdi: A Big Data Start-Up With a Long History,” gives some background about the company.