Big Data and TSM&O

Big data is a term for data sets that are so large or complex that traditional data processing application software is inadequate to deal with them. Challenges include capture, storage, analysis, data curation, search, sharing, transfer, visualization, querying, updating and information privacy. The term "big data" often refers simply to the use of predictive analytics, user behavior analytics, or certain other advanced data analytics methods that extract value from data, and seldom to a particular size of data set. "There is little doubt that the quantities of data now available are indeed large, but that’s not the most relevant characteristic of this new data ecosystem.

Examples of big data approaches or techniques being used in current transportation operations are limited. A number of current transportation projects, such as the Integrated Corridor Management (ICM) systems in San Diego and Dallas and several Michigan Department of Transportation agency fleet connected vehicle projects, feature a number of approaches common to big data. Although very valuable projects, they do not, however, illustrate the full potential for big data, including that which can be realized by applying big data when connected vehicles are widely deployed across the country. Big data examples are somewhat more prevalent in the private sector. Although big data approaches are not yet common, these sorts of examples of big data in transportation suggest that a great deal of potential exists for applying data in the coming traveler data-rich connected vehicle environment.

• The importance of resolving data ownership issues and the implications for roles.
• Investigating potential use of a third-party data broker (or multiple brokers) – an approach that may help address ownership and funding needs (the cost of capturing and managing data may be cost-prohibitive for government but profitable for the private sector).
• Development of data standards is an important step, especially so if transportation agencies are not collecting and managing the data themselves.
• Considering approaches to reduce the volume of connected vehicle and traveler data so that it is more manageable while ensuring that all valuable data is collected.
• Utilization of specific technologies and techniques like crowdsourcing, cloud computing and federated database systems that have come to characterize the state-of-the-practice in big data and which will facilitate transportation operators or private sector data service providers in extracting value from connected-vehicle and traveler data.