Raising Awareness of Artificial Intelligence for Transportation Systems Management and Operations

Chapter 3. Commercialization of Artificial Intelligence Technologies

In this chapter, the capabilities of commercial artificial intelligence (AI) platforms will be presented. In software terminology, a "platform" generally describes a suite of software applications and components that work together to address a variety of functions. Since there are hundreds if not thousands of commercial offerings, this chapter will focus on the major offerings from Google, Microsoft, and Amazon and specialty categories relevant to transportation systems management and operations (TSMO). As all commercial companies continue to quickly add to their capabilities, feature sets will likely continue to evolve, platforms will continue to become easier to use, and cost of ownership or use is likely to decrease. This view of commercial capabilities should be considered just a snapshot of early 2019 and should not be considered an endorsement of any particular provider or technology services. After reading this chapter, you should have an appreciation of the technical maturity and capabilities of major platforms for development of AI applications.

This chapter is organized as follows:

• The state of the practice in commercialization of AI.
• A summary of the capabilities of major AI technologies offered by Google, Amazon, and Microsoft.
• Expected AI capabilities in the future based on current publicized roadmaps from major platform providers.
• How such systems and technologies are typically procured and monetized.

State of the Practice and Commercialization of Artificial Intelligence

There are almost 1,000 commercial companies involved in providing software and services in some way or another related to AI. According to Gartner, widespread adoption of AI is still emerging across all industries.⁴¹ It would generally be true that the larger the organization, the more likely they would be to adopt or pilot AI technologies. Costs associated with software procurement, software development, and database development and population can be substantial while the corresponding expected improvements may be unknown. Larger organizations (from a TSMO perspective, this would imply larger State departments of transportation (DOT) and the largest cities) generally have more tolerance for downside investment risk with the potential for substantial at-scale benefits if successful.

AI technologies also integrate common functions of business intelligence (BI) technologies include reporting, online analytical processing, analytics, data mining, process mining, complex event processing, business performance management, benchmarking, text mining, predictive analytics, and prescriptive analytics. Typically, AI technologies are used to handle large amounts of structured and sometimes unstructured data and aim to allow for the easy interpretation or use of this data. In commercial terms, AI is envisioned to provide users of the various technologies with a competitive advantage by identifying new opportunities and implementing an effective strategy based on insights.⁴² For infrastructure owners and operators (IOO), use of AI in its various forms may reduce costs, augment staff, improve staff efficiency, provide suggested solutions that humans may not have thought through in advance, and provide conveniences in the transportation management center (TMC) similar to those enjoyed by staff on their personal time such as turning on lights and voice activation of other actions. IOOs will need to consider if it is more effective to purchase such AI functions as a service (aaS) or develop and maintain such applications on-premise or in-house. With the current state of the practice in AI, the answer will likely be some hybrid of both methods where AI solution providers/vendors (which could include universities in addition to private companies) use "aaS" technologies in software that is sold to the IOO as a packaged software product.

Some examples of commercially-available AI applications include:

• Natural language processing and natural speech synthesis of digital assistants. Progress in speech synthesis and capture has been significant in just the last five years. 511 systems may be enhanced with such technology.
• Integrated chatbots in virtual assistants enables new ways to interact with software systems. Conversational skills of virtual assistants are likely to continue to increase in the coming years as the software is trained to understand the context of your actions and carry out tasks on your behalf.^{43, 44} TMC operations may be automated and streamlined.
• All major cloud service providers (Google Cloud, Amazon Web Services, Microsoft Azure) now support suites of AI–related tools and software services. AI suites may prove useful to IOOs in developing AI applications with less development time and cost.
• Object recognition and tracking software is commercialized widely, including companies focused solely on the traffic and transportation market. Incident detection and management may be significantly affected.
• Driverless vehicles and unmanned aerial systems (UAS) are likely to be available in the near term for use by IOOs in a variety of applications, including crash abatement, asset surveillance, and equipment delivery.

Commercial Artificial Intelligence Platforms

As stated previously, in software technology, a "platform" generally describes a suite of technologies that are used together to build applications. This typically describes the combination of databases, algorithms, user interfaces, application programming interfaces (API), computing resources and operating systems, testing software, development languages, and other supporting software and hardware. For curious readers with limited software development experience, browsing any of the commercial provider's websites to learn more about any of the tools and components may be challenging. While most platform providers supply an array of tutorials and "quick start guides," most require extensive experience with databases, programming (particularly python), and knowledge of technical jargon related to machine learning and advanced statistics.

Given the many tools that could all combine and overlap to produce one comprehensive AI solution, several deployment options are available based on computing capabilities of the IOO and their information technology (IT) policies on data security and privacy. Figure 12 shows the range of implementation options available.

The "as a service" (aaS) models generally assume a cloud-based deployment (gray boxes) for the aspects of control an organization is willing to sacrifice for simplicity and possibly cost savings.

• On-Premise: Deployment offers users the ability to install, manage, and maintain every aspect of a system deployment. Typical on-premise deployments require significant up-front costs (hardware, software licensing, etc.), but allow for greater control of the system.
• Infrastructure-as-a-Service (IaaS): Deployment provides scalability needs and minimizes responsibility for the DOT. Users are responsible for managing applications, data, runtime, middleware, and operating system. Instead of having to purchase hardware outright, users can purchase IaaS based on consumption (such as database reads/writes, or computing cycles), similar to electricity or other utility billing.
• Platform-as-a-Service (PaaS): Deployment allows users to develop, test, and deploy applications quickly and efficiently. With PaaS, users are only responsible for data and application tiers. Similar to IaaS, users can purchase PaaS on a subscription basis ultimately paying just for what they use.
• Software-as-a-Service (SaaS): Deployment uses the Web to deliver applications. Most SaaS applications can be easily accessed directly from a Web browser on the client's side. This model is maintained entirely by the vendor. Like the other service models, users typically purchase a subscription to access the application.

Note that for the major commercial providers detailed in this chapter, none are available as "on-premise." Cloud services for technologies that rapidly evolve (which includes AI) allow developers and providers to more seamlessly provide new functionality and enhancements to the end customer/user without (sometimes) complex installation processes.

Google Cloud Artificial Intelligence

The Google Cloud AI platform includes the necessary elements to build and train machine learning neural network models for a variety of applications. TensorFlow is a popular neural network tool used across a variety of industries for imagery analysis, natural language processing (NLP), and other unique applications such as automated piano audio transcription. Other elements of the Google Cloud ecosystem include DialogFlow, Actions, and Firebase, which are used for Chatbot development and integration of question-answering applications with Google Assistant.

Since TensorFlow is an open-source software, it is also available on other platforms, including Amazon and Microsoft, among hundreds of other AI platforms. Google has stated that they will continue to develop new modules in their AI ecosystem that make machine learning development less and less about programming and computer science and more about data science and analysis. In addition, tools like Cloud AutoML are being provided more rapid identification of the "right" neural network model architecture for the specific application.⁴⁵ Many neural network applications considered by IOOs (traffic imagery analysis, in particular) will likely include some use of TensorFlow or similar technologies to differentiate between normal and abnormal conditions in a messy data set, such as incident and nonincident conditions.

Google's Duplex technology,⁴⁶ while not commercially available to write apps against yet, represents their next generation of chatbot technology that can make phone calls to humans and perform basic tasks. In their demonstrations, the chatbot makes reservations for dinner and schedules hair appointments (neither which may seem remotely related to TSMO), but the potential for the technology to communicate to humans and respond to them in context is a stepping stone to more relevant activities such as communicating instructions to technicians, coordinating among agencies, performing incident management activities, and modifying traffic signal timings.

One other Google AI technology that is emerging is Temporal Action Localization.⁴⁷ This technology allows object recognition in video streams by associating certain moments in the video with context. While again, Google demonstrates the technology with consumer applications of videos of children playing ("sliding on a slide") and babies being fed with a spoon, the extension to TSMO business applications may be able to take incident detection and video image analysis beyond the realm of existing approaches. TMC operators (or applications in real time) could search thousands of camera feeds simultaneously for "slow traffic where typically not" and many other different kinds of potential natural language queries.

Microsoft Azure Artificial Intelligence

Microsoft provides a suite of tools and technologies for machine learning and AI application development, including neural network models, AI apps and chatbots, and "Cognitive Services," which are prebuilt neural networks that are available to solve common problems (specific object recognition in images, most commonly) using your specific data sets. Cognitive Services also provide functionality, such as removing offensive content from images and videos, extracting key phrases from volumes of text, and automated text translation into multiple languages.

Microsoft (and both Google and Amazon) also release early-access "alpha" tools and services as Labs.⁴⁸ These AI tools allow experimentation with new features and services.

Two of these labs are particularly relevant to TSMO applications, namely "project knowledge exploration"⁴⁹ and "project anomaly finder."⁵⁰ Project knowledge exploration is Microsoft's effort to turn natural language questions into Structured Query Language (SQL) queries. As this technology continues to progress, questions such as, "how many ramp meters had more than 1000 vehicles per hour flow in the PM peak period" or "which traffic signal had the most emergency vehicle preemptions in November 2018," will be enabled without prewritten SQL queries or customized reporting and such questions can be asked by voice. Project Anomaly Finder offers the potential for transportation management system (TMS) modules to analyze and report problems with field equipment without extensive software development. Currently, TMS applications typically have extensive code specifically designed for anomaly detection of various device types. Costly software development is typically required when new types of anomaly analysis is needed.

Amazon Artificial Intelligence on Amazon Web Services

Similar to Google and Microsoft, Amazon offers a suite of AI tools and technologies as part of its Amazon Web Services ecosystem, centered around Lex, Polly, Rekognition, and SageMaker.⁵¹ Lex and Polly represent the suite of tools that power Amazon Alexa. Rekognition is for imagery analysis (as with Microsoft and Google, using deep learning neural networks). SageMaker is Amazon's suite of tools for machine learning and training various types of neural networks.

Like Google and Microsoft, Amazon offers "preview" services that will be launched soon, including Forecast that claims can be used with no machine learning expertise, although a deep understanding of regression models and forecasting background is recommended.⁵²

Procuring and Using Artificial Intelligence Platforms for TSMO

Most TSMO agencies procure TMS applications from vendors and those vendors will likely use components from open-source tools provided by major platforms other third-party AI software providers to integrate AI into existing applications and in developing new applications. Procurement and pricing of platforms, software, and computing resources aaS is a complex undertaking. Procuring software on a subscription basis is an emerging practice for most DOTs and it is quite difficult to estimate how much an application will cost. Pricing models vary based on computing cycles, data storage size, frequency of analytics, and other metrics. Since AI applications will be new, costs will likely be extensive in the beginning and become more affordable over time. A lot of care will need to be taken to architect an AI system, so it does not become price prohibitive to store and process data in the Cloud, or via massive on-premise computing resources and databases. IOOs should consider if it is more effective to purchase such AI functions aaS or develop and maintain such applications on-premise or in-house. Considering the current state of the practice in AI, the answer will likely be some hybrid of both methods where AI solution providers/vendors (which could include universities in addition to private companies) use aaS technologies in software that is sold to the IOO as a packaged software product.

Specialty Categories of Commercial Artificial Intelligence Offerings for TSMO

Computer Vision Processing

There are several companies that now provide AI–based computer vision processing for traffic, including MioVision, Gridsmart, NoTraffic, GoodVision, Waycare, and a host of others. These products are not endorsed by this report but used only as examples of existing commercial offerings to illustrate the rapidly increasing availability of AI (typically using neural networks for object recognition) in video analysis hardware. Newer camera systems can now track vehicles and objects in their field of view without superimposing detection zones on the image.^{53, 54, 55, 56, 57, 58} Anomalous activities, such as persons walking across a freeway or crashes are flagged. Performance of such systems typically depend on the variances in the background (shadows, weather) and other aspects of the video quality. Patented (and open-source) methods using machine learning techniques are now available that post-process video content for applications such as traffic counting and vehicle classification studies.^{59, 60}

Some of these systems work with their own cameras and others are central software that process digital video from other cameras. Centralized processing of closed-circuit television (CCTV) video will likely become more prevalent as the technology continues to mature to handle variances in image sources, such as camera positioning, resolution, and clutter in the traffic scene. The promise of capturing multimodal users, counting turning movements, and acquiring trajectory data of vehicles from video streams will significantly enhance a variety of TSMO applications.

Driverless Systems (Ground)

Driverless vehicles are emerging, but even so the ubiquitous availability of such technology is still far in the future. A driverless shuttle is illustrated in figure 13. Driverless vehicles are being developed primarily now by the private sector and as such they are proprietary offerings, although there is still much university research ongoing to develop new approaches. A handful of aftermarket startups are planning to offer retrofit kits, while it remains to be seen how such retrofits would work with existing agency-owned assets such as maintenance trucks. Essentially, all vehicle original equipment manufacturers are developing integrated automation packages or are partnering with other developers to supply the software systems for automated driving.

Driverless Systems (Air)

Similar to automated ground vehicles, UASs are also typically proprietary and closed systems with integrated command and control software that only manages UASs made by the same manufacturer. Line-of-sight UAS operation is now common and applied in a variety of industries. Beyond visual line of sight (BVLOS) and related airspace deconfliction and safety regulations are necessary policies for deployment of autonomous UASs for public agency use, including public safety agencies. The near-term benefits of autonomous UASs for TSMO could be substantial, but depending on the development of acceptable use regulations, technical standards, and operating policies. These regulatory and policy developments will be required along with the resolution of technical challenges of autonomous flight, sense-and-avoid, intervehicle communication, and mission tasking. The use of a UAS for construction surveillance is illustrated in figure 14.

Future Directions and Likely Timeframes for Maturity of Artificial Intelligence Technologies

Due to the potential of AI applications across a variety of industries, Microsoft, Google, Amazon, International Business Machines (IBM), and thousands of start-ups and other major corporations and Government agencies are pouring substantial investment into AI. One trend is the migration of some AI software to purpose-built AI hardware. This is not surprising as it has been the trend of essentially all previous software tech (that is required to run in real time) to be transitioned from software to hardware to be effective. For example, Optical Character Recognition (OCR) was once only available to the Postal Service, but now anyone can buy a $40 multifunction printer/copier, which can provide OCR translation of any scanned document. The OCR process in such a device is embedded in a microprocessor chip that is quite inexpensive.

Microsoft's project Brainwave is one such effort to move Azure Machine Learning technologies to hardware to provide real-time performance of anomaly detection in high-speed processes, such as production lines or magnetic resonance imaging machines, for example.⁶¹ In the near term, IOOs will probably not require purpose-built AI hardware for any specific application or the AI hardware will be embedded in the procurement of the application or system (e.g., driverless vehicles with specialized AI processors on board).

Other major AI efforts at Microsoft focus on healthcare and genomics, customer relationship management, and digital assistants, as well as integration of their Azure Machine Learning technologies with their popular Power BI business intelligence desktop tool.⁶² Many IOOs that have standardized on the Microsoft Office family of software products may have some use of Power BI already or are considering Power BI for various data analysis and "dashboard" applications.

Google AI is focused on improving image recognition performance with AutoML by leveraging their global database of images to transfer learning of their own models to yours, reducing the number of training images from hundreds of thousands to a few hundred.⁶³ Similarly, their focus in 2019 and beyond is reducing the expertise necessary to stand up an AI system, since the gap between AI talent (software and database specialists with understanding of AI technologies) and the need for AI developers is significant. Google expects products such as AI Hub, KubeFlow Pipeline, Deep Learning Virtual Machines, AI Platform (beta), and Dopamine,⁶⁴ to help continue to close the gap between the need for software engineering expertise and viable AI applications. Both of these focus areas are critical for IOO uses of AI technology. First, imagery analysis is probably the number one application of AI to TSMO functions and second, IOOs (in general) typically lack the software and database expertise to develop and maintain AI applications internally.

The digital assistant direction for Google is centered around Duplex (as discussed earlier). While currently Duplex is focused on making phone calls for tasks such as scheduling appointments, the technology that enables context-sensitive conversations will open up new avenues for digital assistants. As consumers can currently turn on and off lights, set the thermostat, and perform other household actions through voice commands, many TMC operations will likely be able to become voice enabled for those agencies that are so inclined.

Amazon's strategy is comprehensive and focused on AI–purposed hardware and chips (project Nitro),⁶⁵ making data lakes easier to stand up,⁶⁶ and providing training modules on how to use their suite of software products. Amazon also launched a machine learning model marketplace, SageMaker GroundTruth, which automates data labeling in training sets, and RoboMaker, a service that helps developers build and deploy robotic applications (and a $400 miniature autonomous vehicle called DeepRacer for developing automated driving and a racing league to go with it).⁶⁷ Similar to the implications of Google and Microsoft's future strategies on IOOs, the general trend of Amazon's services is towards simplification of development of AI applications for use by more and more customers without deep knowledge of software development and database management.

Summary of Commercial Artificial Intelligence Development

There are thousands of companies competing for dollars in the AI space across essentially every consumer and Government market as the technology continues to mature. Technologies from major platform providers and open-source tools they have either developed or adopted tend to underpin most software and hardware AI products. As with Big Data a few years ago, hype in the capabilities of AI is at a peak. As time moves on, these technologies will come closer and closer to "plug and play," but currently there is still a reasonably large barrier between the dreams of AI–enabled TSMO applications and the need for significant expertise and investment to make those dreams a reality. As fast as the pace of development of AI tools and technologies is progressing, within the next five years, AI applications may find their way from research experiments and pilot demonstrations to fully scalable applications.

Common trends in AI development over the next five years, according to Forbes, will be:⁶⁸

• Development of AI–specific hardware chips for embedding machine learning and training in consumer products, industrial processes, and vehicles.
• Movement of machine learning models from centralized Cloud systems to edge Internet of Things (IoT) devices.
• Interoperability among neural network modeling systems and frameworks via Open Neural Network Exchange (ONNX), a platform-neutral standard supported by the AI industry.⁶⁹
• Automated machine learning with AutoML—speeding the process of building and deploying neural networks.
• Application of AI analysis to IT operations.
• Continued evolution of chatbots and virtual assistants into more comprehensive, context-sensitive question and answering functions.
• Deployment of consumer-ready automated vehicle services.
• Democratization of machine learning services and software to professionals without deep software development and database management skills.
• Improvement of AI responsibility, transparency, and morality; the removal of systematic biases against minorities.

In the field of digital video processing, since existing products have already emerged for TSMO, the pace may be faster. Driverless vehicles are likely to be available to TSMO agencies within the near term and BVLOS-automated UAS operations in the medium term. In the next chapter we present some case studies of AI–enabled applications at DOTs that have been deployed already or are in the process of being piloted.