Big Gains Foreseen For Big Data In Military Applications

  • Amreen Khan
  • Tuesday, May 16, 2017 @ 10:51 AM
  • 46484
Big Gains Foreseen For Big Data In Military Applications

Amid the proliferation of real-time data from sources such as mobile devices, web, social media, sensors, log files and transactional applications, Big Data has found a way to deliver a wealth and variety of mission-critical data to today's warfare.

The big data techniques used by businesses to mine databases are being adopted by the military to glean more information from many different types of data it collects—from drones, automated cybersecurity systems, terrorist databases and many other sources. The technology will not only help warfighters on the battlefield, it will also be used to improve fields ranging from software development to vehicle maintenance.

Market size

According to a new report from SNS Research titled The Big Data Market: 2017- 2030 – Opportunities, Challenges, Strategies, Industry Verticals & Forecasts, the investments in Big Data solutions are expected to grow at a CAGR of roughly 10 percent over the next four years, reaching $76 billion by the end of 2020.

Big Data investments will account for over $57 Billion in 2017 alone, the report says.

The report also states that almost a third (30 percent) of all Big Data workloads will be processed via cloud services.

Big Gains Foreseen For Big Data In Military Applications

Big Data Investments

Big Data technologies include distributed computational systems, distributed file systems, massively parallel-processing (MPP) systems, and data mining based on grid computing, etc.

The US Department of Defense announced a FY 2017 budget request to invest in “Big Mechanism,” an effort to automate computational intelligence for biology, cyber, economics, social science, and military intelligence.

As per the budget, DARPA projects that its planned investment related to big data will rise from approximately $216M in FY 2015 to nearly $269M in FY 2017, a planned increase of 24.5%.

In 2014, Pentagon's high-tech research arm, the Defense Advanced Research Projects Agency (DARPA), announced a program it calls Mining and Understanding Software Enclaves (MUSE) to improve the quality of the military's software.

It wants to develop "big code" software packages capable of managing information without distorting it or collapsing under the weight of the large data sets, the so-called big data, which is at the heart of the Obama administration's technology programs announced in 2012.

Various Pentagon big data initiatives include research by the Defense Threat Reduction Agency to track dangerous pathogens, a DARPA-directed plan to mine health data to track cancer cells, a combination of data mining and patent analysis to track "disruptive technologies" that could alter the future of military equipment and planning and counter-force programs to track the spread of weapons of mass destruction by DTRA.

Big Gains Foreseen For Big Data In Military Applications

Big Data Tools & Technologies:

1.Apache Object Oriented Data Technology (OODT): Birthed at NASA's Jet Propulsion Laboratory, it uses structured XML-based capturing of the processing pipeline that can be understood and modified by non-programmers to create, edit, manage and provision workflow and task execution.

OODT also allows for remote execution of jobs on scalable computational infrastructures so that computational and data-intensive processing can be integrated into OODT’s data processing pipelines using cloud computing and high-performance computing environments.

Apache OODT is in use in many scientific data system projects in Earth science, planetary science, and astronomy at NASA, such as the Lunar Mapping and Modeling Project (LMMP), NPOESS Preparatory Project (NPP) Sounder PEATE Testbed, the Orbiting Carbon Observatory-2 (OCO-2) project, and the Soil Moisture Active Passive mission testbed. In addition, OODT is used for large-scale data management and data preparation tasks in the DARPA MEMEX and XDATA efforts.

2. Apache Bahir:  Bahir bolsters Big Data processing by serving as a home for existing connectors that initiated under Apache Spark, and provides additional extensions/plugins for other related distributed system, storage, and query execution systems.

3. Conditioned-based Maintenance (CBM): The US Air Force officials, data scientists, and other personnel are harnessing advanced data-handling and analytics tools, both software and hardware solutions, to streamline workflows, increase efficiencies and productivity, track and manage assets, and replace scheduled/time-based maintenance with conditioned-based maintenance (CBM) – saving at $1.5 million in one year.

Specifically, Air Force officials opted to use Teradata’s Aster platform to better manage Flightline maintenance; Inventory; and Aircraft depot data.

4. Hadoop software:  Hadoop was strictly a MapReduce batch analytics platform. Today, Hadoop can run a variety of workloads from batch to interactive and real-time.

5. Organizational, Relationship and Contact Analyzer (ORCA): Developed by U.S. Military Academy, this software was initially developed and used in military operations to identify networks of insurgents, and is now being used domestically. The military will use the domestic tests as a way to hone the software for future wartime applications, while in the meantime providing police gang units with a valuable free tool that could eventually see a more widespread deployment.

6. Tableau: It is a very effective tool to create interactive data visualizations very quickly. It is very simple and user-friendly, and can be used by anyone as this tool is designed to be used by developer as well as non-developers. You can also embed the visualizations on your website.

7. D3.js: D3. JS is a JavaScript library that is used for data visualization. To convert unstructured data collected from sources like government sources, social networking, eCommerce portals, etc. into some usable form or some sort of productive or understandable form, D3.JS is a very convenient tool. This JavaScript library is mainly used for creating data visualizations by highlighting web standards using CSS, SVG, and HTML, all with the full capabilities of latest browsers and no limitations of proprietary frameworks.

8. Power BI: Power BI allows developers to create visualizations and display data in a very accurate way that you will not find such facilities in other BI tools. It can be done with minimal efforts and using existing skills. The Power BI Desktop has a large variety of standard visualizations that include a range of reports that people usually needs. These illustrations can essentially be created through Power BI Desktop or the Power BI products. Though, there are always those conditions where a diverse style of graphs, charts or tables are required to get a specific desired result. These types of customizations can be developed internally or developer can retrieve other customized visuals from the Visualization Gallery.

9. R language: R is a language and an environment to run statistical calculations and produce data graphics. According to Google Trends, R is a language and tool very much related with creating data visualizations. It offers a variety of plotting devices where some of them are whole systems that a person needs to learn like a new language. 

Big Gains Foreseen For Big Data In Military Applications

Future Trends In Big Data for Defence:

As told by John Murphy, Head of Programs - Big Data Solutions Architect, US Department of Defence, Missile Defence Agency, who will be speaking at Global Big Data Defence summit 2017 on June 26.

  1. Big Data Vs. Relational Data: Each use case or business case will require an understanding of where and how it is to be hosted. But the trend is away from traditional relational data structures and their supported infrastructure and towards general purpose big data architecture.
  2. Data management concerns to prevent loss in value: With “Big Data” there is a need to apply more traditional data maturation and validation processes to ensure that optimum data value can be achieved.
  3. Data Swamp V. Data Lake: Two approaches with big data have emerged.

The more rapid deployment and iteration cycles focus almost exclusively on using the data and information “as is” for analytics advocating for a more pristine and as close to the original data sources as possible. The resulting data set tends to become more of the “data swamp” environment with few rules on ingestion, quality and upkeep.

The second approach has elements of more traditional data management practices focusing on standardization, cleansing, auditability, consistency and over all managed care.


Big Data Analytics Firms: 




Advanced Systems Concepts, Inc. (ASCI)


By simplifying the integration of Hadoop components within an IT infrastructure, ActiveBatch Version 11 gets data into the hands of end users in real-time, empowering organizations to make better business decisions.


ActiveBatch Extension for Hadoop provides:


-          Production-ready content for components of the Hadoop ecosystem including HDFS, Hive, MapReduce, Spark, HBase, Pig, Sqoop, Oozie and more. With access to pre-built code that is pre-tested, the creation of workflows and workloads is simplified, enabling IT to build and automate end-to-end workflows in half the time.

-          HDFS Event Trigger brings the power of advanced event-driven automation to big data, improving time to insight by getting the latest data into the hands of the end user faster.

-          Ability for users to take full advantage of ActiveBatch’s advanced scheduling and automation functionality, including the creation of re-useable customizable templates. Job Steps can now be saved for future use, making it easy for other developers to re-use and re-purpose content, improving manageability and reducing the time spent scripting.




The patent-pending OLAP on Hadoop solution includes enterprise-grade functionality to improve analysts' access to the data lake, advanced security support, improved performance and scalability, and integration with additional BI tools. Kyvos allows companies to build a BI Consumption Layer directly on Hadoop, which enables analysts to transform their existing BI tools. The BI Consumption Layer also gives analysts instant, interactive access to multi-dimensional analytics at big data scale across the enterprise, with no learning curve or programming required.

Modus Operandi

BLADE technology

US Marine Corps is testing Modus Operandi’s BLADE technology for use in the field. The product generates a semantic wiki from the COP (Collecting, correlating and making sense of big data into a single, cohesive image is called creating a common operating picture) that is built from the original data inputs. If, for instance, a soldier discovers some information about a person of interest who happened to have been involved in a bombing, he can update the COP with that new piece of information. Additionally, the wiki pages for that person and that bombing event would both be updated, as would any other relevant pages.



Leverage standards-compliant SQL to support your Business Intelligence and reporting workloads. Utilize out-of-the box, scalable machine learning algorithms to deliver new analytical models and data science insights. Certified support for 3rd party visualization and analytical tools.



Aster Analytics

Aster analytics allows SQL developers to work with what they know -- SQL -- to get access to advanced analytic techniques that they would use to explore data. Examples of Aster analytic functions include SQL-MapReduce computation for complex multi-step problems; SQL-GR for graph analysis that is used in deciphering the interrelationships of people and/or things; and nPath, which provides path analysis for problems such as analyzing and optimizing how customer navigate online sites.

ITT Exelis’ Geospatial Systems

In-theater intelligence software solutions

It processes imagery and full-motion video from airborne (on-board a drone, for example) sensor systems that monitor and carry out surveillance over a large area of interest, such as regions of a city. If an event like an explosion occurs, this surveillance information is ingested into the enterprise and users can be notified almost immediately, and have access to the data. 

Innovation Lab, ATC

A4CLOUD project

A4CLOUD’s main objective was to frame an ‘advance’ approach to accountability for cloud service provision ecosystems ahead of the EU’s general data protection regulation (GDPR), which enters into force in 2019. It focuses on the Accountability for Cloud and Other Future Internet Services as the most critical prerequisite for effective governance and control of corporate and private data processed by cloud-based IT services. The A4Cloud project will increase trust in cloud computing by devising methods and tools, through which cloud stakeholders can be made accountable for the privacy and confidentiality of information held in the cloud. These methods and tools will combine risk analysis, policy enforcement, monitoring and compliance auditing. They will contribute to the governance of cloud activities, providing transparency and assisting legal, regulatory and socio-economic policy enforcement.

Massive Analytic, a London based company.

big data analytics platform

Collaborated with the Defence Science and Technology Laboratory’s Centre for Defence Enterprise (CDE) last year.


Under the deployment the UK MoD will use the system to bring together surveillance data from static and dynamic sources which will feed an interactive dashboard. The dashboard will be capable of providing insights that will help to identify threats in advance and can be used for planning different scenarios.


Cloud-based system


Application Transformation Accelerator.

Agility Platform.

DXC BizCloud.

Cloud DevOps

ILIAS Solutions

Defense Capability Management

Defense Capability Management is used to map, monitor and manage military organizations in order to make the most of their data.