Locator

Contact

Locator

Vet Centers

Regional Benefits Offices

Regional Loan Centers

Cemetery Locations

VA Technical Reference Model v 24.11

Hortonworks Data Platform (HDP)

General Information

Technologies must be operated and maintained in accordance with Federal and Department security and privacy policies and guidelines. More information on the proper use of the TRM can be found on the TRM Proper Use Tab/Section.

Website:	Go to site
Description:	Hortonworks Data Platform (HDP) is an open-source implementation of Apache Hadoop. HDP is a software framework consisting of libraries, distributed file system (Hadoop Distributed File System [HDFS]), management platform (YARN) and programming module (MapReduce) for storage and large-scale processing of data-sets on clusters of commodity hardware. This technology is used for storing, processing, and analyzing large amounts of data. The TRM decisions in this entry only apply to technologies and versions owned, operated, managed, patched, and version-controlled by VA. This includes technologies deployed as software installations on VMs within VA-controlled cloud environments (e.g., VA Enterprise Cloud (VAEC)). Cloud services provided by the VAEC, which are listed in the VAEC Service Catalog, and those controlled and managed by an external Cloud Service Provider (i.e., SaaS) are not in the purview of the TRM. For more information on the use of cloud services and cloud-based products within VA, including VA private clouds, please see the Enterprise Cloud Solutions Office (ECSO) Portal at: https://dvagov.sharepoint.com/sites/OITECSO
Technology/Standard Usage Requirements:	Users must ensure their use of this technology/standard is consistent with VA policies and standards, including, but not limited to, VA Handbooks 6102 and 6500; VA Directives 6004, 6513, and 6517; and National Institute of Standards and Technology (NIST) standards, including Federal Information Processing Standards (FIPS). Users must ensure sensitive data is properly protected in compliance with all VA regulations. Prior to use of this technology, users should check with their supervisor, Information Security Officer (ISO), Facility Chief Information Officer (CIO), or local Office of Information and Technology (OI&T) representative to ensure that all actions are consistent with current VA policies and procedures prior to implementation.
Section 508 Information:	This technology has not been assessed by the Section 508 Office. The Implementer of this technology has the responsibility to ensure the version deployed is 508-compliant. Section 508 compliance may be reviewed by the Section 508 Office and appropriate remedial action required if necessary. For additional information or assistance regarding Section 508, please contact the Section 508 Office at Section508@va.gov.
Decision:	View Decisions

Decision Source:	TRM Mgmt Group
Decision Process:	One-VA TRM v24.9
Decision Date:	09/23/2024 at 17:31:19 UTC
Aliases:	distributed computing
Introduced By:	TRM Request
Vendor Name:	Cloudera

Vendor Release Information

The Vendor Release table provides the known releases for the TRM Technology, obtained from the vendor (or from the release source).

You must be logged into the RSAA website prior to clicking the Attestation URLs or the URLs won't open.

Version	Release Date	Vendor End of Life Date	Vendor Desupport Date
1.3.x	05/28/2013
2.0.x	10/22/2013
2.1.x	04/22/2014
2.3.x	07/22/2015
2.4.x	05/05/2016
2.6.x	06/01/2017
3.0.x	06/18/2018
3.1.x	10/01/2020

Current Decision Matrix (09/17/2024)

Users must ensure their use of this technology/standard is consistent with VA policies and standards, including, but not limited to, VA Handbooks 6102 and 6500; VA Directives 6004, 6513, and 6517; and National Institute of Standards and Technology (NIST) standards, including Federal Information Processing Standards (FIPS). Users must ensure sensitive data is properly protected in compliance with all VA regulations. Prior to use of this technology, users should check with their supervisor, Information Security Officer (ISO), Facility Chief Information Officer (CIO), or local Office of Information and Technology (OI&T) representative to ensure that all actions are consistent with current VA policies and procedures prior to implementation.

The VA Decision Matrix displays the current and future VAIT position regarding different releases of a TRM entry. These decisions are based upon the best information available as of the most current date. The consumer of this information has the responsibility to consult the organizations responsible for the desktop, testing, and/or production environments to ensure that the target version of the technology will be supported.

Legend:

White	Authorized: The technology/standard has been authorized for use.
Yellow	Authorized w/ Constraints: The technology/standard can be used within the specified constraints located below the decision matrix in the footnote[1] and on the General tab.
Gray	Authorized w/ Constraints (POA&M): This technology or standard can be used only if a POA&M review is conducted and signed by the Authorizing Official Designated Representative (AODR) as designated by the Authorizing Official (AO) or designee and based upon a recommendation from the POA&M Compliance Enforcement, has been granted to the project team or organization that wishes to use the technology.
Orange	Authorized w/ Constraints (DIVEST): VA has decided to divest itself on the use of the technology/standard. As a result, all projects currently utilizing the technology/standard must plan to eliminate their use of the technology/standard. Additional information on when the entry is projected to become unauthorized may be found on the Decision tab for the specific entry.
Black	Unauthorized: The technology/standard is not (currently) permitted to be used under any circumstances.
Blue	Authorized w/ Constraints (PLANNING/EVALUATION): The period of time this technology is currently being evaluated, reviewed, and tested in controlled environments. Use of this technology is strictly controlled and not available for use within the general population. If a customer would like to use this technology, please work with your local or Regional OI&T office and contact the appropriate evaluation office displayed in the notes below the decision matrix. The Local or Regional OI&T office should submit an inquiry to the TRM if they require further assistance or if the evaluating office is not listed in the notes below.

Release/Version Information:

VA decisions for specific versions may include a ‘.x’ wildcard, which denotes a decision that pertains to a range of multiple versions.

For example, a technology authorized with a decision for 7.x would cover any version of 7.(Anything) - 7.(Anything). However, a 7.4.x decision would cover any version of 7.4.(Anything), but would not cover any version of 7.5.x or 7.6.x on the TRM.

VA decisions for specific versions may include ‘+’ symbols; which denotes that the decision for the version specified also includes versions greater than what is specified but is not to exceed or affect previous decimal places.

For example, a technology authorized with a decision for 12.6.4+ would cover any version that is greater than 12.6.4, but would not exceed the .6 decimal ie: 12.6.401 is ok, 12.6.5 is ok, 12.6.9 is ok, however 12.7.0 or 13.0 is not.

Any major.minor version that is not listed in the VA Decision Matrix is considered Authorized w/ Constraints (POA&M).

<Past

CY2023

CY2024

CY2025

Future>

Release

1.3.x

Unapproved

Authorized w/ Constraints (POA&M)

2.0.x

Unapproved

Authorized w/ Constraints (POA&M)

2.1.x

Unapproved

Authorized w/ Constraints (POA&M)

2.3.x

Unapproved

Authorized w/ Constraints (POA&M)

2.4.x

Unapproved

Authorized w/ Constraints (POA&M)

2.6.x

Unapproved

Authorized w/ Constraints (POA&M)

3.0.x

Unapproved

Authorized w/ Constraints (POA&M)

3.1.x

Approved w/Constraints
[16, 17, 18, 19, 20]

Approved w/Constraints
[16, 17, 18, 20, 21]

Approved w/Constraints
[17, 18, 20, 21, 22]

DIVEST
[17, 18, 20, 21, 22, 23]

Authorized w/ Constraints (DIVEST)
[17, 18, 20, 21, 22, 23]

Authorized w/ Constraints (POA&M)

Decision Constraints


[1]	At time of writing, product does not provide native data at rest encryption. This technology should not be used to store PII/PHI sensitive data, unless used with third-party FIPS 140-2 validated encryption products.

[2]	As of April 23, 2015, per the Deputy CIO of Architecture, Strategy and Design (ASD), all technologies in use by the VA require an assessment by the VA Section 508 office. Section 508 of the Rehabilitation Act Amendments of 1998 is a federal law that sets the guidelines for technology accessibility. A VA Section 508 assessment of this technology has not been completed at the time of publication. Therefore, as of April 23, 2015 only users of this technology who have deployed the technology to the production environment, or have project design and implementation plans authorized, may continue to operate this technology. In the case of a project that has implemented, or been authorized for a specific site or number of users, and that project needs to expand operations to other sites or to an increased user base, it may do so as long as the project stays on the existing version of the technology that was authorized or implemented as of April 22, 2015. Use of this technology in all other cases is prohibited.

[3]	At the time of writing, the technology does not provide native data at rest encryption. This technology should not be used to store Personally Identifiable Information (PII) / Protected Health Information (PHI) sensitive data unless used with third-party FIPS 140-2 validated encryption products.

[4]	Veterans Affairs (VA) users must ensure VA sensitive data is properly protected in compliance with all VA regulations. All instances of deployment using this technology should be reviewed by the local ISO (Information Security Officer) to ensure compliance with VA Handbook 6500.

[5]	Users should check with their supervisor, Information Security Office (ISO) or local OIT representative for permission to download and use this software. Downloaded software must always be scanned for viruses prior to installation to prevent adware or malware. Freeware may only be downloaded directly from the primary site that the creator of the software has advertised for public download and user or development community engagement. Users should note, any attempt by the installation process to install any additional, unrelated software is not authorized and the user should take the proper steps to decline those installations.

[6]	Due to potential information security risks, cloud based versions of this product are not permitted without a waiver signed by the Deputy CIO of ASD based upon a recommendation from the Architecture and Engineering Review Board (AERB). In addition, cloud based features of this software may not be used without an Enterprise Security Change Control Board (ESCCB) approval to ensure that confidential organization and/or PII/PHI data are not compromised (ref: VA Directive 6004, VA Directive 6517 and VA Directive 6513). Use of public cloud storage requires documented Federal Risk and Authorization Management Program (FedRAMP) compliance and a Memorandum of Understanding / Interconnection Security Agreement (MOU/ISA) between the vendor and VA prior to ESCCB review.

[7]	Veterans Affairs (VA) users must ensure VA sensitive data is properly protected in compliance with all VA regulations. All instances of deployment using this technology should be reviewed by the local ISO (Information Security Officer) to ensure compliance with VA Handbook 6500.

[8]	At the time of writing, the technology does not provide native data at rest encryption. This technology must not be used to store Personally Identifiable Information (PII) / Protected Health Information (PHI) sensitive data unless used with third-party FIPS 140-2 validated encryption products.

[9]	Due to potential information security risks, cloud based technologies may not be used without the approval of the Enterprise Cloud Solution Office (ECSO). This body is in part responsible for ensuring organizational information, Personally Identifiable Information (PII), Protected Health Information (PHI), and VA sensitive data are not compromised. (Ref: VA Directive 6004, VA Directive 6517, VA Directive 6513 and VA Directive 6102).

[10]	This technology has potential to be deployed with Secure Hypertext Transfer Protocol (S-HTTP). Users must ensure that Secure Hypertext Transfer Protocol (S-HTTP) is not enabled when using this technology.

[11]	Due to potential information security risks for cloud-based technologies, users should coordinate closely with their facility ISSO for guidance and assistance on cloud products. If further guidance is needed contact the Enterprise Cloud Solution Office (ECSO), which is the body responsible for new software development in and migration of existing systems to the VA Enterprise Cloud (VAEC) and ensuring organizational information, Personally Identifiable Information (PII), Protected Health Information (PHI), and VA sensitive data are not compromised within the VAEC. For information about Software as a Service (SaaS) products or to submit a SaaS product request with the Project Special Forces (SPF) team, please use their online form. (Ref: VA Directive 6004, VA Directive 6517, VA Directive 6513 and VA Directive 6102).

[12]	This technology has potential to be deployed with Secure Hypertext Transfer Protocol (S-HTTP) which is unapproved for use. Users must ensure that Secure Hypertext Transfer Protocol (S-HTTP) is not enabled when using this technology. Per the Initial Product Review, users must abide by the following constraints: System owners should employ FIPS compliant software to ensure VA sensitive data containing PII/PHI is protected with FIPS 140-2 validated technology when applicable. HDP uses the Apache Hadoop framework for its datasets. Apache Hadoop is not authorized for use within the VA on the One-VA TRM. Other solutions should be considered. HDP is an open source solution. While HDP does release updates to their platform, relying on vendor support may result in slow response times and non-timely patches. Due to potential information security risks, SaaS/PaaS solutions must complete the Veterans-Focused Integration Process Request (VIPR) process where a collaborative effort between Demand Management (DM), Enterprise Program Management Office Information Assurance (EPMO IA), Project Special Forces (PSF), Enterprise Cloud Solutions Office (ECSO), Chief Technology Officer (CTO), and stakeholders determines the SaaS/PaaS category during the Discovery Phase. All SaaS and Non-AWS/Azure (VAEC) PaaS assets are routed to EPMO IA for Analysis and Approval to Operate (ATO) with technical oversight, acquisition, production, and sustainment provided by PSF.

[13]	Veterans Affairs (VA) users must ensure VA sensitive data is properly protected in compliance with all VA regulations. All instances of deployment using this technology should be reviewed by the local ISO (Information Security Officer) to ensure compliance with VA Handbook 6500.

[14]	Due to potential information security risks for cloud-based technologies, users should coordinate closely with their facility ISSO for guidance and assistance on cloud products. If further guidance is needed contact the Enterprise Cloud Solution Office (ECSO), which is the body responsible for new software development in and migration of existing systems to the VA Enterprise Cloud (VAEC) and ensuring organizational information, Personally Identifiable Information (PII), Protected Health Information (PHI), and VA sensitive data are not compromised within the VAEC. For information about Software as a Service (SaaS) products or to submit a SaaS product request with the Project Special Forces (SPF) team, please use their online form. (Ref: VA Directive 6004, VA Directive 6517, VA Directive 6513 and VA Directive 6102).

[15]	Due to potential information security risks for cloud-based technologies, users should coordinate closely with their facility ISSO for guidance and assistance on cloud products. If further guidance is needed contact the Enterprise Cloud Solution Office (ECSO), which is the body responsible for new software development in and migration of existing systems to the VA Enterprise Cloud (VAEC) and ensuring organizational information, Personally Identifiable Information (PII), Protected Health Information (PHI), and VA sensitive data are not compromised within the VAEC. For information about Software as a Service (SaaS) products or to submit a SaaS product request with the Project Special Forces (PSF) team, please use their online form. (Ref: VA Directive 6004, VA Directive 6517, VA Directive 6513 and VA Directive 6102).

[16]	Due to potential information security risks for cloud-based technologies, users should coordinate closely with their facility ISSO for guidance and assistance on cloud products. If further guidance is needed contact the Enterprise Cloud Solution Office (ECSO), which is the body responsible for new software development in and migration of existing systems to the VA Enterprise Cloud (VAEC) and ensuring organizational information, Personally Identifiable Information (PII), Protected Health Information (PHI), and VA sensitive data are not compromised within the VAEC. For information about Software as a Service (SaaS) products or to submit a SaaS product request with the VA OIT Product Engineering team, please use their online form. (Ref: VA Directive 6004, VA Directive 6517, VA Directive 6513 and VA Directive 6102).

[17]	Users must not utilize the Apache Hadoop, as it is unapproved for use on the TRM. This technology has potential to be deployed with Secure Hypertext Transfer Protocol (S-HTTP) which is unapproved for use. Users must ensure that Secure Hypertext Transfer Protocol (S-HTTP) is not enabled when using this technology. Per the Initial Product Review, users must abide by the following constraints: HDP will require a 3rd party FIPS 140-2 certified solution for any data containing PHI/PII or VA sensitive information. System owners should use the latest version of this product and monitor both the CVE Details and NIST National Vulnerability Database websites for any new security vulnerabilities. Users should check with their supervisor, Information Security Office (ISO) or local OIT representative for permission to download and use this software. Downloaded software must always be scanned for viruses prior to installation to prevent adware or malware. Freeware may only be downloaded directly from the primary site that the creator of the software has advertised for public download and user or development community engagement. Users should note, any attempt by the installation process to install any additional, unrelated software is not authorized and the user should take the proper steps to decline those installations. Due to potential information security risks, SaaS/PaaS solutions must complete the Veterans-Focused Integration Process Request (VIPR) process where a collaborative effort between Demand Management (DM), Enterprise Program Management Office Information Assurance (EPMO IA), Digital Transformation Center (DTC), Enterprise Cloud Solutions Office (ECSO), Chief Technology Officer (CTO), and stakeholders determines the SaaS/PaaS category during the Discovery Phase. All SaaS and Non-AWS or Azure (VAEC) PaaS assets are routed to EPMO IA for Analysis and Approval to Operate (ATO) with technical oversight, acquisition, production, and sustainment provided by DTC.

[18]	Technology must remain patched and operated in accordance with Federal and Department security policies and guidelines in order to mitigate known and future security vulnerabilities.

[19]	Veterans Affairs (VA) users must ensure VA sensitive data is properly protected in compliance with all VA regulations. All instances of deployment using this technology should be reviewed by the local ISSO (Information System Security Officer) to ensure compliance with VA Handbook 6500.

[20]	Users should check with their supervisor, Information System Security Officer (ISSO) or local OIT representative for permission to download and use this software. Downloaded software must always be scanned for viruses prior to installation to prevent adware or malware. Freeware may only be downloaded directly from the primary site that the creator of the software has advertised for public download and user or development community engagement. Users should note, any attempt by the installation process to install any additional, unrelated software is not authorized and the user should take the proper steps to decline those installations.

[21]	Veterans Affairs (VA) users must ensure VA sensitive data is properly protected in compliance with all VA regulations. All instances of deployment using this technology should be reviewed by the local ISSO (Information System Security Officer) to ensure compliance with both VA Handbook 6500 and VA Directive 6500.

[22]	Due to potential information security risks for cloud-based technologies, users should coordinate closely with their facility ISSO for guidance and assistance on cloud products. If further guidance is needed contact the Enterprise Cloud Solution Office (ECSO), which is the body responsible for new software development in and migration of existing systems to the VA Enterprise Cloud (VAEC) and ensuring organizational information, Personally Identifiable Information (PII), Protected Health Information (PHI), and VA sensitive data are not compromised within the VAEC. For information about Software as a Service (SaaS) products or to submit a SaaS product request, visit the Product Marketplace. (Ref: VA Directive 6004, VA Directive 6517, VA Directive 6513 and VA Directive 6102).

[23]	The Federal Information Processing standards (FIPS) 140-2 certification status of this technology was not able to be verified. This technology will require a 3rd party FIPS 140-2 or 140-3 certified solution for any data containing PHI/PII or VA sensitive information, where applicable. More information regarding the Cryptographic Module Validation Program (CMVP) can be found on the NIST website.

Note:

At the time of writing, version 3.1.6 is the most current version, released 10/01/2020.

Type	Name	Source	Description
Vendor Provided*	HDP 3.1.6 Release Notes		This reference provides the detailed information on HDP 3.1.6 Release Notes.
*NOTE: Vendor provided information is subject to the accuracy of the manufacturer.

Name	Description
No components have been identified for this entry.


Information Management Technologies Business Intelligence and Data Warehouse Platforms	Data Analytics (Statistical Analysis, Prediction, and Modeling)

General Analysis

Adoption Benefits

This is a well-documented technology.
This technology may improve productivity for specific staff whose responsibilities include large-scale data sets.
This technology is available at no cost through open-source licensing and supports VA`s open source initiative.
At the time of writing, no National Institute of Standards and Technology (NIST) vulnerabilities had been reported and no VA Cyber Security Operations Center (CSOC) bulletins had been issued for the latest versions of this technology.

Adoption Risks

The potential exists to store Personally Identifiable Information (PII), Protected Health Information (PHI) and/or VA Sensitive data and proper security standards must be followed in these cases.
This technology can potentially use Secure Hypertext Transfer Protocol (S-HTTP), which is unapproved for use.
There are other authorized solutions that provide similar functionality available on the TRM. The use of several similar solutions may increase organization requirements for support and maintenance.
HDP is not Federal Information Processing standards (FIPS) 140-2 certified.
The product uses Apache Hadoop that has numerous published vulnerabilities.
HDP can be deployed in the cloud.
Due to the rapid release schedule of this technology, the VA may be unable to update to the most recent patch and may require a deployment model requiring the use of specific versions.
This technology can potentially use Apache Hadoop, which is unapproved for use.
System scans indicate that this technology is no longer in use within the VA network.
There has been no recent development activity on this technology.
As open-source software, this product may be left without clearly defined support options which may result in suboptimal enterprise level support.

Architectural Benefits

This technology is portable as it runs on multiple TRM-authorized operating systems.

Architectural Risks

This technology includes cloud-based functionality which has potential information security risks.

- The information contained on this page is accurate as of the Decision Date (09/23/2024 at 17:31:19 UTC).

Inside VA

Budget and Performance

Inside the Media Room

National Observances

Special Events

VA Technical Reference Model v 24.11

General Information

Vendor Release Information

Current Decision Matrix (09/17/2024)

Decision Constraints

References

Technology Components

VA Categories

View the VA Categorization Framework

Open Source

Operating Systems Supported by the Technology

Technologies/Standards Relationships

Runtime Dependencies:

Optional Dependencies:

Comparable Technologies:

Companion Technologies/Supported Standards:

General Analysis

Adoption Benefits

Adoption Risks

Architectural Benefits

Architectural Risks

CONNECT

VA HOME

QUICK LIST

RESOURCES

ADMINISTRATION

VA Technical Reference Model v 24.11

General Information

Vendor Release Information

Current Decision Matrix (09/17/2024)

Decision Constraints

References

Technology Components

VA Categories View the VA Categorization Framework

Open Source

Operating Systems Supported by the Technology

Technologies/Standards Relationships

Runtime Dependencies:

Optional Dependencies:

Comparable Technologies:

Companion Technologies/Supported Standards:

General Analysis

Adoption Benefits

Adoption Risks

Architectural Benefits

Architectural Risks

EMAIL UPDATES

VA Categories

View the VA Categorization Framework