Migrating Historical Data into XSOAR from 3rd Party Products

[jfernandes1]

Migrating historical data into Cortex XSOAR involves a multi-phase process designed to ensure a smooth transition while maintaining functionality across both the old and new systems. The migration process can be customized based on specific requirements and constraints.

Below is a structured outline of the approach:

1. Transition Method Identification

 

Phases of Transition:

1. Content Development:
• Create custom fields, incident types, layouts, playbooks, and automations.
• Develop custom integrations if necessary.
2. Historical Data Migration:
• Transfer closed incidents from the old system into XSOAR. This phase is explained in detail in section 4. 
3. Open Incidents (XSOAR):
• Mirror all open incidents into XSOAR.
• Analysts work in XSOAR, with updates mirrored to the existing system.
• Content is developed while analysts use XSOAR.
• Predetermined cutoff to stop replicating incidents into the old system.
4. Open Incidents (Old Solution):
• Use XSOAR as an automation platform while analysts continue working in the old system.
• Build and test content in XSOAR - data mapping, format displayed data and develop automations (Scripts and Playbooks)
• Once testing is complete, automation output can be pushed to the external solution. Ex Data enrichment and auto-closure.
• Predetermined cutoff to stop pushing updates to the old platform.
5. Incident Mirror:
• Continuously replicate incidents into the old platform for compliance and reporting needs.
• Use a post-processing script for final content synchronization before incident closure.

Based on the above phases, we can create scenarios based on our requirement:

• Scenario 1: Content Development -> Historical Data -> Open Incidents (XSOAR)
• Scenario 2: Content Development -> Historical Data -> Open Incidents (Old Solution)

In both the above cases, you can add the Incident Mirror phase to continue mirroring incidents in the old system. This is usually done to ensure uninterrupted compliance and management reporting.

In scenario 2, since analysts are only moved to the XSOAR once the content is tested. This ensures the least disruption and can be further mitigated by moving them in batches if permitted by cost and time factors. 

2. Data Mapping

To transition historical data into XSOAR field mapping needs to be performed between data from the existing solution to fields inside XSOAR. There are multiple field types available inside XSOAR. In some cases data manipulation might be required before field data is saved into XSOAR fields. The below table should help identify requirements based on each data type.

Source Datatype	XSOAR Datatype	Method
Short String\Long String	Short Text Long Text	Direct Mapping
Table Reference with ID	Depends on the final data type	Transformer to lookup value in secondary DB\table\API
Very Long Text Body	Long Text Markdown	Direct Mapping
Comment\Note	XSOAR War room Entry	Some automation around adding user and timestamp details.
File	File Object	Automation needs to return the file object.

The “Table Reference with Id” might need further explanation. Some tools like ServiceNow store a sys_id instead of the value. The sys_id needs to be looked up in a different table to get the actual field value. An example of this is the “Assignment Group” field. When creating scripts to perform this operation, making them a transformer is better since they can be called in mappers too.

Once the field requirement from both solutions have been identified, you should be able to create a table similar to the below.

3. Incident Type and Layout Configuration

Once you’ve created the required fields in XSOAR. The fields need to be assigned to incident layouts. The below articles will help. 

Create Incident Type -> https://docs-cortex.paloaltonetworks.com/r/Cortex-XSOAR/8/Cortex-XSOAR-Administrator-Guide/Create-an...

Incident Layouts -> https://docs-cortex.paloaltonetworks.com/r/Cortex-XSOAR/8/Cortex-XSOAR-Administrator-Guide/Customize...

You will also need to decide how you will map incidents from the existing system to XSOAR. There are 2 general approaches:

1. Map all historical incidents into a new Incident Type like “SNOW Legacy Incident”. All the fields will be mapped to this incident type. This approach is simple as it requires only a single incident type and layout.   
2. Identify a type field in the existing system, for ServiceNow it could be “Category”. Create an Incident Type in XSOAR for all possible values of the category field. You would then create the required layouts and required mappings. 

4. Final Data Migration

 

There are 2 methods to perform data migration. Via an integration or via an exported report. Use the report method when no API or integration exists for the existing solution. 

• Via Integration:
• Setup integration with fetch incidents feature.
• Use a mapper to map input data to XSOAR fields.
• Playbook to handle additional data processing, adding comments, tags, etc.
• Via Exported Report:
• Export data from the old system in CSV or JSON format.
• Ingest the report into XSOAR, with file size considerations**
• Use a playbook\automation to create new incidents from the report data.

** We recommend keeping the file size small. If the file is large, ensure that as little data as possible is pushed into the context. 

5. Post-Migration Verification

• Track statistics and counts of migrated data using XSOAR dashboards and reports.
• Address potential performance issues, especially with large data volumes and indexing in Bolt DB (relevant for XSOAR 6).

By following these steps and adapting the scenarios to the specific needs and constraints of your organization, you can effectively manage the migration of historical data into Cortex XSOAR while minimizing disruptions and ensuring data integrity.