strange files: malware?

[kindzma]

a random dir /a on a VM that seemed to be struggling produced a list of unexpected files:

07/09/2023  09:28 PM            20,000 ZZZZZ2852017353.doc
07/21/2023  10:10 PM            50,240 !!!!!2729304900.doc
07/21/2023  09:34 PM         3,000,000 XORXOR2654977376.doc
07/21/2023  09:53 PM             1,024 smTlX4069337007.txt
07/09/2023  09:22 PM             2,024 ZZZZZ2452917832.docx
07/17/2023  08:59 PM             4,048 !!!!!910538317.pem
06/27/2023  08:58 PM            10,000 XORXOR2801197100.jpg
07/21/2023  08:27 PM            20,000 smTlX1631532574.png
07/21/2023  08:42 PM            25,000 ZZZZZ2426080075.bmp
07/21/2023  08:44 PM            30,000 !!!!!503919568.eml
07/09/2023  08:58 PM           100,000 XORXOR2564426092.xls
06/27/2023  08:53 PM           150,000 smTlX2350243133.xlsx
07/17/2023  09:41 PM           175,000 ZZZZZ3070700973.mdb
07/21/2023  09:53 PM           200,000 !!!!!533403438.ppt
07/21/2023  09:04 PM           225,000 XORXOR395504056.pps
07/09/2023  08:50 PM           250,000 smTlX634944309.pptx
07/09/2023  09:13 PM           275,000 ZZZZZ2775334046.pdf
06/27/2023  08:26 PM           300,000 !!!!!3608986092.avi
06/27/2023  08:56 PM           350,000 XORXOR189161240.db
06/27/2023  09:53 PM           350,000 smTlX416169661.pst
06/27/2023  09:54 PM           400,000 ZZZZZ2740091908.sql

Notes:

• The files aren't showing up in file explorer (even if I enable hidden and system files)
• Cortex XDR shows nothing unusual and no recent alerts
  
• Cortex also shows 545 files scanned in the directory while dir /a /s - over a thousand...

Will open a case with Palo Alto; meanwhile, questions:

• anyone seen these types of files and if so, any hint what this could mean?
• best way to unhide these files so it's easier to, say, zip them, move them around, submit for investigation?
• is there a good place to upload one of these for quick analysis?

Thanks!

[kindzma]

It looks like we can't open a support case: no active subscription or entitlement? (I am fairly new at my job, and it's my first day in the Palo Alto Customer Support Portal and am still figuring things out...)

Is there a definitive way to see what our support subscription is, or entitlement level (or lack thereof) short of not seeing the "get help" buttons in several places mentioned in "How to Open a Support Case and Verify Entitlement"? (The article's instructions on how to verify entitlement by "reviewing the "Support Type" column" in existing cases - are useless to us as we seem to have no existing cases.)

[neelrohit]

Hi @kindzma ,

these are honeypot/decoy files deployed by cortex xdr agent to prevent against ransomware events. The methodology of ransomware detection of Cortex XDR is by the method of deploying honeypot files in different locations depending upon the mode and any attack which is vectored at file modification will be prevented when the attacker process touches any of these files.

[kindzma]

Thanks!

• Would you have a link to an article describing the method and confirming definitively the files are benign?
• Is it normal for these files to exist in pretty much all directories I checked?
• Could they contribute to the explosive growth of disk space used on this specific server? (~95GB / mo)
• Any reason these files only seem to exist on one server (out of 100s where XDR is running)?

[neelrohit]

Hi @kindzma ,

• Since our tech docs are public by nature, we see a serious concern in exposing this information in public docs. With that being said, since this also is a public forum, it would be great if you could alter/mask/remove the data of the filenames provided in the list in your query above. On the query of these files being definitely benign, you can try running malware scan on the endpoint and you should be able to see that none of them would be flagged as malicious. If there is one, that technically would not be file placed by XDR agent.
• Yes, it really depends upon how you have configured your malware profile for anti-ransomware module. We have "Normal" mode and then we have "Aggressive" mode. Normal mode creates honeypot files at many places as per the randomisation logic and does have a limited scope(custom folders not linked to system processes in root, etc.). While the "Aggressive" mode as suggests creates files pretty much everywhere.
• It would depend on Normal/Aggressive configuration and also the number of folder paths and file paths created over a course of the month at targetted system locations. If the folder creation activity at critical spaces is intensive, so would be for this module. We have had a use case to test and configure Aggressive mode on a file server for a month and we found that the honeypot files were created and almost took 4-5Gb of disk space on the server. It soon became better when we reverted to normal mode and rebooted the server.
• It would be everywhere. They are just set to hidden. Again, you might want to check the policy configuration on this VM.

Hope this helps!

Please mark the response as "Accept as Solution" if it answers your query.

[Antony_Chan]

Came across this and I'm pretty sure those are decoy files deployed by anti-ransomware modules.
if you do anything with it, new alert will be generated in the XDR management console. I'm not sure how and why it got exposed to you, normally its hidden. The case i submitted back then was due to a bug that these files somehow were revealed and it got fixed in a newer version.

[kindzma]

Neel - thanks for the answer. How would you like me to mask them? (Provide an example, please.)

(Aren't the filenames and any relevant metadata easily available to anyone - including and especially blackhats - by doing basic checks an a system with XDR running in "aggressive" mode? If I ran into those files - and got seriously spooked - by running 'dir /a', wouldn't blackhats be much wiser about this?)

The bottom line for me (and for my org) here is what Palo Alto can do - and should have done - to prevent org lockdowns, five alarm emergencies, or simply multi-hour wild goose chases that these files can cause. PAN is the one placing them there, and while it may have great reasons for why it does it the way it does while keeping org's first responders in the dark, it should also take great care to avoid unintended consequences like this one. I can think of a few (ways to let a legitimate first responder know what's happening) - and I am sure PAN can think of even better ways.

P.S. Would greatly appreciate not getting into yet another blame game like this one - thanks! (I have a bad headache after spending almost a full day yesterday on this - it was really stressful - and you can guess whom I am thanking for it... :))

[neelrohit]

Hi @kindzma ,

Not to fingerpoint, but there are lot of things which organizations can do products evolve. Not to blame anyone for it, but first responders who would be wise enough can also have a nack ask these questions and get first hand answers during pre-sales and also with professional and education services that we have.  The blackhats would have the info doesn't mean we expose it to even the ones who would not want to be red teaming on things. We suggested masking or removing inorder to ensure you as org do not share information which goes beyond the privacy circle as such. 

With that being said, we appreciate delivering responses in a non-blame game fashion provided where we do not need to defend for something which looks like a targetting rather than nice questions. 

[kindzma]

Ouch... 😀 (That's exactly the finger-pointing exercise I was hoping to avoid. "Wise enough"? Is that how we play it, implying your customer is not "wise enough"? 😀 Also: how often do 1st responders sit in pre-sales meetings? Since when are pre-sales meetings a valid component in incident response playbooks? 🤣)

Thanks for confirming where you and Palo Alto stand on this. 👏

[neelrohit]

Hi @kindzma ,

Not that I meant that our customers aren't wise enough. But finger pointing and blaming at what vendors can do and what not is not the context for this forum. We are not attempting to defame an person, organization or a product in this forum, rather answering questions only. Please note that being a customer for Palo Alto also means that our customer have collective belief in our products and have opted it with the test and trust. 

I am glad that you understand that we at Palo Alto Networks stay firm to defend on use cases which are possibly false implications on our product and also stand welcoming the questions for answers to help our customers with their keen to understand the functions.

As a result, if there are questions which is probably not openly documented,  we have teams who can ensure our customers (including first responders ) do get the information delivered in a secured and detailed fashion as and when they enquire about it. I believe we are misunderstanding this forum as a source of training and possibly you could look for webinars and resources which could help answer your questions better and build your trust further than just fingerpoint.

And, for pre-sales, yes, it may vary, but many a times first responders do also participate in PoC for the product. 

Because you have lot of issues to point at, I would recommend reaching out to your customer success teams if you have one and you can also reach your SEs to guide and enable you on the detailed functionality of the product and you can also raise feature requests if you think could be a use case with them.

Hope that clarifies all the misalignments in conversation and also the expectations on questioning and commenting on this forum.

[kindzma]

Not at all, not by a mile.

Quite a few crucial missing parts in how I'd like a vendor to handle situations like this (and how I strive to handle similar incidents myself).

It's such a classic case of finger-pointing and deflection that it may be worthy of a separate write-up, something like "first response: taking ownership - or what not to do with the customer complains". (I'll think of something that sounds better, and will be more focused on what can be learned from this, vs. what cannot. Like I said, not interested in finger-pointing - only in responsible, holistic approach to incidents and what we can all learn from this.)

[kindzma · Accepted solution]

P.S. My summary of this issue.

1. As @neelrohit mentioned here, these are decoy / honeypot files placed by Cortex XDR agent attempting to detect ransomware attacks. Given it was unclear whether Neel represented Palo Alto, i.e. that it was an official response from them, and the severity of the issue should the files have been malware, I was hesitant to declare an all-clear on this internally. I was also not having lots of luck getting a response from PAN Support.)
2. These appear to be links (rather than actual files), with the actual files residing in 'C:\ProgramData\Cyvera\Ransomware\1234567890123456789\' or a similar directory. (Which was perhaps the more definitive indication of the files not being malware.)
   • A way to check that it's a link in Powershell:
     
     Get-Item "<suspect filename>" | Format-List -Property *
     This should display a long list of properties including a line like this:
     Target : {C:\ProgramData\Cyvera\Ransomware\3196102225123340985\ZZZZZ2852017353.doc}
     ... which indicates that it's a link, and the actual file's path.
     
   • I have yet to figure out a way to check a filesystem for these links (count them, estimate their performance and size impact).
3. Cortex XDR did not flag anything when scanning the directories containing those links. Given the scan summary appeared to not include those files in total "files scanned" counts, and did not indicate any ransomware protection activity, this did not feel as trustworthy.
4. In extreme cases (like ours), these links are placed in every directory in the file system. Per @neelrohit's comments, this should only happen in "aggressive" ransomware protection mode - yet in our cases this seemed to be happening in "standard" mode as well.
5. We saw no indication of this activity (ransomware protection + creation of decoy / honeypot files and links) in Cortex XDR console or dashboard, and thus were unsure what's happening until we discovered the files were links pointing to hidden files in one of Cortex XDR directories. (See (2) above.)
6. In addition, we submitted one of those actual files to another AV vendor for analysis and got a "nothing bad found" response.
7. As of this morning, these links are no longer present in several directories where they were present before. Perhaps PAN Support cleaned them up through the dashboard. (Still waiting for a post-mortem.)
8. This may have been a case of a runaway process, bug, or otherwise unexpected behavior given all of the above.

What I am wishing for (that would have happened instead of what did):

• A clear indication of ransomware protection activity that's easily (enough) accessible to a local admin encountering the issue.
  • (Yes I realize a "local admin" could be someone maliciously accessing the system - and can still think of a way to securely present the necessary information to a legitimate first responder.)
• Vendor taking ownership of their product's behavior, whether expected or not - by responding to incidents quickly. (This did not happen here.)
• Vendor documenting this behavior, possible side effects, ways to stop or mitigate it should it have severe adverse impact on performance - in internally accessible documentation (e.g. customer portal) and quickly referencing it in similar incidents (did not happen here).
• A way to automate checking for similar behavior as well as its impact across the enterprise, answering questions such as:
  • How many honeypot / decoy files have been or are being placed by Cortex XDR in a given file system over a certain time?
  • What impact might this behavior have on disk response time, queue length and other system resources?

Hope this helps someone else having a similar issue!

[neelrohit]

An attempt to end  discussion with an evidence of test result is posted below.

If this is not the expected behaviour in your environment, please open a support ticket and our support team will be able to distinguish between break fix and configuration changes: