Tuesday, October 23, 2018

Crash Dumps: Do I submit them?

TL;DR: No, do not submit your crash dumps.  Consumers: No company has sane crash dump policies to ensure your privacy and PII is protected, minimized and secured.  Companies: You need to ensure that crash dumps are handled in a secure manner and that crash dumps are just that: a crash dump. Anything not directly related to a crash dump should be excluded. Usage statistics and the like do not belong in crash reports.

Why Not Send Dumps?

There is a long history of companies failing to minimize the data and to protect it.  Microsoft for years sent crash dumps over the internet in the clear (WER & Privacy conerns).  This allowed the NSA to harvest them, and develop 0-days for issues that MS failed to fix.  Google's Chrome would send a screencap of the entire Desktop along with it's crash dumps (link).  It previously would only send the window, but now sends the entire screen.  Though they provide a preview, there is no way to see exactly what information will be sent.

I do not relish in advising people to not submit crash dumps as this will impact developers ability to fix bugs.  But as with all aspects of security, companies continue to demonstrate that they are not willing to do the work that is necessary to protect user's data and their privacy.


You need to communicate to your users how crash dumps are handled.  Just saying, trust us, does not inspire confidence, as there are a large number of cases of data breaches where the company has said exactly that leading up to leaks.  The policy is the first step to demonstrating that you have thought about user's concerns and decided how you will handle their personal and sensitive data.

The policy also helps shape how employees will treat the data too.  By having the policy, it is a reiteration to the employees that user data isn't simply chaff, but that it needs to be protected and handled with care.

Just saying that it's protected by a privacy policy isn't enough.  For example, Google Chrome's Report an Issue says that the information is protected by their privacy policy, but if you read the Chrome browser Privacy Policy, there is nothing in there that says how the data is handled.  That it is handled like the rest of the data collected does not inspire confidence that the possibly confidential data that may be included will be handled with greater care.

How to handle dumps

The first step is to ensure that what is collected in the dump has minimum information needed to debug issues.  Code paths (back traces) are likely to be safe.  Data, such as arguments to functions, may include user data and needs to be carefully examined.  There are many different types of data that can be released from embarrassing (what website was visited), to security breach (including cookies/tokens for web sites that may not be yours), to confidential intellectual property leaking (source code, designs, etc).  Each of these may have different impact on the user, but should never happen.

Second, crash dumps need to be transmitted confidentially.  This means either using TLS or encrypting the dumps with a tool like GPG before sending.  This ensures that unauthorized parties are unable to view the contents.  The NSA used the dumps to gather information for their operations, which if Microsoft had properly protected their user's data, this would not have happened.

Third, they need to be stored in a secure manner and able to be expunged.  It should even be possible for the user to remove the crash dump if they discover that information was shared when it should not have been.  The life time that a company keeps the dumps should be limited.  If you haven't fixed a bug from five years ago, how do you know you can reproduce it, or that if you are able to reproduce it, that the code is still present in your current software?  It the crash is a major issue, it is likely that you'll have more recent dumps that exhibit the same issue if it is a problem, so old dumps are just not as useful compared to the data that may be present.

As crash data needs to be deleted, almost any cloud service is immediately excluded unless other precautions are used, such as encryption.  With the cloud, you have zero visibility into how the data is managed and how or when it is backed up.  Cloud providers rarely tell you their retention policies on back ups, and other policies that may keep data around.  Do they securely remove your VM's storage when they migrate it?  Do they ensure that storage is deleted from all clones, shards, servers and backups when you delete it?  If not, how long will that data stay around before it is finally expunged.

Fourth, access to dumps need to be controlled.  Auditing is a good first step to know who is accessing the data, but additional measures like limiting who has access needs to be used.  Not everyone on the team needs access to them.  As they are classified, they can be assigned to teams or people that need access to the data in them.  This helps make sure that an employee isn't trolling for nudes or other confidential information.  It should also limit how easy data is copied out of the archive.  How these controls are put in place will vary by company.

Edit:  Case in point:  I recently opened a support case with Apple.  Apple provides a program to collect data to send to them to help trouble shoot the issue.   The program collected 280 MB of data.  When uploading the data, Apple informs the user that it is their responsibility to NOT submit any personal information that they don't want.  There is no way most people are qualified to look at the data, and even redact it properly.  I attempted to do so, and it took a very long time, and I'm not sure that I got everything.  Expecting a normal computer user to be able to do this is insane.

Monday, October 15, 2018

TLS Client Authentication Leaks User Info (pre-TLS1.3)

It's been long known that TLS is not the best privacy protecting protocol in that SNI leaks what domain the client connects to.  I'm a bit surprised that I haven't seen the failure to protect user information when using client authentication mentioned, but it's likely that TLS client authentication is so rarely used, that this have not been on anyone's radar.

TL;DR: Just don't use TLS client authentication on anything before TLS 1.3.

With TLS 1.2 and earlier, if you use client authentication, the client certificate is transmitted in the clear.  This contains enough information to uniquely identify the user.  If it didn't, then there would be no way for the server to do the authentication.

The danger of this is that Eve (eavesdroppers) on path will be able to track your user's (or your) connections, where they connect from, figure out how much data they transfer between to/from your site and likely profile their usage.

I was confident that this was the case as I know that the entire handshake is in the clear.  It isn't till the Finished messages that the session becomes encrypted.  (TLS 1.3 fixed this by using a new derived key, [sender]_handshake_traffic_secret, to encrypt all the server params, which the client will use to encrypt it's response to the certificate request in the server params.)  I decided to verify that this was the case.

I generated a server and a client certificate and key:
openssl req -batch -new -newkey rsa:1024 -days 365 -nodes -x509 -keyout server.key -out server.crt
openssl req -batch -new -newkey rsa:1024 -days 365 -nodes -x509 -keyout client.key -out client.crt

I then launched the server, and included the -Verify and -CAfile options for s_server to request a client certificate:
openssl s_server -accept 5829 -cert server.crt -key server.key -Verify 5 -CAfile client.crt -debug

Then I ran tcpdump to capture the session:
sudo tcpdump -s 0 -n -i lo0 -w clientcert.tcpdump port 5829

And then the client to connect to the server:
openssl s_client -connect localhost:5829 -key client.key -cert client.crt -debug

A usual, non-client authenticated connection and close was about 17 packets, but when I included the client authentication, it became 42 packets (the answer!).

I loaded the packet capture into wireshark, applied the SSL protocol analysis and confirmed that the client certificate was present in clear text:
Wireshark shows TLS handshake with client authentication, with the client certificate displayed in plaintext.

So, there you have it.  Do not use client authentication, pre-TLS 1.3, if you care about the privacy of your users.

It is safe to use client authentication w/ a TLS 1.3 server as long as the server requires all clients be 1.3 clients.  If the key exchange algorithm is one of DHE_DSA, DHE_RSA, or an ECDH key exchange algorithm, the random bytes in the Hello messages are signed and these bytes are used by TLS 1.3 for downgrade protection.  As the signature covers these bytes, the client would be able to detect any attempts to modify the server or client handshake messages to force a downgrade before it would send the client certificate.

Thanks to Mike Hamburg for reviewing an earlier version of this blog post and pointing out that TLS 1.3 was not vulnerable to this and helping w/ some of the research to prove it.