This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Tech Community.

Introduction

At FastTrack for Independent Software Vendors (ISVs) and Startups, we talk to a lot of ISVs that are doing a lot of different things.  One category we have seen is customers that are deploying Internet-of-Things (IoT) devices to a customer site, where the software developed by the ISV running on that device is the primary product. 

Recently, we had a customer that has this use case.  They create specialized machine learning (ML) models for a particular industry, and those models – that intellectual property (IP) – are the result of years of development.  They lead to substantial improvements in their end customer profit margins, and they charge for their solution accordingly.

In the interests of focusing on the software on the device, and not the hardware design for the device, they use commercial, off-the-shelf (COTS) hardware, in this case, Raspberry Pi devices, running their model, where the model is stored on a microSD card on the device.  This is a relatively common model for small quantity deployments that can’t reasonably support custom hardware designs end-to-end. 

However, because the device is an open device, running common, open software, the ISV found they have had customers that remove the microSD card, copy the software off, and run it on their own hardware, then terminating the contract for the software.  Although this is technically a violation of the license agreement, and exposes the end customer to legal concerns around misappropriation of the ISV’s IP, our customer was too small to effectively fight legally in a reasonable time, and they were losing that revenue in the meantime.  Thus, one of the things they asked us for help with was around how they could, given the constraints of common, COTS hardware, limit the improper use of their IP.

This blog post describes the solution we proposed.

Disclaimer

Important disclaimer: This is a proposed example solution based on discussion with the customer and available knowledge to us at the time.  This solution is provided for the purpose of illustration only and should not be used in a production environment without proper evaluation.  THIS SAMPLE AND ANY RELATED INFORMATION ARE PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A PARTICULAR PURPOSE. We grant You a nonexclusive, royalty-free right to use and modify the solution design, provided that You agree: (i) to not use the Microsoft name, logo, or trademarks to market Your software product in which the solution is embedded; (ii) to indemnify, hold harmless, and defend Microsoft and Our suppliers from and against any claims or lawsuits, including attorneys’ fees, that arise or result from the use of the proposed solution.

High-Level Requirements

OK, now that we’re past that…

The customer really had two primary requirements:

1. The ISV needs to be able to locally log in to the deployed device for troubleshooting, maintenance, etc., in a secure way.  Off-network (e.g. remote support) access is not necessary; in fact, for this particular ISV’s customers, the devices are often running in a disconnected network and thus could not be remotely accessed in any event.
2. They need to ensure the filesystem on the device is properly secured.

Requirement 1: Login

For requirement 1, they currently use an SSH daemon on the device, as is common.  However, they were concerned about a username/password leak; they really wanted some kind of multifactor authentication.  There’s a few different ways to go about that, but they had decided on using physical hardware tokens.  Conveniently enough, our friends at Yubico make FIDO2-compliant keys, and offer a pluggable authentication module (PAM) library to support the use of FIDO2 keys.  Thus, it’s very easy to implement hardware-based authentication for a device.

The implementation is actually quite nice – it has the following interesting characteristics:

1. It acts as a second factor, meaning you must enter the correct password before you are even prompted for the hardware token.  PAM is processing the factors in order and when one fails, the process stops.  There’s a whole discussion that security professionals have had around whether this is a good design or a bad design, but I personally feel that from a troubleshooting viewpoint, it’s helpful, especially if an appropriately strong password is used.
2. If an account needs a token, but one is not present when the login process reaches that point, it doesn’t prompt you to insert one.  This means that someone won’t know that the password is correct, even if it is, without also having a hardware token inserted.  This is less helpful from a troubleshooting perspective, but it is interesting from a security viewpoint.

Thus, we demoed the use of this module to secure the SSH network access for the device.

Requirement 2: Filesystem

A common recommendation for protecting content on the microSD card on a Raspberry Pi is to glue the card in the socket, the hope being that the card will be permanently damaged if someone attempts to remove the card, thus protecting the information on the card in a very brute-force physical manner.  Our customer wasn’t thrilled with that idea for multiple reasons, including wanting to be able to swap the card out if necessary (e.g. card failure, software upgrade failure, etc.).

Thus, the solution shifts from physical lockdown of the card, to encryption of the data on the card.  In theory, if the card is encrypted, it’s not useful without the decryption key.  The problem however is now changed to one of “where do I store the key?”  In a “full” computer, this is often done using a key that is generated and then securely stored in a Trusted Platform Module (TPM) – see for example 11.4.9, Key Generation, in the TPM 2.0 Architecture Guide.  This module is physically integrated into the machine.  It's not a perfect solution, but it’s “good enough” for many use cases.  In fact, on modern “full” processors, there’s often a TPM component of sorts as part of the processor, making this even more straightforward.

I deliberately oversimplified that all a bit because the Raspberry Pi doesn’t have a TPM onboard, so the explanation is just academic rather than practical.  So, what some folks do is use a key file or the like on the boot partition, which is then used to release the root partition.  That pattern can be seen in a Linux HOWTO, a Linux Tutorial, and so on.  The issue is that there is really very little to no security here; if someone is able to physically remove the microSD card and mount it on another device, then the key is readily obtained, and the encryption becomes easily overcome “security by obscurity” and thus irrelevant.

What was our customer to do?  If you review that thread that I linked earlier that talked about gluing the microSD card in place, there’s an interesting post that’s just a link to a vendor’s community post, that in turn links to that vendor’s official documentation.  And that turns out to be the answer to this other piece.  Zymbit is a vendor that offers an add-on security module called the ZYMKEY4 that connects directly to a Raspberry Pi’s GPIO pins (or a device with compatible low number GPIO pins), can be soldered on the board, and has a proprietary mechanism to validate a hardware signature on the connected device.  If the connected device’s signature does not match, then key storage won’t be unlocked, and the boot partition won’t have access to the key to decrypt the root partition.  This is great!  This means that even if the module isn’t soldered, you can’t move it to another Raspberry Pi and have the key unlocked!

This module is a COTS module that is readily available, both directly from the vendor and from online resellers.  For the PoC demo I did for the customer, I ordered mine and had it in my hands in a couple of days, easy-peasy.  I then followed the documentation to encrypt the root partition of my demo device, and was able to demonstrate to the customer that when I removed the card from the demo Raspberry Pi and put it into another device, the partition was indeed encrypted.

Now, to be clear, this is “best effort” – this absolutely has some specific security risks, including:

• For development and test purposes, the ZYMKEY4 ships without validation of the hardware signature, meaning the security module can be used on any device and the stored secrets will be available.  This is perfect for me demoing, but in production, the ISV needs to be sure to enable “production mode” to permanently bind the key to the attached hardware.  Once that’s done, then the binding is permanent, according to the vendor.  But that means that it has to be done and it has to be done in a specific, documented way.
• In theory, a determined enough attacker can monitor the GPIO pins and sniff the traffic, which would lead to extraction of the encryption key.  For the ISV in question, that was considered an acceptable risk.  This could be somewhat helped through judicious use of glue and other physical protection – the “permanent” bonding of the ZYMKEY4 unit to the underlying single-board computer would mean that separation has limited value, unlike the microSD card – but it still wouldn’t be perfect.
• The seemingly undocumented, proprietary nature of the hardware signature mechanism will be an issue for some ISVs, although it’s not for this ISV.

The Zymbit documentation around the use of the ZYMKEY4 is sufficiently clear that someone developing an embedded solution should be able to be successful with it, in my opinion. 

Conclusion

There’s ultimately no such thing as truly perfect security – security is always a cost/benefit analysis to determine what is “good enough”.  For our customer trying to protect their IP on commodity, well-known COTS hardware, this solution was “good enough” for them to feel that their needs were met.