Backdoor Attack on Deep Learning Models in Mobile Apps

Summary

Deep learning models are increasingly used in mobile applications as critical components. Researchers from Microsoft Research demonstrated that many deep learning models deployed in mobile apps are vulnerable to backdoor attacks via "neural payload injection." They conducted an empirical study on real-world mobile deep learning apps collected from Google Play. They identified 54 apps that were vulnerable to attack, including popular security and safety critical applications used for cash recognition, parental control, face authentication, and financial services. Target: ML-based Android Apps Actor: Yuanchun Li, Jiayi Hua, Haoyu Wang, Chunyang Chen, Yunxin Liu Procedure: 1. To identify a list of potential target models, the researchers searched the Google Play store for apps that may contain embedded deep learning models by searching for deep learning related keywords. 2. The researchers acquired the apps' APKs from the Google Play store. They filtered the list of potential target applications by searching the code metadata for keywords related to TensorFlow or TFLite and their model binary formats (.tf and .tflite). The models were extracted from the APKs using Apktool. 3. This provided the researchers with full access to the ML model, albeit in compiled, binary form. 4. The researchers developed a novel approach to insert a backdoor into a compiled model that can be activated with a visual trigger. They inject a "neural payload" into the model that consists of a trigger detection network and conditional logic. The trigger detector is trained to detect a visual trigger that will be placed in the real world. The conditional logic allows the researchers to bypass the victim model when the trigger is detected and provide model outputs of their choosing. The only requirements for training a trigger detector are a general dataset from the same modality as the target model (e.g. ImageNet for image classification) and several photos of the desired trigger. 5. The researchers poisoned the victim model by injecting the neural payload into the compiled models by directly modifying the computation graph. The researchers then repackage the poisoned model back into the APK 6. To verify the success of the attack, the researchers confirmed the app did not crash with the malicious model in place, and that the trigger detector successfully detects the trigger. 7. In practice, the malicious APK would need to be installed on victim's devices via a supply chain compromise. 8. The trigger is placed in the physical environment, where it is captured by the victim's device camera and processed by the backdoored ML model.

Deployers

• ML-based Android Apps

Tags

• MITRE ATLAS
• AML.CS0013