[Neurips2022] sparcl: Specific edges and continuous learning

Source: Specialty

This article is introduced in the paper. It is recommended to read for 5 minutes

In this work, we proposed a new framework called SPARCL, which is the first study to achieve low -cost continuous learning on edge devices using sparseness.

The existing continuous study (CL) research focuses on reducing disaster forgetting, that is, simulation the performance of past tasks when learning new tasks. However, the lack of training efficiency of the CL system is insufficient, which limits the actual application of the CL system in resource limited scenarios. In this work, we proposed a new framework called SPARCL, which is the first study to achieve low -cost continuous learning on edge devices using sparseness. SPARCL achieves training acceleration and accuracy to maintain training through three aspects of sparseness, data efficiency and gradient sparseness. Proposal to learn a sparse network in the entire CL process of mission perception dynamic mask (TDM), dynamic data removal (DDR) to delete training data with less information, and dynamic gradient mask (DGM) to sparse gradient update. These methods have not only improved efficiency, but also further reduce the forgetful forgetting. Through 23 × FLOP training with a maximum of 23 ×, SPARCL continues to improve the existing most advanced (SOTA) CL method training efficiency, and it is surprising that the SOTA accuracy is more than 1.7%. SPARCL is also better than the competitive baseline obtained by applying SOTA sparse training methods to CL settings in terms of efficiency and accuracy. We also evaluated the effectiveness of Sparcl on the real mobile phone, further showing the practical potential of our method. The source code will be published.

https://arxiv.org/pdf/2209.09476.pdf

- END -