Deep Learning Techniques Applied to Predict and Measure Finger Movement in Patients with Multiple Sclerosis

This research focuses on the development of a system for measuring finger joint angles based on camera image and is intended for work within the field of medicine to track the movement and limits of hand mobility in multiple sclerosis. Measuring changes in hand mobility allows the progress of the disease and its treatment process to be monitored. A static RGB camera without depth vision was used in the system developed, with the system receiving only the image from the camera and no other input data. The research focuses on the analysis of each image in the video stream independently of other images from that stream, and 12 measured hand parameters were chosen as follows: 3 joint angles for the index finger, 3 joint angles for the middle finger, 3 joint angles for the ring finger, and 3 joint angles for the pinky finger. Convolutional neural networks were used to analyze the information received from the camera, and the research considers neural networks based on different architectures and their combinations as follows: VGG16, MobileNet, MobileNetV2, InceptionV3, DenseNet, ResNet, and convolutional pose machine. The final neural network used for image analysis was a modernized neural network based on MobileNetV2, which obtained the best mean absolute error value of 4.757 degrees. Additionally, the mean square error was 67.279 and the root mean square error was 8.202 degrees. This neural network analyzed a single image from the camera without using other sensors. For its part, the input image had a resolution of 512 by 512 pixels, and was processed by the neural network in 7–15 ms by GPU Nvidia 2080ti. The resulting neural network developed can measure finger joint angle values for a hand with non-standard parameters and positions.