Neurobehavioral classifier module

A set of algorithms implemented as an extension module to analyse the emotional statements of the users with two main aims:

• Determine an emotional score between an ego and an alter, giving this info to feed the trust model.
• Analyse the content shared creating metrics to give feedback on how the users response to a specific stimulus.

Module with the following functionalities:

• Interface to communicate with Neurobehavioral classifier module
• Metrics of accelerometer sensor for each event in chat activity
• Sentiment analysis of each image sent or received through Helios chat
• Sentiment analysis of Helios chat texts
• SQLite database to store on the device results of sentiment analysis
• egoAlterTrust method to get results of sentiment analysis of an Ego - Alter relationship through NeurobehaviourListener

HELIOS Neuro-behavioral classifier module is one of the HELIOS Module APIs as highlighted in the picture below:

Cloning repository

git clone https://github.com/helios-h2020/h.extension-NeuroBehaviouralClassifier.git

Neurobehaviour module

Current version of Neurobehaviour module works with TestClient app (app + modules) with the purpose of integrate callings to Neurobehaviour module when the user interacts with Helios chat.

Please follow the next steps to integrate Neurobehaviour module into your app.

Integration of the module in the app

Open your app in Android Studio.

File > New > Import Module

Go to neurobehaviouralclassifier folder. Select neurobehaviour folder like source directory. Click Finish to import.

New module should appears in first line of settings.gradle file:

Open build.gradle file of app module. In dependencies section add:

In order to test all features of Neurobehavioral module, you will need to modify some files in TestClient app folder. Please follow these steps.

Integration of the module using Nexus Helios repository

Add the repository in build.gradle file of project:

allprojects {
    repositories {
        google()
        jcenter()
        maven {
            url "https://builder.helios-social.eu/repository/helios-repository/"
            credentials {
                username = heliosUser
                password = heliosPassword
            }
        }

    }
}

Add dependency to the build.gradle file of the app:

dependencies {
	implementation 'eu.h2020.helios_social.modules.neurobehaviour:neurobehaviour:1.1.9'
}

Creation of the AAR library

It's possible to create an AAR file to use the module like a library for any project.

The build.gradle file of Neurobehaviour module includes a line to set the build configuration:

abiFilters "arm64-v8a", "x86"

This setting allows to generate a build for two architectures, PC (Android emulator) and Android mobile.

In order to limit size of AAR file, we can use only "arm64-v8a" value for a production environment. In other hand, we can use "x86" value to test AAR library with Android Studio emulator.

Neurobehaviour module inputs

Helios app should activate Neurobehavioral module with this events and its correspondent calling:

We will use the same calling for all type of message (text / audio / image)

• App starts -> startAccel function to start accelerometer measurement
• App onDestroy method -> stopAccel function

• Settings -> Nick name changed -> createCsv method to create files for analysis results

• User sends a message -> sendingMsg function to start sentimental analysis of message

• Updating Ego - Alter Trust value -> calling to egoAlterTrust function

Neurobehaviour module outputs

Ego - Alter sentimental analysis

Neurobehaviour module > Trust Manager

Neurobehaviour module extracts metrics from user behaviour:

• Arousal: positive / negative
• Valence: positive / negative
• Attention: low / medium / high

Calling to egoAlterTrust function

When Trust module make the call, Neurobehaviour module sends this metrics to calculate the new Trust value.

Sensors analysis

After user has sent or read a message, Neurobehaviour module sends sensor metrics (acceleration average) using SensorValueListener interface (Context module)

DataHandler.java

//Using Context module interface to send acceleration value import eu.h2020.helios_social.core.sensor.ext.DeviceSensor;

obj.msgId = msgId; obj.accelAverage = averageAccel; deviceSensor.receiveValue(obj);

Using the module

AndroidManifest permissions

In order to use sensors of Android device, we have to add permissions in AndroidManifest.xml file of the app:

    <!-- UPV LAB - Using camera to send photos -->
    <uses-permission android:name="android.permission.CAMERA" />
    <uses-permission android:name="android.permission.FLASHLIGHT" />
    <!-- Only for Google Play: -->
    <uses-feature android:name="android.hardware.camera" android:required="false" />
    <uses-feature android:name="android.hardware.camera.flash" android:required="false" />
    <!-- UPV LAB - Using audio recording to send voice messages -->
    <uses-permission android:name="android.permission.RECORD_AUDIO" />

    <!-- UPV LAB · Provider to manage files · Used by the camera -->
    <provider
            android:name="androidx.core.content.FileProvider"
            android:authorities="eu.h2020.helios_social.fileprovider"
            android:exported="false"
            android:grantUriPermissions="true">
            <meta-data
                android:name="android.support.FILE_PROVIDER_PATHS"
                android:resource="@xml/file_paths" />
    </provider>

Calls to Neurobehaviour module methods

How to call module functions from other modules through methods of the listener:

• Contents analysis: Text, image and speech analysis
• Ego - Alter analysis

Module structure - Java classes

Module view in Android Studio

• Neurobehaviour Interface Class
• Neurobehaviour module listener Class
• DataHandler Class
• Acceleration Class
• Acceleration average Class
• Sentiment analysis Class

Sentiment analysis class

Image analysis

This class performs a sentimental analysis of each image sent by user using Helios chat. Using deep learning models, this module extracts:

• Number of faces
• Sentiment of each face: happy, fear, anger...
• Score of each face sentimental analysis

Results are saved in a text file and are drawn over the image:

Text analysis

This class performs too a sentimental analysis of each text sent by user in Helios chat. Using deep learning models, this function extracts:

• Text translation from any language to english using Google API
• Tags for each word of text
• Result of sentiment analysis of text. Positive / negative and score

public class SentimentalAnalysis extends AppCompatActivity {
   public void runThread(final Context context, final String fileName, final HeliosMessageListener messageListener, final HeliosTopic topic, final HeliosMessage message)
   public int realImageWidth(String resource)
   private String getExtension(String word)
   private void textAnalysis (Context context, String message, HeliosMessageListener messageListener, HeliosTopic topic)
   private void pythonAnalysis (Context context, String script, String picture, Float scale, ImageView imageView, HeliosMessageListener messageListener, HeliosTopic topic)
   private void paintingImage(PyObject squares, PyObject emotions, PyObject scores, String imageResource, Float scale, ImageView imageView, Context context, HeliosMessageListener messageListener, HeliosTopic topic)
   public String getDate(long time)
   private void saveImageData (Context context, int numFaces, String emotionsData, String scoreData)
   private void saveTextData (Context context, PyObject origText, PyObject engText, PyObject tags, PyObject emotions) 
   private String extractList(List<PyObject> list)
   private void saveAudioData (Context context, String audioData)
}

Neurobehaviour module storage

The module implements a local storage system to save the sentiment analysis of each message sent or received through Helios chat. This storage system is implemented using a SQLite database and Room like a data object interface.

Design of Neurobehavioral module database:

SQLite + Room database implementation

Testing the module

Unit tests carried out using JUnit4 framework testing library

Instrumented unit tests for Neurobehaviour module in:

Info about Android instrumented unit tests:

Results of conducted tests

Results of unit tests for NeurobehaviourListener class:

Results of unit tests for Acceleration class:

Running the module

Logcat tags to debug the module

• v/listen > show info about listener callbacks
• v/thread > acceleration measurement values (real time linear acceleration and average)
• v/cv > functions for Computer Vision and results of image sentimental analysis
• v/text > results of text sentimental analysis
• v/audio > logs of sentimental analysis of audio files
• v/storage > it shows info about module storage system operations
• v/lab > functions for sessions of Neurobehavioural module validation