How to prevent people from uploading explicit photos via the chat addon

Adding a chat addon in your app can let you increase your revenue a lot, but it also has a big risk: someone could upload adult pictures and put your app in risk of being banned by GooglePlay.

But there's a way to automatically remove all the chat messages that include adult pictures: using Firebase Cloud Functions. Those are some scripts that you can run on the server side provided by Firebase.

Cloud Functions are made for software developers, but if you follow the steps in the Firebase documentation to set up the environment so that you can deploy your Firebase Functions, you'll be able to publish your own functions.

Basically you need to add a function that will be triggered every time a user of your app uploads a message. That function will analyze the content and send the picture to the Google Cloud Vision API, which will answer with some information about it.

Don't worry about programming the function. Once you install the Firebase CLI in your computer (following the steps in the documentation), you'll just need to put this in the index.js file:

functions = require('firebase-functions');

// The Firebase Admin SDK to access Firestore.const admin = require('firebase-admin');

admin.initializeApp( functions.config().firebase );

const vision = require('@google-cloud/vision');

exports.chatMsg = functions.database.ref('/chatname/{docID}/content').onCreate((snap, context) => {

const content = snap.val();

if( content.startsWith("http") ){

  return (async () => {
      const client = new vision.ImageAnnotatorClient();
      

      const [result] = await client.safeSearchDetection(content);
      detections = result.safeSearchAnnotation;

      if( detections.adult == "LIKELY" || detections.adult == "VERY_LIKELY" || detections.adult == "POSSIBLE" ){
          return snap.ref.parent.remove();
      }
      return;

    })()
  .then(() => {
    return;
  })
  .catch( err => console.log( " An error has occurred " + err ) ); 

}
return;
});

You'll also need to add this dependency in the package.json file:

"@google-cloud/vision": "^2.3.1"