Featured Posts

DWD | Commenter-Controlled, FB Livestreamed Performance

March 8, 2018

This project is also a work-in-progress for Project Development Studio class. Initial concept can be found here.

 

As of now, the project does not have a title yet, but it is literally a program to turn comments on Facebook livestream into serial data for physical actuation in a performance. Full codes can be found here & here.

 

The program has several parts:

  1. Facebook Login

  2. Pulling & storing comments from a Facebook livestream using Graph API

  3. Displaying stored comments

  4. Parsing comments from database and turn it into serial data

  5. Using serial data to control mechanical arm

Part 1. Facebook Login

 

The first step to pull comments from a user's livestream is to create an "app" on developers.facebook.com. I created an app called "livestream_show", and put in the IP address that I have on Digital Ocean.

 

 

I used a service called xip.io and appended "xip.io" to my server's IP address because (1) Facebook only allows domain names and (2) I don't have a domain name yet when I first developed the program.

 

The next step is to write a web app to call FB's Graph API.

 

Previously I used an example that I found on Graph API, but the API call happens on client-side and the access token is left exposed (code can be found here). So I rewrote the app and used a nodeJS library from criso called "fbgraph" instead, so that the call can happen on the server-side.

 

Once user clicks on LOGIN button, the nodeJS server will call authorization function for Graph API and prompt the user to login. This authorization function requires redirectURL, app ID, and app secret that I got from FB developer page. I also had to define the "scope" I need from the FB user. For this project, I require access to public_profile, user_videos, user_posts.

 

// OAuth2 Implementation - Redirect to FB login
app.get('/login', function (req, res) {
    console.log("redirecting to: " + authUrl);
    res.redirect(authUrl);
});

 

// FB redirected here after successful login
app.get('/loggedin', function (req, res) {
    // Access Code from Facebook
    console.log("Access Code: " + req.query.code);
    
    // Now "hit" Facebook again with "code" to get "Access Token"
    // Using the graph.authorize function to do this
    graph.authorize({
        "client_id":      fb_appID
      , "redirect_uri":   loginRedirectUrl
      , "client_secret":  fb_secret
      , "code":           req.query.code
    }, function (err, facebookRes) {

        if (err) console.log(err);
        
        // Got the access token            
        console.log("Access Token: " + facebookRes.access_token);
        graph.setAccessToken(facebookRes.access_token);
        accessToken= facebookRes.access_token;


        graph.get('/me/live_videos', function(err, liveVids) { 
            res.render('vidlist.ejs', {liveVids:liveVids});            
        });        
    });    
});

 

(As of now, only my Facebook account is allowed to use the "app" has not been submitted for review on Facebook)

 

Part 2. Pulling and storing comments

 

Once the login and authorization is successful, the server renders the list of live videos onto an EJS template.

 

If there is a live video, the user can trigger the comment tracking. Otherwise, they can see an archive of live video comments, if these live videos have been tracked previously.

 

<h3>LIVE</h3>
    <% for (var i = 0; i < liveVids.data.length; i++) { %>
        <% if (liveVids.data[i].status == 'LIVE') { %>
          start tracking video: <a href="/track?liveVideoID=<%= liveVids.data[i].id %>">
          <%= liveVids.data[i].id%> </a><br />
        <% } %>
    <% } %>
    <h3>ARCHIVED</h3>
    <% for (var i = 0; i < liveVids.data.length; i++) { %>
        <% if (liveVids.data[i].status != 'LIVE') { %>
          <a href="/find?liveVideoID=<%= liveVids.data[i].id %>"> video :
          <%= liveVids.data[i].id%> </a><br />
        <% } %>
    <% } %>

 

 

 

 

Clicking on a live video link will trigger the comment tracking. The comments will be pulled every half a second and stored in a mongo database in mLab. One important thing that I learned the hard way is that FB only returns 25 comments in one call by default. I did not know how to use paging in the API call yet, so I set the parameter "limit" to 500.

 

var id = req.query.liveVideoID;
var getURL = id+'/comments?order=reverse_chronological&limit=500';

 

function getLiveComments(url, id){
    commentInterval = setInterval(function(){
        graph.setAccessToken(accessToken);
        graph.get(url, function(err, resp) { 

 

            for (var i=0; i<resp.data.length; i++){
                var commentID = resp.data[i].id;
                var commentMessage = resp.data[i].message;
                var culpritName = resp.data[i].from.name;
                var culpritID = resp.data[i].from.id;

 

                db.submissions.save({
                "_id":commentID,
                "liveVideoID": id,
                "commentMessage": commentMessage,
                "culpritName" : culpritName,
                "culpritID": culpritID
                }, function(err, saved) {
                    if( err || !saved ) console.log("Not saved");
                    //else console.log("comment saved...iteration: "+count);
                });
            }
            
        });
        count++;
        console.log("comment saved...iteration: "+count);
    },500);

}

Part 3. Displaying Stored Comments

 

 

 

On the website, user can also see the comment list in reverse chronological order (newest first). Since the server stores new comments every half-second, I have an AJAX call to the server that updates the comment list on the comment page. It checks if the comment ID has already been loaded previously, before loading it onto the page.

 

var loadedCommentID = [];

setInterval(loadComments, 1000);
loadComments();

 

 function loadComments(){
        $(document).ready(function(){
          $.ajax({
            url: "/updatefind?liveVideoID=<%=data[0].liveVideoID%>",
            dataType: 'json',
            success: function(data) {
              //console.log(data);
              for (var i=0; i<data.length; i++){
                //console.log("index: "+i);
                var isAlreadySaved = false;
                var currentCommentID = data[i]._id;
                //console.log(i+" | "+data[i]._id);

                if(loadedCommentID){
                  for (var j=0; j<loadedCommentID.length; j++){
                    if (loadedCommentID[j] == currentCommentID){
                      isAlreadySaved = true;
                      break;
                    }
                  }
                }

                if(!isAlreadySaved){
                  loadedCommentID.push(currentCommentID);
                  $("#comments").prepend("<p><b>"+data[i].commentMessage+"</b><br/>"+data[i].culpritName+"</p>");
                }
              }
            },
            error: function() {
              alert("tracking has stopped!");
            }
          });        
        });
      }

Part 4. Parsing stored comments into serial data

 

I wrote a nodeJS program on my machine so that I can turn the stored comments into serial data that get sent to an Arduino. I used a package called "serialport".

 

How the program parses the comments depends on the physical part of the performance. In this case, I used a mechanical arm that can move in 4 directions, so the program only parses comments that have the words "up", "down", "left", and "right". These strings are turned into angle values for the servo to move.

 

 

 

Since the server stores new comments every half-second, this program also calls into the database every half-second. One short coming in the algorithm is that it does not have paging, so every call I have to load all the comments stored, and only parse those that have not been parsed before by way of comparing IDs.

 

Source code can be found here.

Part 5. Using serial data to control mechanical arm

 

This part is fairly straightforward (also, it's outside of the scope of DWD class). I'll just post the code here:

 

if (Serial.available() > 0) { // if there's serial data available
    input_for = Serial.parseInt();
    input_wipe = Serial.parseInt();
    Serial.print("input_for: ");
    Serial.print(input_for);
    Serial.print(" | ");
    Serial.print("input_wipe: ");
    Serial.println(input_wipe);

  } else {
    input_for = 0;
    input_wipe = 0;
  }

  angle_for += (input_for*2);
  angle_for = constrain(angle_for,min_for,max_for);
  
  angle_wipe += (input_wipe*2);
  angle_wipe = constrain(angle_wipe,min_wipe,max_wipe);
  

  servoMotor_for.write(angle_for);
  servoMotor_wipe.write(angle_wipe);
  delay(50);

 

Things to improve on, & possible future works:

 

  1. Use session to store access token and make it so that multiple user can use the program.

  2. Web socket for a more stable stream of user input & physical affect(?)

  3. There is an inherent 10-sec delay in FB livestream, so the association between comment and physical affect may not be obvious.

  4. Live video mixing on processing. The live mixing could be affected by user comments as well.

 

Share on Facebook
Share on Twitter
Please reload

Related Posts
Please reload

Generative Music | Latent Space Sequencer

December 13, 2018

1/10
Please reload

Recent Posts
Please reload

Search By Tags

Hafiyyandi | Creative Technologist

New York, New York | hafiyyandi@gmail.com