Concept -> Final Model... Easy, right?

The concept centered around a futuristic neon colored headset that would use lights to expess the unseen thoughts of the user.
The block diagram detailed what would be used for the input - 2 IR reflectance sensors and a 9dof accelerometer - and how those would translate to the output of the neopixels. It also the original plan for the microcontroller and power source.

So... how can we track eye movement?

I had the idea to use 2 carefully placed IR reflectance sensors to track the pupil on each side of the eyes but it took testing to make sure they could work how I wanted and that this could really be used to track eye movement without being thrown off by blinks.

Hardware Debugging Methods

The first attempt of moving to a soldered breadboard was unsuccessful after repeated testing. Here are some of the methods IĀ used to troubleshoot the exact problem:

  • Used multimeter to test that soldered wires are connected
  • Used multimeter to test that breadboard had no faulty row connections
  • Desoldered and re-soldered connection points
  • Referenced resources and the sensors' documentation
  • Changed the order of the resistor in the sequence of the [collector pin - A0 connected wire - 5v power routed through 10k pullup resistor]
    • Collector - A0 - resistor - 5v
    • Collector - resistor - 5v||A0 (parallel)
    • Collector - A0||resistor - 5v
    • Collector - resistor - A0 - 5v
  • Tested IR sensor on breadboard with other microcontroller
  • Tested IR sensor on breadboard with same microcontroller
  • Tested IR sensor on breadboard with soldered connection points (resistors on bb)
  • Replaced the resistors soldered on the breadboard
  • Looked on the internet for other projects that had similar issues
  • Tried solid core wires
  • Asked for help
I was ultimately unable to repair the hardware on the original breadboard and fixed this problem by restarting fresh with a more detailed construction plan. After the Circuit was fixed I was then able to integrate the electronics into a rough model and really start testing the thresholds of when to move between expressive states and how best to express the users thoughts.

Expressive Animation States

State 1: Focused


Initialized when the glasses are lowered onto the face, this state communicates the laser focus of a user that isn't looking around or distracted. This state will also be triggered when little to no head or eye movement is detected and the user is looking straight forward at their work.

State 2: Thinking


Triggered when the eyes move frequently, this animated light sequence illustrates the moving thoughts of the user. This is frequently triggered when the user is reading and their eyes are quickly scanning back and forth.

State 3: Distracted


When the head is detected to be frequently moving side to side the distracted state is triggered. This state has lights randomly blinking on and off to illustrate the scattered thoughts of someone who may have turned to talk to a friend or is shaking their head in confusion.

Final Code

// neopixel timer control to take out delay referenced https://learn.adafruit.com/multi-tasking-the-arduino-part-3/some-common-bits
// Arduino_LSM6DSOX simple accelerometer code used and edited
// used and edited fastLED library sample code https://fastled.io (glitter and juggle functions from DemoReel100 sample code)
// used and edited adafruit neopixel library sample code https://learn.adafruit.com/adafruit-neopixel-uberguide/arduino-library-use

//include accel, neopixel, and fastLED libraries
#include <Arduino_LSM6DSOX.h>
#include <Adafruit_NeoPixel.h>
#include "FastLED.h"

//neopixels initialization and parameter setting
#define DATA_PIN 13                                                   //which digital pin is controlling the Neopixel strip
#define NUMPIXELS 33                                                  //defining the number of pixels on the strip
Adafruit_NeoPixel pixels(NUMPIXELS, DATA_PIN, NEO_GRB + NEO_KHZ800);  //initializing array of pixels for adafruit library

//fast LED neopixel setup
CRGB leds[NUMPIXELS];    //initializing array of pixels for fastLED library
#define LED_TYPE WS2811  //define type of Neopixel strip
#define COLOR_ORDER GRB  //define the order of the color setting
uint8_t gHue = 0;        // rotating "base color" used by many of the patterns in fastLED sample code

//event detection eyes
bool right;    //boolean if the pupil is detected on the right side
bool left;     //boolean if the pupil is detected on the left side
bool forward;  //boolean if the pupil is detected on neither side
bool lastForward;
bool lastLeft;
bool lastRight;

//eye move counters
int moveCount;      //track how many times the eye moves
int headMoveCount;  //track how many times the head moves side to side

//time tracking for movement periods
long now;
long lastTimePeriod;

//head event detection
bool headDown;  // result from reading z acceleration value of the 9dof to read tilt of the glasses
bool headRight;
bool headLeft;

bool lastHeadDown;
bool lastHeadRight;
bool lastHeadLeft;

float x, y, z;  //variable to store acceleration read data

float initialForwardX;  //read forward direction of Y when initialized and glasses put down

//event detection for animation start/stopping
bool focusedAnimation;
bool lastFocused;
bool distractedAnimation;
bool lastDistracted;

void setup() {
  Serial.begin(9600);  // initialize serial communication at 9600 bits per second:
  pixels.begin();      // INITIALIZE NeoPixel pixels object

  if (!IMU.begin()) {  //check sensor reading from 9dof
    Serial.println("Failed to initialize IMU!");
    while (1)
      ;
  }

  //fastLED setup
  FastLED.addLeds<LED_TYPE, DATA_PIN, COLOR_ORDER>(leds, NUMPIXELS);
}

void loop() {

  now = millis();  //get current loop time

  if (IMU.accelerationAvailable()) {  //read acceleration
    IMU.readAcceleration(x, y, z);
    //adjust values for readability
    x *= 100;
    y *= 100;
    z *= 100;
  }

  int rightSensorValue = analogRead(A1);  // read the input on analog pin 0 for the right side
  int leftSensorValue = analogRead(A0);   // read the input on analog pin 1 for the left side

  headDown = (z < 10);     //read if the head/glasses is down or not (headDown true if z < 10), a 0 reading is straight ahead and 10 is slightly up, a wider range prevents accidentally turning device on and off during use
  headRight = (x <= -20);  //check if the head is tilted right
  headLeft = (x > 20);     //check is the head is tilted left

  if (headDown) {                       //if the head is down (system is on)
    right = (rightSensorValue >= 960);  //evaluate if the right sensor is being looked at (higher value assosiated with less reflectance/darker colors (like the pupil of the eye))
    left = (leftSensorValue >= 850);    //evaluate if the left sensor is being looked at
    //these values must be calibrated for the distance of the eye from the sensor (where placed in glasses)
    forward = (right && left);  //looking ahead (not registering left or right pupils)

    //check and count if eyes have moved
    if (forward != lastForward) {
      moveCount += 1;
    } else if (left != lastLeft) {
      moveCount += 1;
    } else if (right != lastRight) {
      moveCount += 1;
    }

    //check and count head movements
    if (headRight != lastHeadRight) {
      headMoveCount += 1;
    } else if (headLeft != lastHeadLeft) {
      headMoveCount += 1;
    }

    if (headDown && !lastHeadDown) {   //if the head is down and was just not down
      initializing();                  //intializing animation
      focusedAnimation = false;                   //record that it is not currently focused
      distractedAnimation = false;                //record that it is not currently distracted
    } else if (headMoveCount >= 10) {  //head moving a lot side to side (count must be calibrated to time period to not accidentally trigger)
      if (!lastDistracted) {             //if distracted just started clear strip to prepare for the animation
        pixels.clear();
        pixels.show();
      }
      distracted();  //distracted animation
      focusedAnimation = false;
      distractedAnimation = true;
    } else if (moveCount >= 15) {  //have the eyes moved more than 5 times in 5 seconds
      thinking();                  //thinking animation
      focusedAnimation = false;
      distractedAnimation = false;

    } else {
      focusedAnimation = true;
      if (!lastFocused) {
        if (now - lastTimePeriod >= 1000) {
          focused();  //focused animation
          Serial.println("now playing");
        }
      } else {
        focused();
      }
      distractedAnimation = false;
    }

    if (now - lastTimePeriod >= 7000) {  //are we still in the time period, if not we need to reset
      moveCount = 0;                     //reset eye move count
      headMoveCount = 0;                 //reset head move count
      lastTimePeriod = now;              //reset time marker
      Serial.println("reset time");
    }

  } else {                                 // turn off it head or glasses are up
    for (int i = 0; i < NUMPIXELS; i++) {  // For each pixel...
      pixels.clear();                      // Send the updated pixel colors to the hardware.
      pixels.show();
    }
    focusedAnimation = false;
    distractedAnimation = false;
  }

  //update event tracking variables
  lastForward = forward;
  lastLeft = left;
  lastRight = right;
  lastHeadDown = headDown;
  lastHeadRight = headRight;
  lastHeadLeft = headLeft;
  lastFocused = focusedAnimation;
  lastDistracted = distractedAnimation;
}

//progressively light up pixels from each ear to the center
void initializing() {
  delay(200);                                           // slight delay to give user time to take hands off of device if they are placing
  pixels.clear();                                       //clear any lights left from last animation
  pixels.show();                                        //publish the cleared pixels
  for (int position = 0; position <= 17; position++) {  //loop through half the number of pixels
    for (int i = 0; i <= position; i++) {               //right side (according to wearer) adding one pixel each iteration of larger outside loop, note it is comparing the current pixel to the larger for loop positioning
      pixels.setPixelColor(i, 100, 200, 255);           //set pixel on to bluish/white color
    }
    for (int i = 33; i >= (33 - position); i--) {  //loop through other side, starting at the end and adding one more pixel each time by comparing to position variable
      pixels.setPixelColor(i, 100, 200, 255);      //set pixel on to bluish/white color
    }
    pixels.show();  //publish result (for both sides at the same time)
    delay(50);      //editable delay to change speed of animation
  }
  delay(1000);           //delay to keep light lit for 1 second
  lastTimePeriod = now;  //restart time period
}

//juggled function from demoReel100 in fastLED example library
void thinking() {  // eight colored dots, weaving in and out of sync with each other
  uint8_t dothue = 0;
  for (int i = 0; i < 8; i++) {
    leds[beatsin16(i + 7, 0, NUM_LEDS - 1)] |= CHSV(dothue, 200, 255);
    dothue += 32;
  }
  FastLED.show();  //publish changes to strip
}

//adapted glitter function from demoReel100 in fastLED example library
void distracted() {                           //randomly turns pixels on and off
  if (random8() < 10) {                       // 10/255 chance of occuring
    leds[random16(NUM_LEDS)] += CRGB::White;  //selects random pixel from within the pixel array and turns it on
  }
  if (random8() < 30) {                       // 30/255 chance of occuring
    leds[random16(NUM_LEDS)] -= CRGB::White;  //selects random pixel from within the pixel array and turns it off
  }
  FastLED.show();  //publish changes to neopixel strip
}


void focused() {                                  //light up strip with bluish white light
  for (int i = 0; i < pixels.numPixels(); i++) {  //loop through all pixels in the strip
    pixels.setPixelColor(i, 100, 200, 255);       //sets current pixel within loop to bluish/white color
  }
  pixels.show();  //publishes changes to strip (all at once not one pixel at a time)
}