Data Science

A Machine for Living In

Sensing Strategy:

  • Physical Sensors
    • vibration, temperature, humidity, beam break, physical switch, Passive Infra Red (PIR)
  • Audio Nodes
    • ICS43432 MEMS microphone interface via I2S
    • recordingĀ  24bit 48kHz audio
    • running on-device voice activity detection written in python (webrtcvad)
  • Video nodes
    • chip camera
    • on-device motion detection using motion field thresholding fromĀ  h264 encoder block in the SOC.
    • v2: on-device image analysis with movidius VPU
    • FLIR camera


  • Audio Analysis:
    • feature extraction (12 points)
      • 4 Mel-Frequency Cepstral Coefficients
      • Spectral Centroid
      • 6-point Chromagram
      • Fast Fourier Transform crest
    • clustering
      • k-Means
      • HDBSCAN
  • Video Analysis:
    • background removal and motion path estimation (knn, MOG, HOG with opencv)
    • v2: unwarping and Google Image Net classifier
  • Sensor Analysis:
    • time distribution of sensor activations over the day