Building a skill on Jibo

Concepts

What a skill is

A skill is a folder of code that Jibo's behaviour engine, "Be", loads at runtime. It lives under sdk/skills/<name>/ and needs three things: a class that performs the behaviour, an index.html that starts it, and a package.json that declares it to Jibo.

The skill lifecycle

The behaviour class extends BeSkill and implements a few methods. Be calls them at the appropriate times; you do not call them directly. The methods you will use most:

Within these methods, the robot is controlled through the jibo module — for example jibo.face.views.forceEyeView() to show the eye, jibo.expression.setLEDColor([r,g,b]) to set the light ring, and jibo.embodied.speech.speak(text) to speak. Call this.exit() to return control to Be.

Concepts

Prerequisites

Confirm the following before starting. Each is required by a later step.

• A Linux environment (native Linux or WSL 2 on Windows) with git (submodule support), bash, make, unzip, and curl or wget.
• Network access to the package registry at npm.jibo.media.mit.edu:8081 (VPN if required).
• Repository access to the SDK and the skill branch you intend to build.
• A robot with its IP address (for example 192.168.41.201) and root SSH access. Obtain the login from whoever provisioned the robot.

Part 1

Set up the SDK

A one-time setup of the build toolchain. When complete, node --version reports v6.5.0 and yarn build:all succeeds. The SDK dates from 2018, so a few steps work around aging dependencies; each is explained where it appears.

1.1Clone and initialise submodules

git clone <repo-url> sdk
cd sdk
git submodule update --init --recursive

1.2Run setup

Installs Node 6.5.0 and yarn into virtualize-node/node/ using tj/n.

export VIRTUALIZE_NODE_VERSION=6.5.0
bash setup.sh

1.3Activate the virtual environment

Prepends the local node/yarn/npm binaries to PATH. Run unactivate to undo. This must be done in every new shell.

source ./activate
node --version       # expect v6.5.0

1.4Install dependencies

Node 6.5.0's bundled root certificates have expired and cannot validate modern HTTPS endpoints. The first command disables certificate verification for this environment. The second installs dependencies while skipping postinstall scripts, some of which fail (handled below).

export NODE_TLS_REJECT_UNAUTHORIZED=0
yarn install --frozen-lockfile --ignore-scripts

Run the two Electron postinstall scripts that were skipped:

cd node_modules/electron-prebuilt && node install.js && cd ../..
cd node_modules/electron-chromedriver && node download-chromedriver.js && cd ../..

If the Electron downloads fail with SSL errors, fetch the binaries with the system curl, then re-run the two scripts above:

mkdir -p ~/.electron
curl -L -o ~/.electron/electron-v1.4.3-linux-x64.zip \
  https://github.com/electron/electron/releases/download/v1.4.3/electron-v1.4.3-linux-x64.zip
curl -L -o ~/.electron/SHASUMS256.txt-1.4.3 \
  https://github.com/electron/electron/releases/download/v1.4.3/SHASUMS256.txt
curl -L -o ~/.electron/chromedriver-v1.8.0-linux-x64.zip \
  https://github.com/electron/electron/releases/download/v1.8.0/chromedriver-v1.8.0-linux-x64.zip
curl -L -o ~/.electron/SHASUMS256.txt-1.8.0 \
  https://github.com/electron/electron/releases/download/v1.8.0/SHASUMS256.txt

1.5Build

yarn build:all

1.6Make the jibo CLI available

Linking the CLI provides the jibo command used to deploy and control the robot later.

cd packages/jibo-cli && yarn link && cd ../..
export PATH="$HOME/.yarn/bin:$PATH"   # add to ~/.bashrc to persist
jibo --help | head -3

git config core.symlinks true
git ls-files -s | awk '$1=="120000"{print $4}' | while read f; do
  rm -f "$f"; git checkout -- "$f"
done

Part 2

Create a skill

The smallest skill that loads and runs consists of four files and a single build command. The structure below is the standard layout for any skill.

2.1Folder structure

2.2package.json

The jibo block declares the folder as a skill and sets the on-screen name. devDependencies are used at build time; syncDependencies lists the packages copied to the robot alongside the skill.

{
  "name": "@be/myskill",
  "version": "1.0.0",
  "main": "index.js",
  "config": { "standalone": true },
  "jibo": {
    "main": "index.html",
    "type": "asset-pack",
    "display-name": "My Skill"
  },
  "monorailConfig": { "public": false, "output": "index.js" },
  "files": ["assets", "resources", "mims", "index.*"],
  "scripts": {
    "build":     "jibo-dev build",
    "clean":     "jibo-dev clean",
    "watch":     "jibo-dev watch",
    "sync":      "jibo-rsync",
    "addsshkey": "jibo-rsync --init"
  },
  "devDependencies": {
    "@be/be-framework": "^12.0.0",
    "jibo": "^15.0.0",
    "jibo-dev": "^6.0.0",
    "jibo-rsync": "^5.0.0",
    "jibo-anim-db-animations": "19.0.2"
  },
  "syncDependencies": ["@be/be-framework", "jibo", "jibo-anim-db-animations"]
}

2.3index.html

Hosts a #face div and instantiates the skill class.

<!DOCTYPE html>
<html><head>
  <title>Be - My Skill</title>
  <style>
    body { margin:0; overflow:hidden; background:#000; }
    #face { width:1280px; height:720px; }
  </style>
</head><body>
  <div id="face"></div>
  <script>
    const MySkill = require('./index');
    const skill = new MySkill();
  </script>
</body></html>

2.4The skill class

A complete skill: show the eye, speak once, and exit. src/index.ts re-exports the class.

/// <reference path="../typings/index.d.ts" />
import { BeSkill } from '@be/be-framework';
import jibo = require('jibo');

export default class MySkill extends BeSkill {
  constructor(assetPack?) { super(assetPack); }

  public open(result?: any, refresh?: boolean): void {
    jibo.face.views.forceEyeView();
    jibo.embodied.speech.speak('Hello, I am my skill.', {})
      .then(() => this.exit())
      .catch((err) => { this.log.warn(err); this.exit(); });
  }

  public close(done: () => void): void { done(); }
}

// src/index.ts
import MySkill from './MySkill';
module.exports = MySkill;

2.5Build

cd ~/jibo/sdk/skills/myskill
yarn build

Part 3

Add it to the main menu

Two edits: register the skill with Be so it is loaded, then add a tile to the menu so it can be selected.

3.1Register the skill with Be

In skills/be/package.json, add the skill to the jibo.skills list and to dependencies. Be has no syncDependencies array — that field belongs to the individual skill.

"jibo": { "skills": [ ..., "@be/myskill" ] },
"dependencies": { ..., "@be/myskill": "^1.0.0" }

3.2Add a menu tile

Place a PNG icon (≤200×200) in skills/main-menu/resources/icons/, then add an entry to the list in skills/main-menu/resources/views/main-menu-verbal.json. The example below is the "Play with Jibo" tile:

{
  "id": "bilingual",
  "label": "Play with Jibo",
  "colors": ["0x25F2FB", "0x0A6E8A"],
  "iconSrc": "resources/icons/play-with-jibo.png",
  "action": {
    "type": "utterance",
    "data": { "utterance": {
      "intent": "loadMenu",
      "entities": { "destination": "bilingual" }
    } }
  }
}

For another skill, copy the block and change id, label, iconSrc, and destination accordingly.

Part 4

Deploy & run

Copy the skill to the robot and start it. Two steps — the symlink and the permissions fix — are required for the robot to load the skill; skipping either is the common reason a skill copies over but does not take effect.

4.1Register the robot and set up SSH

jibo add myrobot 192.168.41.201
jibo set myrobot && jibo list                # confirm * next to myrobot
cd packages/skills-service-manager && yarn addsshkey && cd -   # installs your SSH key
ssh root@192.168.41.201 "echo KEY_AUTH_WORKS"

4.2Create the load symlink

ssh root@192.168.41.201 "mount -o remount,rw /"
ssh root@192.168.41.201 "rm -rf /opt/jibo/Jibo/Skills/@be/myskill && \
  ln -sf be/node_modules/@be/myskill /opt/jibo/Jibo/Skills/@be/myskill"
ssh root@192.168.41.201 "ls -la /opt/jibo/Jibo/Skills/@be/"   # verify the link
ssh root@192.168.41.201 "mount -o remount,ro /"

4.3Sync the skill

If yarn sync:be works (it runs cd skills/be && yarn sync):

cd ~/jibo/sdk && rm -rf skills/be/.staged && yarn sync:be

If the dev shell times out (ETIMEDOUT on port 8686), copy the files directly with rsync:

ssh root@192.168.41.201 "mount -o remount,rw /"
rsync -av --delete --exclude='node_modules' --exclude='.staged' \
  ~/jibo/sdk/skills/myskill/ \
  root@192.168.41.201:/opt/jibo/Jibo/Skills/@be/be/node_modules/@be/myskill/
ssh root@192.168.41.201 "mount -o remount,ro /"

4.4Fix file permissions (after every sync)

ssh root@192.168.41.201 "mount -o remount,rw /"
ssh root@192.168.41.201 "chown -R 2000:10 …/@be/be/node_modules/@be/myskill && \
  find …/myskill -type f -exec chmod 640 {} \; && \
  find …/myskill -type d -exec chmod 750 {} \;"
ssh root@192.168.41.201 "mount -o remount,ro /"

4.5Choose a boot mode and launch

ssh root@192.168.41.201 "jibo-setmode int-developer && reboot"   # wait ~2 min
cd ~/jibo/sdk/skills/be && jibo run                       # boot to idle / menu

Tap the screen to open the main menu and select the tile. To start a skill by voice or proactively, see the optional features below.

Part 5 · Optional

Voice trigger

A skill can register its own launch grammar so that spoken phrases start it. This is configured with a launch.rule file and one line of metadata. The bilingual skill uses it to respond to phrases such as "let's play".

5.1The launch rule

launch.rule maps phrases to an intent and a priority. $* matches any surrounding words, so the phrase need not be exact:

TopRule =
(
    ( $* let'?s play $* )      {% intent='bilingual' %} {% priority='high' %}
  | ( $* play with jibo $* )   {% intent='bilingual' %} {% priority='high' %}
  | ( $* play with you $* )    {% intent='bilingual' %} {% priority='high' %}
  | ( $* bilingual $* )        {% intent='bilingual' %} {% priority='high' %}
  | ( $* play a story $* )     {% intent='bilingual' %} {% priority='high' %}
);

5.2Register the rule in package.json

Point jibo.launchRule at the file and include it in files so it is deployed:

"jibo": {
  "main": "index.html",
  "type": "asset-pack",
  "launchRule": "launch.rule",
  "display-name": "Play with Jibo"
},
"files": ["assets", "resources", "mims", "index.*", "launch.rule"]

When a phrase matches, Be launches the skill and passes the intent to open(), where the skill can act on it (see Part 6).

Part 6 · Optional

Proactive prompt

A skill can offer to start on its own — for example when the robot detects a person — rather than waiting to be selected. A direct request (voice or menu) skips the offer. The logic is contained in open().

6.1Deciding whether to ask

public open(result?: any, refresh?: boolean): void {
  jibo.face.views.forceEyeView();
  this._options = result || {};

  const intent = this._getLaunchIntent(this._options);   // reads options.nlu.intent etc.
  const skipAsk = intent === 'bilingual'           // started by voice
              || intent === 'menu'                // started from the menu
              || this._options.skipAsk === true;

  if (skipAsk) {
    this._startElevenlabs();                            // start directly
  } else {
    this._askToPlay();                                  // offer first
  }
}

6.2The offer

The prompt is a multi-modal interaction (a MIM) of type question with a yes/no menu — mims/en-us/BilingualSurprise.mim:

The skill starts the MIM in _askToPlay() and branches on the result:

onSuccess: (results) => {
  const intent = results.asrResults && results.asrResults.intent;
  this._teardownMim(() => {
    if (intent === 'yes') { this._startElevenlabs(); }
    else            { this.cleanupViews(() => this.exit()); }
  });
},
onFailure: () => { this.cleanupViews(() => this.exit()); }

6.3On acceptance

• Acknowledge — set the light ring (setLEDColor([0.6, 0, 1])) and speak a short cue while the interaction starts.
• Record state — store lastPlayed = Date.now() in the KB, which can be used to rate-limit the proactive offer (for example, once per day).
• Start — run the skill's behaviour. When it finishes, the skill exits and control returns to Be.

Part 7 · Optional

Example: launching an external service

A skill's behaviour can be self-contained, or it can hand off to a separate program. The "Play with Jibo" skill takes the second approach: it launches an external conversational-AI service and manages its lifecycle.

The skill starts the service with /usr/local/jibo-elevenlabs/run.sh and stops it with kill -TERM $(pgrep -f "jibo-elevenlabs"). Its credentials live in dist/.env (ELEVENLABS_AGENT_ID, ELEVENLABS_API_KEY). The service is built against its own Node environment (Node 6.9.2), separate from the SDK.

Troubleshooting

Quick reference

Troubleshooting

Detailed fixes

git config core.symlinks true
git ls-files -s | awk '$1=="120000"{print $4}' | while read f; do
  rm -f "$f"; git checkout -- "$f"
done
# for CRLF noise: git diff -w <file> — if empty, only line endings differ; git checkout -- <file>

ssh root@192.168.41.201 "mount -o remount,rw /usr/local"
ssh root@192.168.41.201 "cp /usr/local/bin/jibo-ssm/lib/skills-service-manager.js /tmp/ssm.js.bak"
ssh root@192.168.41.201 "sed -i 's|let currentErrorId = ...|let currentErrorId = null;|' \
  /usr/local/bin/jibo-ssm/lib/skills-service-manager.js"
ssh root@192.168.41.201 "mount -o remount,ro /usr/local"
ssh root@192.168.41.201 "/etc/init.d/S78jibo-system-manager stop && sleep 5 && /etc/init.d/S78jibo-system-manager start"

One-line fixes

Reading the robot log

ssh root@192.168.41.201 "tail -300 /var/log/messages" | grep -iE 'bilingual|elevenlabs|mcp|error' | tail -40
ssh root@192.168.41.201 "tail -f /var/log/messages | grep --line-buffered -iE 'error|reboot|skill'"

Reference

Command cheat sheet

New terminal for SDK work

cd ~/jibo/sdk
source ./activate
export PATH="$HOME/.yarn/bin:$PATH"
export NODE_TLS_REJECT_UNAUTHORIZED=0
node --version   # v6.5.0

Build and deploy a skill

# build
cd ~/jibo/sdk/skills/myskill && yarn build && cd ~/jibo/sdk
# sync
ssh root@192.168.41.201 "mount -o remount,rw /"
rsync -av --delete --exclude='node_modules' --exclude='.staged' \
  skills/myskill/ root@192.168.41.201:/opt/jibo/Jibo/Skills/@be/be/node_modules/@be/myskill/
# fix permissions (every time)
ssh root@192.168.41.201 "chown -R 2000:10 …/myskill && \
  find …/myskill -type f -exec chmod 640 {} \; && find …/myskill -type d -exec chmod 750 {} \;"
ssh root@192.168.41.201 "mount -o remount,ro /"
# launch
cd skills/be && jibo run

Robot control

jibo stop / jibo run                       # stop a skill / launch Be
ssh root@192.168.41.201 "reboot"             # reboot the robot
jibo set-volume 0.7 --robot myrobot ; jibo diskspace ; jibo build-version

Reference

Pre-flight checklist

1. Environment activated. node --version is v6.5.0 and which jibo resolves.
2. Symlink present on the robot. ls -la /opt/jibo/Jibo/Skills/@be/ shows the skill pointing to be/node_modules/@be/<skill>.
3. Permissions correct. Ownership 2000 10, modes 640/750.
4. Registered with Be. The skill appears in both jibo.skills and dependencies of skills/be/package.json.
5. Voice trigger (if used). launch.rule is deployed and referenced via jibo.launchRule.
6. Logs checked. /var/log/messages filtered for the skill name and errorId.