​
MentraOS ASG Client Development Guidelines

This guide provides an overview and development guidelines for the MentraOS ASG (Android Smart Glasses) Client, which runs on Android-based smart glasses like Mentra Live.

​
Overview

The MentraOS ASG Client is an Android application that runs directly on Android-based smart glasses hardware. It serves as the bridge between the glasses’ hardware capabilities and the MentraOS ecosystem, enabling features like camera capture, display rendering, Bluetooth communication, and network management.

​
Architecture

​
Core Components

  1. AsgClientService - The main service that coordinates all functionality:
    • Parses Bluetooth messages from the MentraOS mobile app
    • Handles command processing from those messages
    • Parses messages from Mentra Live’s microcontroller (button presses, swipes)
    • Triggers actions like taking pictures, videos, and starting RTMP streams
    • Manages network state and battery status reporting
  2. Manager Systems - Interface-based factory pattern for device abstraction:
    • NetworkManager - WiFi and hotspot control
    • BluetoothManager - BLE communication with phone app
    • MediaCaptureService - Photo/video capture and upload
    • RtmpStreamingService - Live video streaming

​
Network Manager System

The NetworkManager uses a factory pattern to support different device types: 
// Interface defining network operations
public interface INetworkManager {
    void setWifiEnabled(boolean enabled);
    void connectToWifi(String ssid, String password);
    void startHotspot(String ssid, String password);
    // ... other network operations
}
Implementations:
  • K900NetworkManager - For Mentra Live (K900) devices using proprietary broadcasts
  • SystemNetworkManager - For devices with system permissions using reflection
  • FallbackNetworkManager - For regular devices, prompts user for manual configuration

​
Bluetooth Manager System

Similarly structured for Bluetooth/BLE operations: 
// Interface for Bluetooth operations
public interface IBluetoothManager {
    void initialize();
    void startAdvertising();
    void sendData(byte[] data);
    boolean isConnected();
    // ... other BLE operations
}
Implementations:
  • K900BluetoothManager - Uses serial port communication with BES2700 chip
  • StandardBluetoothManager - Full BLE GATT server implementation for standard Android devices

​
Media System

The media system handles camera button presses and media capture: Button Press Flow:
  • Physical button press detected by microcontroller
  • MCU sends command to ASG Client (cs_pho for short press, cs_vdo for long press)
  • ASG Client checks button press mode configuration:
    • PHOTO mode: Takes photo/video locally
    • APPS mode: Sends button event to phone/apps
    • BOTH mode: Does both actions
  • Photos are captured and queued for upload when connected

​
RTMP Streaming

Supports live video streaming with four main commands:
  1. start_rtmp_stream - Initiates streaming to specified RTMP URL
  2. stop_rtmp_stream - Terminates active stream
  3. keep_rtmp_stream_alive - Must be sent every 15 seconds to prevent 60-second timeout
  4. get_rtmp_status - Queries current streaming status
The stream automatically stops if no keep-alive is received for 60 seconds. The system uses ACK-based reliability where each keep-alive must be acknowledged by the glasses.

​
Development Guidelines

​
Code Style

  • Java:
    • Use Java SDK 17
    • Classes: PascalCase
    • Methods: camelCase
    • Constants: UPPER_SNAKE_CASE
    • Member variables: mCamelCase (with m prefix)
    • 2-space indentation
    • EventBus for component communication

​
Adding Support for New Glasses

To add support for new Android-based smart glasses:
  1. Fix Device Detection (REQUIRED): 
    // In NetworkManagerFactory.java and BluetoothManagerFactory.java
// Change this:
if (true || isK900Device()) {  // Currently forced to K900

// To this:
if (isK900Device()) {  // Proper device detection
  2. Create a SmartGlassesCommunicator in augmentos_core:
    • Extend SmartGlassesCommunicator base class
    • Implement BLE client to connect to StandardBluetoothManager
    • Handle display, audio, and sensor capabilities
    • See ANDROID_FIRMWARE_GUIDE.md for detailed instructions
  3. Update Device Detection: 
    private boolean isYourGlassesDevice() {
    String model = Build.MODEL.toLowerCase();
    return model.contains("your-device-identifier");
}
  4. Configure Manager Selection:
    • StandardBluetoothManager will work for most devices
    • Choose appropriate NetworkManager based on permissions
    • Test with both System and Fallback network managers

​
Important Notes on Current State

StandardBluetoothManager: A fully implemented BLE GATT server exists with:
  • Complete BLE GATT server functionality
  • Auto-pairing capabilities
  • MTU negotiation
  • Serial-like data exchange over characteristics
However, it’s not currently used because:
  • The K900 (Mentra Live) uses proprietary serial communication via K900BluetoothManager
  • The MentraLiveSGC.java in augmentos_core is designed specifically for K900’s serial protocol
  • To use StandardBluetoothManager, you’d need a new SmartGlassesCommunicator that acts as a BLE client

​
Building and Testing

  1. Environment Setup: 
    # Copy environment file
cp .env.example .env

# For local development, modify .env:
MENTRAOS_HOST=localhost  # or your computer's IP
MENTRAOS_PORT=8002
MENTRAOS_SECURE=false
  2. Build Requirements:
    • Android Studio with Java SDK 17
    • Set Gradle JDK to version 17 in Android Studio settings
  3. Testing on Mentra Live Glasses: Mentra Live only supports ADB over WiFi (no USB ADB). To connect: a. Setup WiFi ADB Connection: 
    # 1. Pair Mentra Live with the MentraOS mobile app
# 2. In the app, connect glasses to your WiFi network
# 3. Note the IP address shown on the "Glasses" screen
# 4. Connect via ADB (computer must be on same WiFi network):
adb connect [IP_ADDRESS]:5555

# Example:
adb connect 192.168.1.123:5555

# Verify connection:
adb devices
# Should show: 192.168.1.123:5555    device
    b. Build and Install: 
    # Build the APK
./gradlew assembleDebug

# Install on glasses
adb install app/build/outputs/apk/debug/app-debug.apk

# For local development server, forward ports:
adb reverse tcp:8002 tcp:8002

​
Message Processing

The client processes JSON commands from the cloud: 
// Example command processing in AsgClientService
switch (type) {
    case "take_photo":
        String requestId = dataToProcess.optString("requestId", "");
        mMediaCaptureService.takePhotoAndUpload(photoFilePath, requestId);
        break;

    case "start_rtmp_stream":
        String streamId = dataToProcess.optString("streamId", "");
        String rtmpUrl = dataToProcess.optString("rtmpUrl", "");
        RtmpStreamingService.startStreaming(this, rtmpUrl);
        RtmpStreamingService.startStreamTimeout(streamId);
        break;

    case "display_text":
        // For glasses with displays
        showTextOnDisplay(text);
        break;
}

​
Compatible Devices

Currently supported:
  • Mentra Live (K900)
The ASG Client could potentially be adapted to work on other Android-based smart glasses with sufficient modifications, such as TCL Rayneo X2/X3, INMO Air 2/3, and others.

​
Next Steps for Contributors

  1. Fix the K900 detection in both factory classes
  2. Create device-specific SmartGlassesCommunicators in augmentos_core
  3. Add device detection for your specific glasses model
  4. Test the StandardBluetoothManager with your device
  5. Submit PRs with your device support additions

​
Resources