AI Robotics Developer — Autonomous AI Agent, Personality, Memory & Real-Time Interaction (Jetson Orin NX)

I'm looking for an experienced AI/robotics developer to help complete an autonomous AI-powered robot running on an NVIDIA Jetson Orin NX 16GB.

This is not a project to build a simple voice assistant or chatbot. The goal is to create a robot with a consistent personality, long-term memory, emotional behavior, independent decision-making, and natural interactions, capable of reacting to its environment in real time.

The robot is a character with an established personality, history, and relationship with another humanoid robot. Its behavior should reflect that identity through dialogue, voice, visual expressions, and physical movements.

Current Development Status

The project is already partially implemented.

The following components are installed and operational:

Whisper (STT): Speech recognition is implemented and working.

Qwen (LLM): A local language model is installed and capable of generating conversational responses.

Piper (TTS): Local text-to-speech is implemented.

Basic conversational pipeline: The robot can already recognize speech, generate an answer, and speak it aloud.

The main issue is that the responses are not yet consistently aligned with the robot's intended personality. The current implementation also needs further development to become a truly autonomous, responsive character rather than a basic question-and-answer system.

The goal is to improve and extend the existing system, not unnecessarily rebuild components that already work.

Main Features to Develop

1. Personality and Autonomous Intelligence

Implement a persistent personality system based on an existing character profile.

Maintain consistent personality traits, speaking style, opinions, and emotional reactions.

Develop an internal emotional state that influences speech and physical behavior.

Enable independent reasoning and decision-making.

Allow the robot to decide when to speak, react, initiate conversations, or remain silent.

Generate context-aware humor and sarcasm naturally, without repetitive scripted responses.

Maintain consistent behavior throughout long conversations and across restarts.

2. Long-Term Memory and Character History

Implement persistent short-term and long-term memory.

Import a pre-existing fictional character history containing events that occurred before the robot's first activation.

Differentiate between objective story history and the robot's subjective knowledge and memories.

Remember conversations, events, relationships, and significant experiences.

Retrieve relevant memories without overloading the LLM context.

Support manual memory editing, deletion, and protection.

Implement memory consolidation and a dream-generation system during charging or inactivity, keeping simulated dreams separate from factual memories.

3. Autonomous Conversation and Interruption Handling

Implement natural conversational turn-taking.

Enable the robot to detect and respond to interruptions while speaking.

Develop an interruption-priority system: the robot should not necessarily stop speaking every time someone interrupts.

Allow the robot to pause, resume, continue, or interrupt based on context, emotion, and conversational importance.

Support independent conversational initiative rather than requiring a question or wake word for every interaction.

4. Vision, Hearing, and Attention

Integrate camera-based face, person, and object tracking.

Integrate the ReSpeaker XVF3800 four-microphone array for sound detection and direction.

Develop an attention manager combining visual and audio information.

Enable the robot to automatically orient toward speakers or relevant visual events.

Prioritize competing stimuli and avoid unnecessary attention switching.

Implement fast reactions independently of the LLM.

5. Physical Movement and Emotional Expression

Integrate the Pimoroni Servo 2040 controller.

Support eight servos controlling the quadruped's legs, with additional servos planned for eye movement.

Implement basic locomotion, including standing, sitting, walking, and turning.

Integrate an animated eye and physical mechanical iris for expressive behavior.

Synchronize eye, iris, and body movements with speech, emotions, and attention.

Ensure smooth movements, safe servo limits, and appropriate failure recovery.

6. Voice and Low-Latency Optimization

Optimize the existing STT → LLM → TTS pipeline.

Reduce the delay between the end of the user's speech and the robot's spoken response.

Explore streaming LLM output and streaming TTS.

Improve natural speech delivery, emotion, timing, and expressiveness.

Maintain responsiveness while vision, audio, reasoning, and motor control operate simultaneously.

7. Cloud AI Integration and Comparison

In addition to the fully local system, I want to implement an optional cloud-based AI pipeline for comparison.

Requirements:

Integrate a cloud LLM as an alternative to the local Qwen model.

Evaluate cloud-based TTS as an alternative to the local voice system.

Allow switching between local and cloud processing.

Maintain the same personality, character memory, and behavior logic regardless of which backend is used.

Compare response latency, conversational quality, personality consistency, voice quality, resource usage, and operating costs.

Benchmark both solutions under realistic conditions.

The cloud system is intended for testing and comparison. The robot must remain fully functional locally without internet access.

8. Architecture, Reliability, and Testing

Work with or extend the existing ROS 2 architecture.

Keep hardware reflexes separate from slower AI reasoning.

Ensure the language model generates high-level behavioral intentions rather than directly controlling servos.

Implement appropriate watchdogs, safety limits, and failure handling.

Maintain a modular and maintainable codebase.

Measure system performance, including STT latency, LLM first-token latency, TTS startup latency, and total conversational response time.

Provide documentation, source code, and reproducible performance benchmarks.

Hardware

NVIDIA Jetson Orin NX 16GB

ReSpeaker XVF3800 four-microphone array

Pimoroni Servo 2040 controller

Eight leg servos, with additional eye/iris actuation

Camera

LCD eye display and mechanical iris

Speaker/audio output

Required Experience

I'm looking for someone with practical experience in:

Python and Linux development on NVIDIA Jetson devices

Local LLM inference and optimization

AI agents, autonomous behavior, and persistent memory architectures

Whisper, Qwen, Piper, or comparable speech/AI technologies

ROS 2 and robotics integration

Real-time audio processing and conversational AI

Computer vision and servo control

Cloud LLM/TTS APIs and latency benchmarking

Experience creating expressive robotic characters or socially interactive robots would be particularly valuable.

Expected Outcome

The finished system should behave like an independent, intelligent robotic character, not a conventional AI assistant.

It should be capable of listening, observing, reasoning, remembering, developing opinions, expressing emotions, initiating interactions, and coordinating speech with physical behavior.

Personality consistency, natural interaction, and low latency are the most important priorities.

Please include examples of relevant work, especially projects involving local AI agents, robotics, persistent memory, or real-time conversational systems.

I would also appreciate a brief explanation of how you would approach improving the existing system rather than rebuilding it from scratch.

---
來源：Freelancer.com | 出價次數：3 | 剩餘時間：6 日 | 專案類型：固定預算