HOMIE

Human-object Centric Video Personalization via Multimodal Intelligent Enhancement

Yiyang Cai^* Nan Chen^* Rongchang Xie Junwen Pan Chunyang Jiang Cheng Chen Wen Zhou Zhenbang Sun Wei Xue Wenhan Luo Yike Guo

Hong Kong University of Science and Technology (HKUST)

Contact: ycaicj@connect.ust.hk

^* Equal contribution

Codes Technical Report Project Page HF Model

Abstract

Human-object-centric video personalization (HOCVP) is a core task within subject-driven video generation. However, existing methods suffer from two key limitations. First, most approaches focusing on inter-subject personalization still struggle to strike a balance between high subject fidelity and accurate interaction patterns between humans and diverse objects, especially when objects represent abstract concepts such as logos. Second, while intra-subject references (e.g., OCR maps, multi-view inputs) are expected to enhance subject fidelity, most existing works lack mechanisms understand such latent correspondence. To address both challenges, we propose HOMIE, an HOCVP framework that tackles both inter- and intra-subject input settings in a unified manner. Compared to previous approaches, HOMIE proposes a better MLLM integration strategy to extract knowledge of reference level relationship without compromising the controllability of text encoders or incurring costly re-alignment. Specifically, we introduce global multimodal guidance within self-attention to better align MLLM-derived semantic features with VAE visual tokens. Furthermore, we propose modality-reference embedding to differentiate tokens from MLLM features and VAE tokens and associate intra-subject reference image tokens. Extensive experiments validate that our method achieves state-of-the-art performance across various HOCVP tasks.

1. Demonstration Part 1: Inter-subject HOCVP of General Human-object Interaction
2. Demonstration Part 2: Inter-subject HOCVP (Logo Personalization with MLLM Reasoning)
3. Demonstration Part 3: Intra-subject HOCVP with Multi-view Consistency
4. Demonstration Part 4: Intra-subject HOCVP with High Text Fidelity Enhanced by OCR Maps
5. Qualitative Comparison with Previous Video Personalization Methods
6. Demonstration of Practical Application

(Some videos may take a few seconds to be completely loaded. Please wait patiently or feel free to reload the webpage if several videos are not shown.)

Demonstration Part 1: Inter-subject HOCVP of General Human-object Interaction

HOMIE effectively handles complex inter-subject HOCVP scenarios, supporting general multi-human-object video personalization with high subject fidelity and strong text alignment. It demonstrates robust performance across diverse object categories and various interaction patterns.

These videos correspond to Fig. 4, 8, 9, and 12 in our submission.

☕

A young man sits relaxed on a lush, soft grassland, playing an acoustic guitar gently. Beside him, a young woman sits comfortably, reading a book. The scene is set in a peaceful natural park with scattered trees, soft warm sunlight filtering through leaves, and a calm, cozy atmosphere.

☕

A young woman and a man sit comfortably on a soft brown leather sofa in a cozy, modern living room. Between them, the woman gently plays with a cute, fluffy cat, petting it softly as the cat relaxes. The background features warm indoor lighting, a clean coffee table, subtle home decor, and soft, calm ambient atmosphere. The scene has realistic textures, smooth natural motion, warm tones, and high-definition details.

☕

A man carrying a pink backpack stands and converses naturally with another man on a winding dirt path in a lush, sun‑dappled forest. Tall green trees with dense foliage surround the pathway, soft sunlight filters through the leaves, casting gentle shadows on the ground.

☕

A middle-to-close shot of an animate rabbit and fox walking in the street together. The rabbit is holding a white handbag in her hand casually.

☕

A young woman is holding an exquisite Japanese-style traditional toy, gently gesturing as she talks and interacts naturally with a man. They are sitting comfortably on a soft fabric sofa in a warm, cozy living room, with soft indoor lighting, subtle wooden furniture, and delicate decorative details in the background. The scene is calm and lifelike, with smooth character movements and realistic daily dynamics.

☕

In a bright, quiet modern library, a woman sits comfortably at a clean wooden desk and reading a book. On the desk beside her, there is a small, fresh green potted plant. Soft natural daylight illuminates the scene, with tall bookshelves and a calm, peaceful study atmosphere in the background. The video features smooth, realistic motion, high deail, and warm, soft lighting.

☕

A woman wearing a stylish black shirt sits comfortably on a soft dark leather sofa, holding a cute cat gently in her arms. Beside the sofa stands a tall, lush green indoor plant that brings natural freshness and vitality to the room. The setting is a cozy, modern living room with warm ambient lighting, clean minimalist decor, and a calm, relaxed atmosphere. The scene features realistic textures, smooth subtle movements, and high-definition visual quality.

☕

A man and a woman are sitting comfortably on a cozy leather sofa in a bright, warm living room, chatting actively and naturally with relaxed body language. Soft ambient light fills the room, with a clean, modern background including a subtle coffee table and gentle indoor decor. The scene features smooth, realistic movements, natural facial expressions, and a relaxed, friendly atmosphere, high detail, realistic texture, cinematic quality, stable video generation.

☕

A woman wearing a kimono is facing the camera and introducing a product of decorative doll figurine. She smiles warmly, holding the product gently as she presents it. The background features a cozy living room with a bright and professional atmosphere.

☕

A woman sits comfortably on a minimalist-style sofa in a bright, cozy living room, savoring a delicious donut. The space features clean lines, soft neutral tones, and gentle natural lighting, with simple decorative elements and a tidy, elegant atmosphere in the background. The scene is calm and realistic, with smooth, natural movements and high-definition details.

☕

A man in a baseball cap holds a protein powder container and introduces it to the camera. He stands in a bright gym with dumbbells, treadmills, and fitness posters in the background.

☕

A man stands in a cozy, well-lit music studio, passionately playing a blue-colored guitar. Soft warm lighting illuminates the scene, with musical instruments, soundproof panels, and audio equipment neatly arranged in the background. The atmosphere is calm and artistic, with smooth, natural body movements, high realism, and soft cinematic texture.

Demonstration Part 2: Inter-subject HOCVP (Logo Personalization with MLLM Reasoning)

HOMIE successfully handles additional inter-subject HOCVP scenarios, where abstract concepts such as logos are incorporated as reference inputs. Leveraging knowledge from MLLMs, HOMIE is able to attach logos to the most semantically relevant objects. Notably, these implicit semantic relationships are not explicitly described in the textual prompt.

These videos correspond to Fig. 4, 10 and 12 in our submission.

☕

An elder man sits at his cluttered wooden desk in a bright, bustling office. He reaches for a disposable coffee cup and takes a slow sip.

☕

A young man is standing in a bright, bustling office. He is holding a disposable coffee cup and taking a slow sip.

☕

A young woman is facing the camera and introducing a chocolate product. She holds the chocolate bar in her hand, smiling as she explains its features. The background includes a white countertop and a few kitchen items, creating a warm and inviting atmosphere.

☕

A man is holding a bucket of French fries while walking in a fast-food restaurant towards the camera.

☕

A young man is standing in a clean and tidy kitchen, introducing a potato chip package. He holds the product in his hand, presenting it to the viewer with a slight smile. The background features open shelves, ceramic dishes, and soft under-cabinet lighting, creating a rustic and inviting atmosphere.

☕

A young man is facing the camera and introducing a box of milk product. He holds the product in his hand, smiling as he explains its features. The background includes a white countertop and a few kitchen items, creating a warm and inviting atmosphere.

☕

A woman is facing the camera and introducing a package of sweet candy. She holds the product, smiling as she presents it. The background features minimalist furniture and decorations, creating a bright and cozy atmosphere.

☕

A man wearing a grey long-sleeved hoodie is walking on a track on the playground.

☕

A woman sits at her cluttered wooden desk in a bright, bustling office. She reaches for a disposable coffee cup and takes a slow, satisfying sip.

Demonstration Part 3: Intra-subject HOCVP with Multi-view Consistency

HOMIE can also handle intra-subject HOCVP tasks given multiple reference images captured from different perspectives of the same object (inlcuding both human and objects). When the object or human undergoes motion (e.g., rotating) that reveals their various aspects, HOMIE is able to render these perspectives consistently and coherently.

These videos correspond to Fig. 5, 11 and 13 in our submission.

☕

A man is sitting on a sofa facing the camera, holding an anime character figure and introducing it, while slowly rotating the figure to show its side; the scene is set in a warm, clean and cozy living room with soft lighting and minimalist home decor, medium close-up, bust shot, focusing on his upper body and the figure.

☕

A man is sitting at a desk facing the camera, holding a robot model and introducing it while slowly rotating the model to show its side. The background is a simple and tidy office with bright soft lighting, clean desks, modern minimalist furnishings and a calm working atmosphere.

☕

A middle shot of a woman sits at an office desk with a calm expression, holding a 3D-printed owl model about 15 centimeters tall. She introduces the model and slowly rotates it to clearly show its front and both sides. The office is equipped with various daily office supplies, desks and chairs, as well as a clean whiteboard, creating a neat and practical working space.

☕

A man sits at a desk with a calm smile, holding a 3D-printed monkey astronaut model around 20 centimeters tall as he introduces it. He slowly spins the model to showcase its front and both sides. The office is designed with modern decor, including sturdy wooden shelving units and well‑placed green plants, making the space bright and natural.

☕

A woman sits on a double sofa in the living room with a warm smile, holding a 3D-printed fox model that is about 10 centimeters tall. She introduces the model gently and slowly rotates it to show its front and both sides clearly. The living room is furnished with modern furniture and decorated in a cozy style, with soft lighting and neat layouts enhancing the warm atmosphere.

☕

A young man sits at a desk in his personal study, holding a blue-colored robot model. He explains the model while slowly rotating it to present its front and both sides. The room is styled as a youthful workshop, with well-organized bookshelves and artistic posters hanging on the wall, full of personality and energy.

☕

A lady in ethnic folk white top and red pants stands facing the lens and smiling softly. She has a cute dog-patterned backpack on her back. Moments later, she turns around to show her back and present the backpack completely. The scene is set on an open lawn with a quaint wooden house in the quiet natural background.

☕

A young man in casual top and sunglasses stands with his back to the camera, then quickly turns around to face the lens. He holds a brown teddy bear doll in his hands and begins introducing the doll. The background is a cozy, minimalist living room with warm ambient lighting and simple home decor.

☕

A man in a dark blue suit walks from the right side of the frame to the center, then stops and faces the camera. He holds a brown rabbit plush toy in his hands and starts introducing the toy. The background is a warm, cozy and minimalist living room with elegant simple furnishings and soft ambient light.

☕

A woman is facing the camera, holding a radio with both hands and introducing it while showing its different sides; the background is a clean and tidy office with bright lighting, desks, office supplies and a neat modern working environment, and the shot is a medium close-up focusing on her upper body to highlight the object in her hands.

☕

A man is sitting on a sofa facing the camera, holding an ancient‑style clay cup and taking a sip from it, with a medium close‑up and bust shot focusing on his upper body and the cup; the scene is set in a warm and cozy living room with soft lighting, simple and elegant furniture, and clean and tidy surroundings, creating a quiet and comfortable home atmosphere.

☕

A man walks on a 19th-century-style train station platform facing the camera, carrying a vintage antique suitcase in his hand; the background features classic retro station architecture, old-fashioned signs, warm soft lighting, and a nostalgic historical atmosphere.

Demonstration Part 4: Intra-subject HOCVP with High Text Fidelity Enhanced by OCR Maps

HOMIE effectively addresses intra-subject HOCVP tasks in which reference images with rich textual regions are paired with their corresponding OCR maps. The generation results produced by HOMIE exhibit high textual fidelity, demonstrating its great potential for real-world practical applications.

These videos correspond to Fig. 5, 11 and 13 in our submission.

☕

A woman is facing the camera, holding a baby-care soap, and introducing it. She smiles gently and explains its features. The background features a bathroom with a clawfoot tub, vintage mirror and woven basket, charming and tranquil.

☕

A man is facing the camera and introducing a chocolate bar. He smiles warmly, holding the product. The background features a kitchen with rustic wooden cabinets, a brick backsplash, and cast-iron cookware, cozy and vintage.

☕

A man is facing the camera and introducing a tube of tire repair adhesive. He holds the product, presenting it to the viewer with a calm and professional demeanor. The background features a slim desk, a notebook and a small bookshelf, creating a tidy and efficient atmosphere.

☕

A man is facing the camera and introducing a chocolate bar product. He holds the product with his hand, presenting it to the viewer eagerly. The background features a living room with a gray fabric sofa, striped rug and built-in fireplace, warm and welcoming.

☕

A young woman is facing the camera and introducing a product of instant noodles. She holds the product and prepares to enjoy it with chopsticks. The background features open shelves, glass jars and potted chives, creating a down-to-earth and inviting atmosphere.

☕

A muscular man is facing the camera and introducing a black container of protein powder eagerly. He holds the product, looking directly at the viewer. The background features a gym with lots of people exercising.

☕

A young woman is facing the camera and holding a box of children's high-protein milk. She smiles gently and introduces the product. The background features a small vanity, woven baskets and muted bed linens, creating a heartfelt and relaxing atmosphere.

☕

A young woman is facing the camera and introducing a red packaged product. She holds the package, smiles, and gestures towards it. The background features a kitchen with wooden cabinets, stainless steel appliances and a breakfast bar, warm and functional.

☕

A man is facing the camera and introducing a product of potato chips. He smiles happily, holding the product. The background features a living room with a gray linen sofa, striped rug and trailing pothos, calm and stylish.

Qualitative Comparison with Previous SOTA methods

We present qualitative video comparisons between HOMIE and existing baseline methods. These video samples are provided as supplementary materials to complement the qualitative evaluation results discussed in the main paper, covering a diverse set of HOCVP scenarios.

Qualitative comparison on general inter-subject HOCVP

Qualitative comparison on inter-subject HOCVP with MLLM reasoning (logo personalization). We include methods that also integrate MLLMs in their models. (SkyReels-A2, BindWeave, UniVideo, VINO)

Qualitative comparison on HOCVP with intra-subject inputs of multi-view references

Qualitative comparison on HOCVP with intra-subject inputs of OCR maps

Practical Application (e.g, livestream product promoting with 1280 x 720 resolution)

HoCo enables HOC video personalization with vertical high resolution, which is well-suited for a variety of practical applications including live-stream product promotion and creative content generation.

☕

A man wearing a clean green T-shirt gently hands a sleek protein container product to another man in a bright, modern kitchen. The kitchen has clean white cabinetry, stainless steel appliances, a marble countertop, and soft natural daylight coming through a window, creating a warm, realistic, and high-definition indoor scene. Smooth, natural, and realistic human movements, high video quality, sharp details, stable composition, professional lighting.

☕

A blue-skinned Na'vi woman and a man wearing a dark-blue shirt sit comfortably on a soft leather sofa, engaged in calm, natural conversation. The background features a warm, modern interior with soft ambient lighting, minimalist decor, and a relaxed, cozy atmosphere. Smooth, realistic character movements, high detail, cinematic lighting, stable composition, lifelike facial expressions and natural body language.

☕

A man wearing a cowboy hat, holding a water bottle and smiling warmly at the camera. He stands in a classic American Western-style town with wooden storefronts, saloon signs, and dirt roads.

☕

A young woman is facing the camera and introducing a jar of herbal powder and gently placing it on a table. Next to the jar are a ceramic cup and a small spoon. The background features tatami mats, shoji screens and a bamboo tea set..

☕

A young man is standing in a clean and tidy kitchen, introducing potato chip package. He holds the product in his hand, presenting it to the viewer with a slight smile. The background features open shelves, ceramic dishes and soft under-cabinet lighting, creating a rustic and inviting atmosphere.

☕

A man is sitting in a fast-food restaurant, facing the camera and holding a bucket of fried chicken. He introduces the fried chicken with a gentle smile.

☕

A man is facing the camera and placing a bag of dog food gently on the table and introducing the product. The background is a cozy indoor space with a soft carpet, a lovely dog, creating a warm and pet-loving atmosphere.

☕

A woman is facing the camera and introducing a bottle of sake. She holds the bottle, smiles, and gestures towards it. The background features a Japanese tea room with low tables, sliding doors and potted moss, serene and refined.

☕

A young lady wearing a white pleated skirt stands facing the lens, puts a cute animal plush doll down on the desktop with both hands. After that, she turns her back to the camera and walks off slowly. The setting is a clean minimalist office with soft indoor lighting.

☕

An animated boy with blue hair is standing, introducing a yellow alarm clock. He holds it up, smiling. The background features a basic bed, curtains and potted peace lily, creating a bright and calm atmosphere.

☕

An animate fox is facing the camera and holding a pink backpack. The fox presents the backpack with a gentle smile. The background features an office with a wooden desk, desk lamp, creating a focused productive atmosphere.

☕

A 3D-figure in a red and silver uniform is facing the camera and introducing a black motorcycle helmet. He holds the helmet, presenting it to the viewer. The background is a sci-fi-style commanding room with a large monitor screen.

HOMIE

Human-object Centric Video Personalization via Multimodal Intelligent Enhancement

Abstract

Contents