1<!DOCTYPE html><html><head><meta charSet="utf-8"/><title>Creating Better Virtual Human â Felipe Jeon</title><meta name="robots" content="index,follow"/><meta name="description" content="This short article overlays methods to build virtual human from generative AI"/><meta property="og:title" content="Creating Better Virtual Human â Felipe Jeon"/><meta property="og:description" content="This short article overlays methods to build virtual human from generative AI"/><meta name="viewport" content="width=device-width, initial-scale=1.0, viewport-fit=cover"/><style> 2 :root { 3 --nextra-primary-hue: 215deg; 4 --nextra-navbar-height: 4rem; 5 --nextra-menu-height: 3.75rem; 6 --nextra-banner-height: 2.5rem; 7 } 8 9 .dark { 10 --nextra-primary-hue: 215deg; 11 } 12 </style><link rel="icon" href="/favicon.ico" type="image/ico"/><meta name="theme-color" content="#fff"/><meta name="og:title" content="Boseong Jeon"/><meta name="description" content="Web page of Boseong Jeon"/><meta name="next-head-count" content="12"/><link rel="preload" href="/_next/static/css/0fa7b2a11118d4fa.css" as="style"/><link rel="stylesheet" href="/_next/static/css/0fa7b2a11118d4fa.css" data-n-g=""/><noscript data-n-css=""></noscript>
12<script defer="" nomodule="" src="/_next/static/chunks/polyfills-c67a75d1b6f99dc8.js"></script>
12<script src="/_next/static/chunks/webpack-36d12a75f0098f30.js" defer=""></script>
12<script src="/_next/static/chunks/framework-8d78bf989db74c8f.js" defer=""></script>
12<script src="/_next/static/chunks/main-2aa13b80f5cbf18c.js" defer=""></script>
12<script src="/_next/static/chunks/pages/_app-d0d1aae468005726.js" defer=""></script>
12<script src="/_next/static/chunks/249-acccf50fa6e9cb26.js" defer=""></script>
12<script src="/_next/static/chunks/pages/blog/post23-6a6ce1a5e77fc9cd.js" defer=""></script>
12<script src="/_next/static/AC_89zEFVe5ph3g8AqoeM/_buildManifest.js" defer=""></script>
12<script src="/_next/static/AC_89zEFVe5ph3g8AqoeM/_ssgManifest.js" defer=""></script>
12</head><body><div id="__next">
12<script>!function(){try{var d=document.documentElement,c=d.classList;c.remove('light','dark');var e=localStorage.getItem('theme');if('system'===e||(!e&&true)){var t='(prefers-color-scheme: dark)',m=window.matchMedia(t);if(m.media!==t||m.matches){d.style.colorScheme = 'dark';c.add('dark')}else{d.style.colorScheme = 'light';c.add('light')}}else if(e){c.add(e|| '')}if(e==='light'||e==='dark')d.style.colorScheme=e}catch(e){}}()</script>
12<div dir="ltr">
12<script>document.documentElement.setAttribute('dir','ltr')</script>
12<div class="nextra-nav-container nx-sticky nx-top-0 nx-z-20 nx-w-full nx-bg-transparent print:nx-hidden"><div class="nextra-nav-container-blur nx-pointer-events-none nx-absolute nx-z-[-1] nx-h-full nx-w-full nx-bg-white dark:nx-bg-dark nx-shadow-[0_2px_4px_rgba(0,0,0,.02),0_1px_0_rgba(0,0,0,.06)] dark:nx-shadow-[0_-1px_0_rgba(255,255,255,.1)_inset] contrast-more:nx-shadow-[0_0_0_1px_#000] contrast-more:dark:nx-shadow-[0_0_0_1px_#fff]"></div><nav class="nx-mx-auto nx-flex nx-h-[var(--nextra-navbar-height)] nx-max-w-[90rem] nx-items-center nx-justify-end nx-gap-2 nx-pl-[max(env(safe-area-inset-left),1.5rem)] nx-pr-[max(env(safe-area-inset-right),1.5rem)]"><a class="nx-flex nx-items-center hover:nx-opacity-75 ltr:nx-mr-auto rtl:nx-ml-auto" href="/"><p class="hover:scale-105">ð²</p></a><a class="nx-text-sm contrast-more:nx-text-gray-700 contrast-more:dark:nx-text-gray-100 nx-relative -nx-ml-2 nx-hidden nx-whitespace-nowrap nx-p-2 md:nx-inline-block nx-font-medium nx-subpixel-antialiased" aria-current="true" href="/blog"><span class="nx-absolute nx-inset-x-0 nx-text-center">Blog</span><span class="nx-invisible nx-font-medium">Blog</span></a><div class="nextra-search nx-relative md:nx-w-64 nx-hidden md:nx-inline-block mx-min-w-[200px]"><div class="nx-relative nx-flex nx-items-center nx-text-gray-900 contrast-more:nx-text-gray-800 dark:nx-text-gray-300 contrast-more:dark:nx-text-gray-300"><input spellcheck="false" class="nx-block nx-w-full nx-appearance-none nx-rounded-lg nx-px-3 nx-py-2 nx-transition-colors nx-text-base nx-leading-tight md:nx-text-sm nx-bg-black/[.05] dark:nx-bg-gray-50/10 focus:nx-bg-white dark:focus:nx-bg-dark placeholder:nx-text-gray-500 dark:placeholder:nx-text-gray-400 contrast-more:nx-border contrast-more:nx-border-current" type="search" placeholder="search anything ð¦" value=""/></div></div><button type="button" aria-label="Menu" class="nextra-hamburger -nx-mr-2 nx-rounded nx-p-2 active:nx-bg-gray-400/20 md:nx-hidden"><svg fill="none" width="24" height="24" viewBox="0 0 24 24" stroke="currentColor" class=""><g><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M4 6h16"></path></g><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M4 12h16"></path><g><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M4 18h16"></path></g></svg></button></nav></div><div class="nx-mx-auto nx-flex nx-max-w-[90rem]"><div class="max-xl:nx-hidden nx-h-0 nx-w-64 nx-shrink-0"></div><div class="motion-reduce:nx-transition-none [transition:background-color_1.5s_ease] nx-bg-transparent"></div><aside class="nextra-sidebar-container nx-flex nx-flex-col md:nx-top-16 md:nx-shrink-0 motion-reduce:nx-transform-none nx-transform-gpu nx-transition-all nx-ease-in-out print:nx-hidden md:nx-w-64 md:nx-hidden max-md:[transform:translate3d(0,-100%,0)]"><div class="nx-px-4 nx-pt-4 md:nx-hidden"><div class="nextra-search nx-relative md:nx-w-64"><div class="nx-relative nx-flex nx-items-center nx-text-gray-900 contrast-more:nx-text-gray-800 dark:nx-text-gray-300 contrast-more:dark:nx-text-gray-300"><input spellcheck="false" class="nx-block nx-w-full nx-appearance-none nx-rounded-lg nx-px-3 nx-py-2 nx-transition-colors nx-text-base nx-leading-tight md:nx-text-sm nx-bg-black/[.05] dark:nx-bg-gray-50/10 focus:nx-bg-white dark:focus:nx-bg-dark placeholder:nx-text-gray-500 dark:placeholder:nx-text-gray-400 contrast-more:nx-border contrast-more:nx-border-current" type="search" placeholder="search anything ð¦" value=""/></div></div></div><div class="nx-overflow-y-auto nx-overflow-x-hidden nx-p-4 nx-grow md:nx-h-[calc(100vh-var(--nextra-navbar-height)-var(--nextra-menu-height))] nextra-scrollbar"><ul class="nx-flex nx-flex-col nx-gap-1 md:nx-hidden"><li class="nx-flex nx-flex-col nx-gap-1"><a class="nx-flex nx-rounded nx-px-2 nx-py-1.5 nx-text-sm nx-transition-colors [word-break:break-word] nx-cursor-pointer [-webkit-tap-highlight-color:transparent] [-webkit-touch-callout:none] contrast-more:nx-border nx-text-gray-500 hover:nx-bg-gray-100 hover:nx-text-gray-900 dark:nx-text-neutral-400 dark:hover:nx-bg-primary-100/5 dark:hover:nx-text-gray-50 contrast-more:nx-text-gray-900 contrast-more:dark:nx-text-gray-50 contrast-more:nx-border-transparent contrast-more:hover:nx-border-gray-900 contrast-more:dark:hover:nx-border-gray-50" href="/">Welcome to my websiteð</a></li><li class="open"><a class="nx-items-center nx-justify-between nx-gap-2 nx-flex nx-rounded nx-px-2 nx-py-1.5 nx-text-sm nx-transition-colors [word-break:break-word] nx-cursor-pointer [-webkit-tap-highlight-color:transparent] [-webkit-touch-callout:none] contrast-more:nx-border nx-text-gray-500 hover:nx-bg-gray-100 hover:nx-text-gray-900 dark:nx-text-neutral-400 dark:hover:nx-bg-primary-100/5 dark:hover:nx-text-gray-50 contrast-more:nx-text-gray-900 contrast-more:dark:nx-text-gray-50 contrast-more:nx-border-transparent contrast-more:hover:nx-border-gray-900 contrast-more:dark:hover:nx-border-gray-50" href="/blog">Blog<svg fill="none" viewBox="0 0 24 24" stroke="currentColor" class="nx-h-[18px] nx-min-w-[18px] nx-rounded-sm nx-p-0.5 hover:nx-bg-gray-800/5 dark:hover:nx-bg-gray-100/5"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M9 5l7 7-7 7" class="nx-origin-center nx-transition-transform rtl:-nx-rotate-180 ltr:nx-rotate-90 rtl:nx-rotate-[-270deg]"></path></svg></a><div class="nx-transform-gpu nx-overflow-hidden nx-transition-all nx-ease-in-out motion-reduce:nx-transition-none"><div class="nx-transition-opacity nx-duration-500 nx-ease-in-out motion-reduce:nx-transition-none nx-opacity-100 ltr:nx-pr-0 rtl:nx-pl-0 nx-pt-1"><ul class="nx-flex nx-flex-col nx-gap-1 nx-relative before:nx-absolute before:nx-inset-y-1 before:nx-w-px before:nx-bg-gray-200 before:nx-content-[""] dark:before:nx-bg-neutral-800 ltr:nx-pl-3 ltr:before:nx-left-0 rtl:nx-pr-3 rtl:before:nx-right-0 ltr:nx-ml-3 rtl:nx-mr-3"><li class="nx-flex nx-flex-col nx-gap-1 active"><a class="nx-flex nx-rounded nx-px-2 nx-py-1.5 nx-text-sm nx-transition-colors [word-break:break-word] nx-cursor-pointer [-webkit-tap-highlight-color:transparent] [-webkit-touch-callout:none] contrast-more:nx-border nx-bg-primary-100 nx-font-semibold nx-text-primary-800 dark:nx-bg-primary-400/10 dark:nx-text-primary-600 contrast-more:nx-border-primary-500 contrast-more:dark:nx-border-primary-500" href="/blog/post23">Creating Better Virtual Human</a><ul class="nx-flex nx-flex-col nx-gap-1 nx-relative before:nx-absolute before:nx-inset-y-1 before:nx-w-px before:nx-bg-gray-200 before:nx-content-[""] dark:before:nx-bg-neutral-800 ltr:nx-pl-3 ltr:before:nx-left-0 rtl:nx-pr-3 rtl:before:nx-right-0 ltr:nx-ml-3 rtl:nx-mr-3"><li><a href="#1-face-swapping-after-generation" class="nx-flex nx-rounded nx-px-2 nx-py-1.5 nx-text-sm nx-transition-colors [word-break:break-word] nx-cursor-pointer [-webkit-tap-highlight-color:transparent] [-webkit-touch-callout:none] contrast-more:nx-border nx-flex nx-gap-2 before:nx-opacity-25 before:nx-content-["#"] nx-text-gray-500 hover:nx-bg-gray-100 hover:nx-text-gray-900 dark:nx-text-neutral-400 dark:hover:nx-bg-primary-100/5 dark:hover:nx-text-gray-50 contrast-more:nx-text-gray-900 contrast-more:dark:nx-text-gray-50 contrast-more:nx-border-transparent contrast-more:hover:nx-border-gray-900 contrast-more:dark:hover:nx-border-gray-50">1. Face Swapping after Generation</a></li><li><a href="#2-ip-adaptors-faceid" class="nx-flex nx-rounded nx-px-2 nx-py-1.5 nx-text-sm nx-transition-colors [word-break:break-word] nx-cursor-pointer [-webkit-tap-highlight-color:transparent] [-webkit-touch-callout:none] contrast-more:nx-border nx-flex nx-gap-2 before:nx-opacity-25 before:nx-content-["#"] nx-text-gray-500 hover:nx-bg-gray-100 hover:nx-text-gray-900 dark:nx-text-neutral-400 dark:hover:nx-bg-primary-100/5 dark:hover:nx-text-gray-50 contrast-more:nx-text-gray-900 contrast-more:dark:nx-text-gray-50 contrast-more:nx-border-transparent contrast-more:hover:nx-border-gray-900 contrast-more:dark:hover:nx-border-gray-50">2. IP-Adaptors (FaceID)</a></li><li><a href="#3-training-customized-models-dreambooth-and-lora" class="nx-flex nx-rounded nx-px-2 nx-py-1.5 nx-text-sm nx-transition-colors [word-break:break-word] nx-cursor-pointer [-webkit-tap-highlight-color:transparent] [-webkit-touch-callout:none] contrast-more:nx-border nx-flex nx-gap-2 before:nx-opacity-25 before:nx-content-["#"] nx-text-gray-500 hover:nx-bg-gray-100 hover:nx-text-gray-900 dark:nx-text-neutral-400 dark:hover:nx-bg-primary-100/5 dark:hover:nx-text-gray-50 contrast-more:nx-text-gray-900 contrast-more:dark:nx-text-gray-50 contrast-more:nx-border-transparent contrast-more:hover:nx-border-gray-900 contrast-more:dark:hover:nx-border-gray-50">3. Training Customized Models (Dreambooth and LoRA)</a></li></ul></li><li class="nx-flex nx-flex-col nx-gap-1"><a class="nx-flex nx-rounded nx-px-2 nx-py-1.5 nx-text-sm nx-transition-colors [word-break:break-word] nx-cursor-pointer [-webkit-tap-highlight-color:transparent] [-webkit-touch-callout:none] contrast-more:nx-border nx-text-gray-500 hover:nx-bg-gray-100 hover:nx-text-gray-900 dark:nx-text-neutral-400 dark:hover:nx-bg-primary-100/5 dark:hover:nx-text-gray-50 contrast-more:nx-text-gray-900 contrast-more:dark:nx-text-gray-50 contrast-more:nx-border-transparent contrast-more:hover:nx-border-gray-900 contrast-more:dark:hover:nx-border-gray-50" href="/blog/post21">Hierarchial Framework - Combination of Planners</a></li><li class="nx-flex nx-flex-col nx-gap-1"><a class="nx-flex nx-rounded nx-px-2 nx-py-1.5 nx-text-sm nx-transition-colors [word-break:break-word] nx-cursor-pointer [-webkit-tap-highlight-color:transparent] [-webkit-touch-callout:none] contrast-more:nx-border nx-text-gray-500 hover:nx-bg-gray-100 hover:nx-text-gray-900 dark:nx-text-neutral-400 dark:hover:nx-bg-primary-100/5 dark:hover:nx-text-gray-50 contrast-more:nx-text-gray-900 contrast-more:dark:nx-text-gray-50 contrast-more:nx-border-transparent contrast-more:hover:nx-border-gray-900 contrast-more:dark:hover:nx-border-gray-50" href="/blog/post20">Types of Motion Planners at a Glance</a></li><li class="nx-flex nx-flex-col nx-gap-1"><a class="nx-flex nx-rounded nx-px-2 nx-py-1.5 nx-text-sm nx-transition-colors [word-break:break-word] nx-cursor-pointer [-webkit-tap-highlight-color:transparent] [-webkit-touch-callout:none] contrast-more:nx-border nx-text-gray-500 hover:nx-bg-gray-100 hover:nx-text-gray-900 dark:nx-text-neutral-400 dark:hover:nx-bg-primary-100/5 dark:hover:nx-text-gray-50 contrast-more:nx-text-gray-900 contrast-more:dark:nx-text-gray-50 contrast-more:nx-border-transparent contrast-more:hover:nx-border-gray-900 contrast-more:dark:hover:nx-border-gray-50" href="/blog/post19">Gentle introduction to Hybrid A star in minutes</a></li></ul></div></div></li></ul></div>
12<div class="nx-sticky nx-bottom-0 nx-bg-white dark:nx-bg-dark nx-mx-4 nx-py-4 nx-shadow-[0_-12px_16px_#fff] nx-flex nx-items-center nx-gap-2 dark:nx-border-neutral-800 dark:nx-shadow-[0_-12px_16px_#111] contrast-more:nx-border-neutral-400 contrast-more:nx-shadow-none contrast-more:dark:nx-shadow-none nx-border-t" data-toggle-animation="off"><div class="nx-grow nx-flex nx-flex-col"><button title="Change theme" class="nx-h-7 nx-rounded-md nx-px-2 nx-text-left nx-text-xs nx-font-medium nx-text-gray-600 nx-transition-colors dark:nx-text-gray-400 hover:nx-bg-gray-100 hover:nx-text-gray-900 dark:hover:nx-bg-primary-100/5 dark:hover:nx-text-gray-50" id="headlessui-listbox-button-:R2njcm:" type="button" aria-haspopup="listbox" aria-expanded="false" data-headlessui-state=""><div class="nx-flex nx-items-center nx-gap-2 nx-capitalize"><svg fill="none" viewBox="3 3 18 18" width="12" height="12" stroke="currentColor"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" fill="currentColor" d="M12 3v1m0 16v1m9-9h-1M4 12H3m15.364 6.364l-.707-.707M6.343 6.343l-.707-.707m12.728 0l-.707.707M6.343 17.657l-.707.707M16 12a4 4 0 11-8 0 4 4 0 018 0z"></path></svg><span class="">Light</span></div></button></div><button title="Hide sidebar" class="max-md:nx-hidden nx-h-7 nx-rounded-md nx-transition-colors nx-text-gray-600 dark:nx-text-gray-400 nx-px-2 hover:nx-bg-gray-100 hover:nx-text-gray-900 dark:hover:nx-bg-primary-100/5 dark:hover:nx-text-gray-50"><svg height="12" width="12" viewBox="0 0 16 16" fill="currentColor"><path fill-rule="evenodd" d="M4.177 7.823l2.396-2.396A.25.25 0 017 5.604v4.792a.25.25 0 01-.427.177L4.177 8.177a.25.25 0 010-.354z" class=""></path><path fill-rule="evenodd" d="M0 1.75C0 .784.784 0 1.75 0h12.5C15.216 0 16 .784 16 1.75v12.5A1.75 1.75 0 0114.25 16H1.75A1.75 1.75 0 010 14.25V1.75zm1.75-.25a.25.25 0 00-.25.25v12.5c0 .138.112.25.25.25H9.5v-13H1.75zm12.5 13H11v-13h3.25a.25.25 0 01.25.25v12.5a.25.25 0 01-.25.25z"></path></svg></button></div></aside><nav class="nextra-toc nx-order-last nx-hidden nx-w-64 nx-shrink-0 xl:nx-block print:nx-hidden nx-px-4" aria-label="table of contents"><div class="nextra-scrollbar nx-sticky nx-top-16 nx-overflow-y-auto nx-pr-4 nx-pt-6 nx-text-sm [hyphens:auto] nx-max-h-[calc(100vh-var(--nextra-navbar-height)-env(safe-area-inset-bottom))] ltr:-nx-mr-4 rtl:-nx-ml-4"><p class="nx-mb-4 nx-font-semibold nx-tracking-tight">On This Page</p><ul><li class="nx-my-2 nx-scroll-my-6 nx-scroll-py-6"><a href="#1-face-swapping-after-generation" class="nx-font-semibold nx-inline-block nx-text-gray-500 hover:nx-text-gray-900 dark:nx-text-gray-400 dark:hover:nx-text-gray-300 contrast-more:nx-text-gray-900 contrast-more:nx-underline contrast-more:dark:nx-text-gray-50 nx-w-full nx-break-words">1. Face Swapping after Generation</a></li><li class="nx-my-2 nx-scroll-my-6 nx-scroll-py-6"><a href="#pros" class="ltr:nx-pl-4 rtl:nx-pr-4 nx-inline-block nx-text-gray-500 hover:nx-text-gray-900 dark:nx-text-gray-400 dark:hover:nx-text-gray-300 contrast-more:nx-text-gray-900 contrast-more:nx-underline contrast-more:dark:nx-text-gray-50 nx-w-full nx-break-words">Pros</a></li><li class="nx-my-2 nx-scroll-my-6 nx-scroll-py-6"><a href="#cons" class="ltr:nx-pl-4 rtl:nx-pr-4 nx-inline-block nx-text-gray-500 hover:nx-text-gray-900 dark:nx-text-gray-400 dark:hover:nx-text-gray-300 contrast-more:nx-text-gray-900 contrast-more:nx-underline contrast-more:dark:nx-text-gray-50 nx-w-full nx-break-words">Cons</a></li><li class="nx-my-2 nx-scroll-my-6 nx-scroll-py-6"><a href="#how-to-overcome-face-resolution-drop" class="ltr:nx-pl-4 rtl:nx-pr-4 nx-inline-block nx-text-gray-500 hover:nx-text-gray-900 dark:nx-text-gray-400 dark:hover:nx-text-gray-300 contrast-more:nx-text-gray-900 contrast-more:nx-underline contrast-more:dark:nx-text-gray-50 nx-w-full nx-break-words">How to Overcome Face Resolution Drop?</a></li><li class="nx-my-2 nx-scroll-my-6 nx-scroll-py-6"><a href="#2-ip-adaptors-faceid" class="nx-font-semibold nx-inline-block nx-text-gray-500 hover:nx-text-gray-900 dark:nx-text-gray-400 dark:hover:nx-text-gray-300 contrast-more:nx-text-gray-900 contrast-more:nx-underline contrast-more:dark:nx-text-gray-50 nx-w-full nx-break-words">2. IP-Adaptors (FaceID)</a></li><li class="nx-my-2 nx-scroll-my-6 nx-scroll-py-6"><a href="#pros-1" class="ltr:nx-pl-4 rtl:nx-pr-4 nx-inline-block nx-text-gray-500 hover:nx-text-gray-900 dark:nx-text-gray-400 dark:hover:nx-text-gray-300 contrast-more:nx-text-gray-900 contrast-more:nx-underline contrast-more:dark:nx-text-gray-50 nx-w-full nx-break-words">Pros</a></li><li class="nx-my-2 nx-scroll-my-6 nx-scroll-py-6"><a href="#cons-1" class="ltr:nx-pl-4 rtl:nx-pr-4 nx-inline-block nx-text-gray-500 hover:nx-text-gray-900 dark:nx-text-gray-400 dark:hover:nx-text-gray-300 contrast-more:nx-text-gray-900 contrast-more:nx-underline contrast-more:dark:nx-text-gray-50 nx-w-full nx-break-words">Cons</a></li><li class="nx-my-2 nx-scroll-my-6 nx-scroll-py-6"><a href="#3-training-customized-models-dreambooth-and-lora" class="nx-font-semibold nx-inline-block nx-text-gray-500 hover:nx-text-gray-900 dark:nx-text-gray-400 dark:hover:nx-text-gray-300 contrast-more:nx-text-gray-900 contrast-more:nx-underline contrast-more:dark:nx-text-gray-50 nx-w-full nx-break-words">3. Training Customized Models (Dreambooth and LoRA)</a></li><li class="nx-my-2 nx-scroll-my-6 nx-scroll-py-6"><a href="#pros-2" class="ltr:nx-pl-4 rtl:nx-pr-4 nx-inline-block nx-text-gray-500 hover:nx-text-gray-900 dark:nx-text-gray-400 dark:hover:nx-text-gray-300 contrast-more:nx-text-gray-900 contrast-more:nx-underline contrast-more:dark:nx-text-gray-50 nx-w-full nx-break-words">Pros</a></li><li class="nx-my-2 nx-scroll-my-6 nx-scroll-py-6"><a href="#cons-2" class="ltr:nx-pl-4 rtl:nx-pr-4 nx-inline-block nx-text-gray-500 hover:nx-text-gray-900 dark:nx-text-gray-400 dark:hover:nx-text-gray-300 contrast-more:nx-text-gray-900 contrast-more:nx-underline contrast-more:dark:nx-text-gray-50 nx-w-full nx-break-words">Cons</a></li></ul><div class="nx-mt-8 nx-border-t nx-bg-white nx-pt-8 nx-shadow-[0_-12px_16px_white] dark:nx-bg-dark dark:nx-shadow-[0_-12px_16px_#111] nx-sticky nx-bottom-0
12nx-flex nx-flex-col nx-items-start nx-gap-2 nx-pb-8 dark:nx-border-neutral-800 contrast-more:nx-border-t contrast-more:nx-border-neutral-400 contrast-more:nx-shadow-none contrast-more:dark:nx-border-neutral-400"><a class="nx-text-xs nx-font-medium nx-text-gray-500 hover:nx-text-gray-900 dark:nx-text-gray-400 dark:hover:nx-text-gray-100 contrast-more:nx-text-gray-800 contrast-more:dark:nx-text-gray-50" href="https://github.com/shuding/nextra/pages/blog/post23.mdx"></a></div></div></nav><div id="reach-skip-nav"></div><article class="nx-w-full nx-break-words nextra-content nx-flex nx-min-h-[calc(100vh-var(--nextra-navbar-height))] nx-min-w-0 nx-justify-center nx-pb-8 nx-pr-[calc(env(safe-area-inset-right)-1.5rem)] nextra-body-typesetting-article"><main class="nx-w-full nx-min-w-0 nx-max-w-6xl nx-px-6 nx-pt-4 md:nx-px-12"><h1 class="nx-mt-2 nx-text-4xl nx-font-bold nx-tracking-tight nx-text-slate-900 dark:nx-text-slate-100">Creating Better Virtual Human</h1> 13<div class="text-center"><div class="my-4 text-sm text-neutral-400 dark:text-neutral-500">October 13, 2024</div><img alt="This short article overlays methods to build virtual human from generative AI" srcSet="/_next/image?url=%2Fimages%2Fpost23%2F42_05.png&w=750&q=75 1x, /_next/image?url=%2Fimages%2Fpost23%2F42_05.png&w=1920&q=75 2x" src="/_next/image?url=%2Fimages%2Fpost23%2F42_05.png&w=1920&q=75" width="700" height="700" decoding="async" data-nimg="1" class="rounded-xl mx-auto" loading="lazy" style="color:transparent"/></div> 14<p class="nx-mt-6 nx-leading-7 first:nx-mt-0">ð¨ Final result: <a href="https://www.instagram.com/geni.baek.art/" target="_blank" rel="noreferrer" class="nx-text-primary-600 nx-underline nx-decoration-from-font [text-underline-position:from-font]">https://www.instagram.com/geni.baek.art/<span class="nx-sr-only"> (opens in a new tab)</span></a></p> 15<p class="nx-mt-6 nx-leading-7 first:nx-mt-0">It is extremely common to create virtual human these days, and there are multiple ways to it. 16You may heard about virtual influencers such as <a href="https://www.instagram.com/rozy.gram/" target="_blank" rel="noreferrer" class="nx-text-primary-600 nx-underline nx-decoration-from-font [text-underline-position:from-font]">Rozy<span class="nx-sr-only"> (opens in a new tab)</span></a>, 17<a href="https://www.instagram.com/lilmiquela/" target="_blank" rel="noreferrer" class="nx-text-primary-600 nx-underline nx-decoration-from-font [text-underline-position:from-font]">Miquela<span class="nx-sr-only"> (opens in a new tab)</span></a>, and <a href="https://www.instagram.com/imma.gram/" target="_blank" rel="noreferrer" class="nx-text-primary-600 nx-underline nx-decoration-from-font [text-underline-position:from-font]">Imma<span class="nx-sr-only"> (opens in a new tab)</span></a>. 18They are not 100% generation in the sense that their faces are rigged on real human.</p> 19<p class="nx-mt-6 nx-leading-7 first:nx-mt-0">This article discusses how to create a good virtual human from end to end generation using diffusion or other AI models. 20Roughly, there are three ways for this: 1) face swap after image generations, 212) <a href="https://huggingface.co/h94/IP-Adapter-FaceID" target="_blank" rel="noreferrer" class="nx-text-primary-600 nx-underline nx-decoration-from-font [text-underline-position:from-font]">Ip-Adaptor Face ID<span class="nx-sr-only"> (opens in a new tab)</span></a> (or similar adaptor approaches), and 223) training an existing model for adaptation (Dreambooth or LoRA).</p> 23<p class="nx-mt-6 nx-leading-7 first:nx-mt-0">Here, I analyze their pros and cons in terms of costs, text alignment, and balance between diversity and consistency.</p> 24<h2 class="nx-font-semibold nx-tracking-tight nx-text-slate-900 dark:nx-text-slate-100 nx-mt-10 nx-border-b nx-pb-1 nx-text-3xl nx-border-neutral-200/70 contrast-more:nx-border-neutral-400 dark:nx-border-primary-100/10 contrast-more:dark:nx-border-neutral-400">1. Face Swapping after Generation<span class="nx-absolute -nx-mt-20" id="1-face-swapping-after-generation"></span><a href="#1-face-swapping-after-generation" class="subheading-anchor" aria-label="Permalink for this section"></a></h2> 25<p class="nx-mt-6 nx-leading-7 first:nx-mt-0"><em>Face swapping</em> is using the code like <a href="https://github.com/haofanwang/inswapper?tab=readme-ov-file" target="_blank" rel="noreferrer" class="nx-text-primary-600 nx-underline nx-decoration-from-font [text-underline-position:from-font]">inswapper<span class="nx-sr-only"> (opens in a new tab)</span></a>, and it is well provided from webui or comfyui via (<a href="https://github.com/Gourieff/sd-webui-reactor" target="_blank" rel="noreferrer" class="nx-text-primary-600 nx-underline nx-decoration-from-font [text-underline-position:from-font]">Reactor<span class="nx-sr-only"> (opens in a new tab)</span></a>). 26In can be used for any generated images. (If you use this on the real human pictures, this act is called Deepfake.)</p> 27<div class="flex flex-col items-center"><img srcSet="/_next/image?url=%2Fimages%2Fpost23%2Ffaceswap.png&w=1080&q=75 1x, /_next/image?url=%2Fimages%2Fpost23%2Ffaceswap.png&w=2048&q=75 2x" src="/_next/image?url=%2Fimages%2Fpost23%2Ffaceswap.png&w=2048&q=75" width="1000" height="500" decoding="async" data-nimg="1" class="my-3" loading="lazy" style="color:transparent"/><p><strong>Fig.</strong> Face swap pipeline</p></div> 28<h3 class="nx-font-semibold nx-tracking-tight nx-text-slate-900 dark:nx-text-slate-100 nx-mt-8 nx-text-2xl">Pros<span class="nx-absolute -nx-mt-20" id="pros"></span><a href="#pros" class="subheading-anchor" aria-label="Permalink for this section"></a></h3> 29<ul class="nx-mt-6 nx-list-disc first:nx-mt-0 ltr:nx-ml-6 rtl:nx-mr-6"> 30<li class="nx-my-2">
30You do not depend on a specific generation models. You can use swapping on the images generated from various styles</li> 31<li class="nx-my-2">No training is required.</li> 32</ul> 33<h3 class="nx-font-semibold nx-tracking-tight nx-text-slate-900 dark:nx-text-slate-100 nx-mt-8 nx-text-2xl">Cons<span class="nx-absolute -nx-mt-20" id="cons"></span><a href="#cons" class="subheading-anchor" aria-label="Permalink for this section"></a></h3> 34<ul class="nx-mt-6 nx-list-disc first:nx-mt-0 ltr:nx-ml-6 rtl:nx-mr-6"> 35<li class="nx-my-2">Most of the swapping models are limited to <a href="https://github.com/deepinsight/insightface/issues/2270" target="_blank" rel="noreferrer" class="nx-text-primary-600 nx-underline nx-decoration-from-font [text-underline-position:from-font]">low resolution face images<span class="nx-sr-only"> (opens in a new tab)</span></a>.</li> 36<li class="nx-my-2">Lose lighting or details.</li> 37<li class="nx-my-2">Mal-performing on face pose changes.</li> 38<li class="nx-my-2">Very difficult to have makeup variations such as face painting.</li> 39</ul> 40<div class="flex flex-col items-center"><img srcSet="/_next/image?url=%2Fimages%2Fpost23%2Fdetails.png&w=1080&q=75 1x, /_next/image?url=%2Fimages%2Fpost23%2Fdetails.png&w=2048&q=75 2x" src="/_next/image?url=%2Fimages%2Fpost23%2Fdetails.png&w=2048&q=75" width="1000" height="500" decoding="async" data-nimg="1" class="my-3" loading="lazy" style="color:transparent"/><p><strong>Fig.</strong> Lighting and resolution degraded.</p></div> 41<div class="flex flex-col items-center"><img srcSet="/_next/image?url=%2Fimages%2Fpost23%2Fpose.png&w=1080&q=75 1x, /_next/image?url=%2Fimages%2Fpost23%2Fpose.png&w=2048&q=75 2x" src="/_next/image?url=%2Fimages%2Fpost23%2Fpose.png&w=2048&q=75" width="1000" height="500" decoding="async" data-nimg="1" class="my-3" loading="lazy" style="color:transparent"/><p><strong>Fig.</strong> Only frontal pose is working good.</p></div> 42<h3 class="nx-font-semibold nx-tracking-tight nx-text-slate-900 dark:nx-text-slate-100 nx-mt-8 nx-text-2xl">How to Overcome Face Resolution Drop?<span class="nx-absolute -nx-mt-20" id="how-to-overcome-face-resolution-drop"></span><a href="#how-to-overcome-face-resolution-drop" class="subheading-anchor" aria-label="Permalink for this section"></a></h3> 43<p class="nx-mt-6 nx-leading-7 first:nx-mt-0">Despite of the simplicity, the biggest problem of this approach is resolution drop of the face region. 44Using GAN based upscalers (e.g., GFPGAN) is not working very well. I would recommend inpainting a little bit ont he facial region with low strength as mush as you keep the original face identity.</p> 45<div class="flex flex-col items-center"><img srcSet="/_next/image?url=%2Fimages%2Fpost23%2Fovercome.png&w=1080&q=75 1x, /_next/image?url=%2Fimages%2Fpost23%2Fovercome.png&w=2048&q=75 2x" src="/_next/image?url=%2Fimages%2Fpost23%2Fovercome.png&w=2048&q=75" width="1000" height="500" decoding="async" data-nimg="1" class="my-3" loading="lazy" style="color:transparent"/></div> 46<h2 class="nx-font-semibold nx-tracking-tight nx-text-slate-900 dark:nx-text-slate-100 nx-mt-10 nx-border-b nx-pb-1 nx-text-3xl nx-border-neutral-200/70 contrast-more:nx-border-neutral-400 dark:nx-border-primary-100/10 contrast-more:dark:nx-border-neutral-400">2. IP-Adaptors (FaceID)<span class="nx-absolute -nx-mt-20" id="2-ip-adaptors-faceid"></span><a href="#2-ip-adaptors-faceid" class="subheading-anchor" aria-label="Permalink for this section"></a></h2> 47<p class="nx-mt-6 nx-leading-7 first:nx-mt-0">IP adaptor is essentially an image <strong>prompt</strong>. 48The prompt is extracted from a given image based on CLIP vision encoder. 49<a href="https://huggingface.co/h94/IP-Adapter-FaceID" target="_blank" rel="noreferrer" class="nx-text-primary-600 nx-underline nx-decoration-from-font [text-underline-position:from-font]">FaceID<span class="nx-sr-only"> (opens in a new tab)</span></a> is a specialized IP adaptor to extract meanings from face iamges. 50By using FaceID IPadaptor, it is basically providing an additional prompt: "resembles this face".</p> 51<h3 class="nx-font-semibold nx-tracking-tight nx-text-slate-900 dark:nx-text-slate-100 nx-mt-8 nx-text-2xl">Pros<span class="nx-absolute -nx-mt-20" id="pros-1"></span><a href="#pros-1" class="subheading-anchor" aria-label="Permalink for this section"></a></h3> 52<ul class="nx-mt-6 nx-list-disc first:nx-mt-0 ltr:nx-ml-6 rtl:nx-mr-6"> 53<li class="nx-my-2">Supports high resolution 1024 x 1024 image without resolution drop of the face, if you use FaceID SDXL (<a href="https://huggingface.co/h94/IP-Adapter-FaceID" target="_blank" rel="noreferrer" class="nx-text-primary-600 nx-underline nx-decoration-from-font [text-underline-position:from-font]">here<span class="nx-sr-only"> (opens in a new tab)</span></a>).</li> 54<li class="nx-my-2">We can output various pose, lighting, and so on.</li> 55</ul> 56<h3 class="nx-font-semibold nx-tracking-tight nx-text-slate-900 dark:nx-text-slate-100 nx-mt-8 nx-text-2xl">Cons<span class="nx-absolute -nx-mt-20" id="cons-1"></span><a href="#cons-1" class="subheading-anchor" aria-label="Permalink for this section"></a></h3> 57<ul class="nx-mt-6 nx-list-disc first:nx-mt-0 ltr:nx-ml-6 rtl:nx-mr-6"> 58<li class="nx-my-2">
58It decreases text alignment, as the image prompt can be a big disrupter for your textual prompts. 59Thus, it is very difficult to use long description.</li> 60<li class="nx-my-2">This adaptor will work well only with the models with which IP adaptor is trained. 61In general, those models are basic models such as sd1.5, sdxl, which produce poor realistic photos (that is why <a href="https://civitai.com/models/4201/realistic-vision-v60-b1" target="_blank" rel="noreferrer" class="nx-text-primary-600 nx-underline nx-decoration-from-font [text-underline-position:from-font]">these models<span class="nx-sr-only"> (opens in a new tab)</span></a> were picking up among users).</li> 62<li class="nx-my-2">It is very well known that CLIP vision encoder is capable of grab global semantic, not a detailed information.</li> 63</ul> 64<p class="nx-mt-6 nx-leading-7 first:nx-mt-0">Personally, using IP adaptor for preserving face identity can be an over-use of CLIP vision encoder, 65and I did not get satisfactory result in both identity preservation and text alignment.</p> 66<div class="flex flex-col items-center"><img srcSet="/_next/image?url=%2Fimages%2Fpost23%2Fipadator.png&w=1080&q=75 1x, /_next/image?url=%2Fimages%2Fpost23%2Fipadator.png&w=2048&q=75 2x" src="/_next/image?url=%2Fimages%2Fpost23%2Fipadator.png&w=2048&q=75" width="1000" height="500" decoding="async" data-nimg="1" class="my-3" loading="lazy" style="color:transparent"/></div> 67<h2 class="nx-font-semibold nx-tracking-tight nx-text-slate-900 dark:nx-text-slate-100 nx-mt-10 nx-border-b nx-pb-1 nx-text-3xl nx-border-neutral-200/70 contrast-more:nx-border-neutral-400 dark:nx-border-primary-100/10 contrast-more:dark:nx-border-neutral-400">3. Training Customized Models (Dreambooth and LoRA)<span class="nx-absolute -nx-mt-20" id="3-training-customized-models-dreambooth-and-lora"></span><a href="#3-training-customized-models-dreambooth-and-lora" class="subheading-anchor" aria-label="Permalink for this section"></a></h2> 68<p class="nx-mt-6 nx-leading-7 first:nx-mt-0">In the end, I arrived here. This is the best method if you have enough GPU power. 69Basic concept is feeding a foundation model with your character. 70I selected 10 best quality photos with face swapping on the images generated by <a href="https://huggingface.co/black-forest-labs/FLUX.1-dev" target="_blank" rel="noreferrer" class="nx-text-primary-600 nx-underline nx-decoration-from-font [text-underline-position:from-font]">flux.dev<span class="nx-sr-only"> (opens in a new tab)</span></a>. 71Then, I trained flux.dev model with dreambooth objective by making the model to generate my character if given <code class="nx-border-black nx-border-opacity-[0.04] nx-bg-opacity-[0.03] nx-bg-black nx-break-words nx-rounded-md nx-border nx-py-0.5 nx-px-[.25em] nx-text-[.9em] dark:nx-border-white/10 dark:nx-bg-white/10" dir="ltr">sks woman</code>. 72<code class="nx-border-black nx-border-opacity-[0.04] nx-bg-opacity-[0.03] nx-bg-black nx-break-words nx-rounded-md nx-border nx-py-0.5 nx-px-[.25em] nx-text-[.9em] dark:nx-border-white/10 dark:nx-bg-white/10" dir="ltr">sks</code> means nothing but a word not used in our dictionary. Personally, I used the dreambooth training script after some fixes (here is my <a href="https://github.com/huggingface/diffusers/pull/9565" target="_blank" rel="noreferrer" class="nx-text-primary-600 nx-underline nx-decoration-from-font [text-underline-position:from-font]">PR<span class="nx-sr-only"> (opens in a new tab)</span></a> to huggingface). 73I selected flux as the foundation, as it is the best-performing in fingers, with a insane text alignment.</p> 74<h3 class="nx-font-semibold nx-tracking-tight nx-text-slate-900 dark:nx-text-slate-100 nx-mt-8 nx-text-2xl">Pros<span class="nx-absolute -nx-mt-20" id="pros-2"></span><a href="#pros-2" class="subheading-anchor" aria-label="Permalink for this section"></a></h3> 75<ul class="nx-mt-6 nx-list-disc first:nx-mt-0 ltr:nx-ml-6 rtl:nx-mr-6"> 76<li class="nx-my-2">No face resolution drop, little gap between target image and model outputs (if correctly trained).</li> 77<li class="nx-my-2">Keeping face / lighting details while makeup variation.</li> 78</ul> 79<div class="flex flex-col items-center"><img srcSet="/_next/image?url=%2Fimages%2Fpost23%2Flora.png&w=1080&q=75 1x, /_next/image?url=%2Fimages%2Fpost23%2Flora.png&w=2048&q=75 2x" src="/_next/image?url=%2Fimages%2Fpost23%2Flora.png&w=2048&q=75" width="1000" height="500" decoding="async" data-nimg="1" class="my-3" loading="lazy" style="color:transparent"/></div> 80<h3 class="nx-font-semibold nx-tracking-tight nx-text-slate-900 dark:nx-text-slate-100 nx-mt-8 nx-text-2xl">Cons<span class="nx-absolute -nx-mt-20" id="cons-2"></span><a href="#cons-2" class="subheading-anchor" aria-label="Permalink for this section"></a></h3> 81<ul class="nx-mt-6 nx-list-disc first:nx-mt-0 ltr:nx-ml-6 rtl:nx-mr-6"> 82<li class="nx-my-2">Require many trials and errors to pick the training checkpoint, balancing id-consistency and diversity. 83If you train too long or <a href="https://huggingface.co/docs/diffusers/training/dreambooth#prior-preservation-loss" target="_blank" rel="noreferrer" class="nx-text-primary-600 nx-underline nx-decoration-from-font [text-underline-position:from-font]">without prior preservation<span class="nx-sr-only"> (opens in a new tab)</span></a>, your model will collapse into only making the photos you provided. 84In contrast, too short training will give you unsatisfactory id.</li> 85<li class="nx-my-2">Require decent GPUs.</li> 86</ul> 87<div class="mt-10"></div><div class="nx-mt-16"></div></main></article></div><footer class="nx-bg-gray-100 nx-pb-[env(safe-area-inset-bottom)] dark:nx-bg-neutral-900 print:nx-bg-transparent"><div class="nx-mx-auto nx-flex nx-max-w-[90rem] nx-gap-2 nx-py-2 nx-px-4 nx-flex"><button title="Change theme" class="nx-h-7 nx-rounded-md nx-px-2 nx-text-left nx-text-xs nx-font-medium nx-text-gray-600 nx-transition-colors dark:nx-text-gray-400 hover:nx-bg-gray-100 hover:nx-text-gray-900 dark:hover:nx-bg-primary-100/5 dark:hover:nx-text-gray-50" id="headlessui-listbox-button-:R2jkm:" type="button" aria-haspopup="listbox" aria-expanded="false" data-headlessui-state=""><div class="nx-flex nx-items-center nx-gap-2 nx-capitalize"><svg fill="none" viewBox="3 3 18 18" width="12" height="12" stroke="currentColor"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" fill="currentColor" d="M12 3v1m0 16v1m9-9h-1M4 12H3m15.364 6.364l-.707-.707M6.343 6.343l-.707-.707m12.728 0l-.707.707M6.343 17.657l-.707.707M16 12a4 4 0 11-8 0 4 4 0 018 0z"></path></svg><span class="">Light</span></div></button></div><hr class="dark:nx-border-neutral-800"/><div class="nx-mx-auto nx-flex nx-max-w-[90rem] nx-justify-center nx-py-12 nx-text-gray-600 dark:nx-text-gray-400 md:nx-justify-start nx-pl
87-[max(env(safe-area-inset-left),1.5rem)] nx-pr-[max(env(safe-area-inset-right),1.5rem)]"><div></div></div></footer></div></div>
87<script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{}},"page":"/blog/post23","query":{},"buildId":"AC_89zEFVe5ph3g8AqoeM","nextExport":true,"autoExport":true,"isFallback":false,"scriptLoader":[]}</script>
87</body></html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.