StreamDiffusionTD v 0.1.1

dotsimulate

Any production build of 2023. I would recommend using version 0.1.5 or most up to date version. many improvements that should make install a bit smoother were added.

Alexia Adana

Which version of TD has this been tested with

adrian arvidsson

i dont see anything in my settings tab in the streamdiffusion window. why could this be?

I am researching simmilar problem whole morning. Reinstalled Visual Studio packages (Windows SDK) as some suggested, tried to activate venv in separate window and install ndi-python in countless ways. Some people say, it's about Windows Defender Firewall, some about CMAKE path problems. No solution yet, maybe these can help your case

יובל כהן

non local will wowwww!

dotsimulate

very reliant on gpu speed currently. Possibly I will have an option for non local at some point. hoping to have time to figure that out soon.

are there ways to increase fps or does it only rely on gpu power?

JANUS

oh thanks. Well I should buy a new pc

dotsimulate

Hey . If your gpu has only 2 gigs of vram, that won't enough to load the model into memory unfortunately. minimum of 6 is recommended.

and I had ndisdk installed already.

JANUS

hello I have this problem with my memory and model too. How can I allocate the memory and how can I download the model from hugging face? Should I download just the safetensors file and put in model? Traceback (most recent call last): File "C:\Users\matte\StreamDiffusion\ndi-stream\..\utils\wrapper.py", line 417, in _load_model pipe: StableDiffusionPipeline = StableDiffusionPipeline.from_pretrained( File "C:\Users\matte\StreamDiffusion\venv\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 864, in to module.to(device, dtype) File "C:\Users\matte\StreamDiffusion\venv\lib\site-packages\torch\nn\modules\module.py", line 1160, in to return self._apply(convert) File "C:\Users\matte\StreamDiffusion\venv\lib\site-packages\torch\nn\modules\module.py", line 810, in _apply module._apply(fn) File "C:\Users\matte\StreamDiffusion\venv\lib\site-packages\torch\nn\modules\module.py", line 810, in _apply module._apply(fn) File "C:\Users\matte\StreamDiffusion\venv\lib\site-packages\torch\nn\modules\module.py", line 810, in _apply module._apply(fn) [Previous line repeated 7 more times] File "C:\Users\matte\StreamDiffusion\venv\lib\site-packages\torch\nn\modules\module.py", line 833, in _apply param_applied = fn(param) File "C:\Users\matte\StreamDiffusion\venv\lib\site-packages\torch\nn\modules\module.py", line 1158, in convert return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking) torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 26.00 MiB. GPU 0 has a total capacty of 2.00 GiB of which 0 bytes is free. Of the allocated memory 1.70 GiB is allocated by PyTorch, and 53.94 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF Model load has failed. Doesn't exist.

dotsimulate

sdxlturbo is not supported yet. That is something that the developers of StreamDiffusion have mentioned they plan to do, but I am unsure on the exact timeline.

Ismael Abrego

Hi, has anyone used the model stabilityai/sdxl-turbo? I'm getting error messages, and streaming is unsuccessful

dotsimulate

Hey I didn't see your second reply. if you are able to shoojt a message over on discord about conda, I think we can make past install work.

dotsimulate

I don't see any errors ! just warnings. if it wasn't working, save/close all TD and cmd windows and reopen the TD project and try again !

HI, I can't log on to discord ... keeps saying my mails are already used. So that is a deadend for me. I keep running in to this error that I don't understand and the stream never starts : ==================================== Initializing NDI Streaming... Sender: 'StreamDiffusion_NDI_out' ==================================== Starting OSC server on 127.0.0.1:4567 `text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["id2label"]` will be overriden. `text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["bos_token_id"]` will be overriden. `text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["eos_token_id"]` will be overriden. C:\Users\******\streamdiff\StreamDiffusion\venv\lib\site-packages\transformers\models\clip\feature_extraction_clip.py:28: FutureWarning: The class CLIPFeatureExtractor is deprecated and will be removed in version 5 of Transformers. Please use CLIPImageProcessor instead. warnings.warn( The config attributes {'skip_prk_steps': True} were passed to LCMScheduler, but are not expected and will be ignored. Please verify your scheduler_config.json configuration file.

dotsimulate

I believe if you reinstall with python 3.10.9 as your python version on PATH then it should work. You will need to delete the old venv folder before resinstalling.

dotsimulate

Both the Lora and the SD model have to both be 1.5 based and either full local file paths to the file or a huggingfaceid to download them.

I've actually got streamdiffusion running locally with conda before so not too sure why it isn't working in TD through venv?

Hi there! I've got this error while running setup.py install for ndi-python ... error error: subprocess-exited-with-error × Running setup.py install for ndi-python did not run successfully. │ exit code: 1 ╰─> [12 lines of output] running install C:\Users\User\Documents\StreamDiffusion\venv\Lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. warnings.warn( running build running build_py creating build creating build\lib.win-amd64-cpython-311 creating build\lib.win-amd64-cpython-311\NDIlib copying NDIlib\__init__.py -> build\lib.win-amd64-cpython-311\NDIlib running build_ext cmake .. -DCMAKE_INSTALL_PREFIX=C:\Users\jleung\AppData\Local\Temp\pip-install-l4jeb7w6\ndi-python_918fa6417cb640af864d665da90d8060\build\install error: command 'cmake' failed: None [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: legacy-install-failure × Encountered error while trying to install package. ╰─> ndi-python note: This is an issue with the package mentioned above, not pip.

Great TOX!!! Anyway, do you know any Lora which works with this TOX? I download and attach some Lora, but nothing happen. I am not sure Lora file is collapsed or I have a wrong operation.

dotsimulate

thanks Amir! thats awesome haha. 10 minutes !!

Absolutely insane TOX. Thank you so much for piecing this together. Was up and running in like 10 minutes producing insane results at 17FPS. Pro tip - I started using Nvidia upscaler TOP downstream of streamdiffusion and the results are really crisp without sacrificing my streamdiffusion FPS. Again, absolutely insane TOX. You're the man!

dotsimulate

Yep. 3.10.9 and check the box to add to PATH

thanks. i should also have python installed right?

dotsimulate

Also ndi-sdk - https://ndi.video/download-ndi-sdk/ Installing using python 3.10.9 and cuda 11.8 is recommended currently. Possibly soon will have update with clearer install process.

dotsimulate

Git can be found here. Make sure to check add to PATH when installing. https://git-scm.com/downloads Git is used for downloading and updating code repositories.

it requires installing GIT. I'm not that familiar with what it actually is, and its install specifications are a bit confusing.is there anything i should pay attention to or should i go with its default installation settings?

Andrea Tamburrini

I solved in a different way: reinstall everything without antivirus ahahah

Downgrade to python 3.10.9 and it worked.

dotsimulate

ah. confused myself with your error. If it is saying there are no ndi device / source, perhaps try restarting or dragging in a new operator. And also make sure something is plugged in, although as far as I know , it should work with no input. Basically, the issue is that your ndi from Touchdesigner is not cooking/active for some reason.

dotsimulate

If you are able to shoot a screenshot over to me on discord, that is easiest for me or someone to help with !

dotsimulate

I thought I responded to this ! not sure where that message disappeared to. Its mostly about the VAE that is running which is https://huggingface.co/madebyollin/taesd When this encodes and decodes the images a lot of details are lost. There are definitely other factors and it improves with more steps added.

Andrea Tamburrini

no nothing! Also NDI-sdk is installed

dotsimulate

When you run the install NDI/Spout, are there any errors or red text in the cmd window? Also make sure to NDI-sdk installed.

Andrea Tamburrini

Hi guys! Thanks @DotSimulate for your work! I still had an issue: after the settings and then clicking on pulse to start stream my cmd don't find any NDI device. In the Setup tab I did pulse on the Install NDI/Spout req only. How can I fix it?

And on another computer I keep getting errors about python not found although I installed the latest and referenced in Path in TouchDesigner.

Why are most of the generated pictures soft focus? compared to let's say ComfyUI or Automatic1111

dotsimulate

Adjusting the step sliders can help as well as smoothing out / simplifying the input. The model doesn't have any temporal understanding which makes it tricky and very prone to flicker.

Tommy Etkin

Is there a way to slow it down or reduce the sensitivity so it's more consistent over time?

dotsimulate

currently only working on windows + nvidia graphics. Hopefully would like to have a non local option at some point, but need to figure out best way to do that.

dotsimulate

so all of the python stuff for streamdiffusion is external to the python version built into TD. You need to download python 3.10.9 to your PC and add it to PATH. It will not effect the TD python build. also, DM or discord is better for sending screenshots or long error messages if its possible to send over there.

dotsimulate

Hey. I believe that this is an issue with the model path you are providing. There are a few others that have run into this as well and there might be info in troubleshoot-central help forum thing on my discord server. Feel free to post in there or shoot me a message over there if possible. Patreon works as well, but doesn't support screenshots so its a bit hard.

Hi, I'm having this issue and I haven't been able to run it. This is the error on the CMD: D:\StreamDiffusion\StreamDiffusion\venv\lib\site-packages\diffusers\configuration_utils.py:135: FutureWarning: Accessing config attribute `requires_safety_checker` directly via 'StableDiffusionPipeline' object attribute is deprecated. Please access 'requires_safety_checker' over 'StableDiffusionPipeline's config object instead, e.g. 'scheduler.config.requires_safety_checker'. deprecate("direct config name access", "1.0.0", deprecation_message, standard_warn=False) Process Process-4: Traceback (most recent call last): File "D:\StreamDiffusion\StreamDiffusion\ndi-stream\..\utils\wrapper.py", line 417, in _load_model etc, etc finishes with NotImplementedError: Cannot copy out of meta tensor; no data! what should I do? thanks!🙏

dotsimulate

I think you would get a little bit more speed with a 4080. I am not sure how to set which gpu the process runs on, but would assume it is running on your main graphics card. That is something to look into about multiple cards. as far as speed, installing and running with tensorrt acceleration mode will improve fps. there is information about installing it oon the github. It is a bit of a tricky installation but there are a few issues about it as well.

Awesome! I totally missed that toggle! Also, do you have an ideas on improving the FPS? I'm on a 13900k with 128gb ram with a 3080 and 4080 but only getting about 6fps. Perhaps more a StreamDiffusion setting but just checking if you have any ideas.

dotsimulate

Oh, amazing ! I can fix that in next version. For now, in the Setup page, you can manually turn Off the Streamactive toggle at any point if it thinks it is on when it is not.

This is so freaking awesome! But... I think i found a little bug. I saved and closed my project while StreamDiffusion was still running. When I opened it back up this morning, the operator still thought SD was running and I couldn't start it from the operator.

dotsimulate

You should be good to download and set the full path.

Faraz Kh

Sorry for so many questions. I am new to the SD ecosystem. For example, I found this lora and the base model is 1.5: https://civitai.com/models/222107/sculpture I should be able to simply download the safetensor file and set the lora path to the download, right? Do I also need to move the lora to the lora folder of the models in the StreamDiffusion repo?

dotsimulate

I have had good luck with all 1.5 Loras as long as I give the full file path similar to loading a local 1.5 model. I still have all my Loras / SD checkpoints in my ComfyUI/models folder and they seem to load.

Faraz Kh

Thank you for all the quick support! You are amazing. Have you tried with any lora? Can you please post a sample link?

dotsimulate

Due to the way StreamDiffusion offloads the VAE to the CPU, it tends to be quite high on CPU usage. VRAM usage remains to be pretty low, with the distilled diffusion model taking only a few gigs of the gpu. I do believe this is an opportunity for optimization.

dotsimulate

Hey Faraz. SDXL turbo is not supported yet, but hopefully soon it will be added by devs of StreamDIffusion. currently only 1.5 and 2.1 (sd-turbo) will work. Lora models should load from the Lora parameter page, and you should see a line that clearly says the weights are loaded and the path you specified, as the second to last line in the CMD right before the stream starts.

Faraz Kh

Have you tried with Lora option? I tried installing a couple that are compatible with SD-1.5 but didn't appear to work. Do you have any working options?

I tried different models and seems it only works with the sd_turbo based on 1.5. And if you get it from the modelID, otherwise says "no data"

Faraz Kh

Is it possible to support SDXL-turbo in addition to SD-turbo?

Hi. Thank you for this! However, I have some questions: I have an A5000 with 24gb, and a Xeon 8-core 3.5Ghz. I noticed that the VRAM used is only 5gb VRAM, and the CPU is maxing at 80%. I cannot install xfromers nor tensorrt for some reason ( i guess i have to install them in the env?. Seems as if the system is only using the CPU somehow. I can get only 10fps at 512x512. Do you have any tips on how to use the full VRAM? Thanks again

Cor Langerak

Would this also work on Mac M1? I am not able to getting up and running after looking at you tutorial... great project though, would love to try it out