Lyell Hintz patreon

StreamDiffusionTD 0.1.11 🚀🎮🖼️ [ Changelog]

🕑 Added 2024-05-16 04:49:31 +0000 UTC

Comments

Raja Singa

help me solve this problem, I'm quite frustrated looking for this problem : from huggingface_hub import HfFolder, cached_download, hf_hub_download, model_info ImportError: cannot import name 'cached_download' from 'huggingface_hub' (D:\StreamDiffusion\venv\lib\site-packages\huggingface_hub\__init__.py) [process exited with code 1 (0x00000001)] You can now close this terminal with Ctrl+D, or press Enter to restart.

Rotem Mor

Hey! Couldnt get the TensorRT to work, and can't find the Discord for support :)

Ismael Abrego

Hi, I've been experimenting with this this Tox for StreamDiffusion and really like it; except that the images that I'm getting are far from great (txt to img), no matter how well defined and detailed is the prompt that I am using. SD Turbo model works best that others but is still far from good images. Any advice?

wang haiqiang

正常工作了，谢谢。

Pixadoo Visuals

Its running perfectly, can anyone guide me how to use it on GPU. thank you

fergus laidlaw

absolutely STOKED to have controlnet added thank you so much!

Philippe Hughes

Hello DotSimulate. Thanks for the GREAT tool. Sadly I cannot run it. I have a Razer laptop with an RTX3080 Ti . Do not know if it's powerfull enough . Anyways the error I get when pulsing the "Start Stream" is ModuleNotFoundError: No module named 'diffuser' in the cmd window. (ps: it opens and closes quickly so I did a screen recording to read the error. ) Any clue on what this is ? I've searched this thread but not found any info. Thanks in advance

Leon

Same thing for me

Youssef El Idrissi

hey! I have a trouble running it. I followed the step-by-step tutorial, but it doesn't seem to work. Where can I tell you about the error? First it's "A matching Triton is not available, some optimizations will not be enabled. Error caught was: No module named 'triton'" and then while running the model ESRGAN (that work normally fine in automatic 1111), it shows me this exception 'OSError: Error no file named model_index.json found in directory D:\ARTISTIC PROJECTS\2024\Stable Diffusion AI\stable-diffusion-webui\models\ESRGAN.' & "UnboundLocalError: local variable 'pipe' referenced before assignment" Do you have any idea how to fix it? Thank you

Gus

working great but "ModuleNotFoundError: No module named 'polygraphy' Acceleration has failed. Falling back to normal mode" what did I not install ?.

Wu shuyang

I'm currently having the same problem as you, and it worked normally before.

Cheng Yuan Wang

Hi, your works are awesome! Can I get the discord link? Thank you very much

삭 이

I need a tutorial on how to change the model ID using the Hugging Face or Civitai website

Yones R

Such a great tool. Are there any resources on how to run completely local, no internet? I have everything running ok I think, max 16fps on 3090 with a single Step Schedule. Should be more, ya? Very much appreciate any leads on these.

Mark

Hey loving this tool thankyou for making. I'm having trouble loading a controlnet model. I get the following error \Users\markw\OneDrive\Documents\Touch\StreamDiffusion\streamdiffusionTD\pipeline_td.py", line 499, in predict_x0_batch control_image = torch.cat((control_image, self.control_image_buffer), dim=0) TypeError: expected Tensor as element 1 in argument 0, but got NoneType

arunr6600 .

My issue was with the controlnet model and SD1.5. The lllyasviel/control_v11f1p_sd15_depth, didn't work for me. Instead I used https://huggingface.co/webui/ControlNet-modules-safetensors/blob/main/control_depth-fp16.safetensors and now its streaming.

publicoprivado

Stuck in the same place, I was using sd-Turbo seems to be based on 2.1 so I thought to try the correct canny processor but no success. Is there anything I am missing?https://www.patreon.com/posts/streamdiffusiont-104339324

CICI

你好，我点击start stream pulse之后，一直显示Loading pipeline components...: 20%|██████████▍ | 1/5 [00:00<00:02, 1.75it/s]和RuntimeError: Numpy is not available和UnboundLocalError: local variable 'pipe' referenced before assignment，请问怎么解决？

Arvol Dingess

Hello, Thank you for making this tool <3 In terms of processing power and hardware, do you know what hardware we should buy to make this more efficient? Does a gaming GPU outperform a Quadro? (OpenCL vs OpenGL) Or would an AI card be better? Would having more CPU power and ram vs GPU help? We're getting about 15-20 fps on a 4090 and I'd love to buy something that would give us better fps, but we don't know WHAT we need to buy for better performance? Any insight would be greatly appreciated. :)

Benlloyd Goldstein

could beto do with egg-info deprecation ?

Benlloyd Goldstein

i have cuda 11.8 , but when i run install i get this error : ------------------------------------- 991.4/991.4 kB 30.7 MB/s eta 0:00:00 Collecting nvidia-cudnn-cu116 (from nvidia-cudnn-cu11->tensorrt_libs==9.0.1.post11.dev4) Downloading nvidia-cudnn-cu116-0.0.1.dev5.tar.gz (4.4 kB) Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'error' error: subprocess-exited-with-error python setup.py egg_info did not run successfully. exit code: 1 [17 lines of output] Traceback (most recent call last): File "", line 2, in File "", line 34, in File "C:\Users\benll\AppData\Local\Temp\pip-install-ugh06s25\nvidia-cudnn-cu116_5b65ba7b510e46268f7994b13fc547a8\setup.py", line 152, in raise RuntimeError(open("ERROR.txt", "r").read()) RuntimeError: ########################################################################################### The package you are trying to install is only a placeholder project on PyPI.org repository. This package is hosted on NVIDIA Python Package Index. This package can be installed as: ``` $ pip install nvidia-pyindex $ pip install nvidia-cudnn-cu116 ``` ########################################################################################### [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed Encountered error while generating package metadata. See above for output.

Matias Gomez De Falcis

Dot it could be that the Guidance Scale parameter of Stream Setting 1, when it exceeds the value 1 in this version, it breaks?

wang haiqiang

可以运行了。谢谢。

dotsimulate

Hey. Try again with the operator in this post (I made a tiny change and reuploaded just now.)

wang haiqiang

我在TD里运行 view installlation guide ,出现黑屏。

Scott Hedstrom

FYI when you have the correct versions of cuda 11.8, this sorta just works using the install tensorrt button, then selecting tensorrt from the acceleration dropdown on Stream Settings 1.

Scott Hedstrom

Ok after a bit of debugging and experiments, it looks like cuda 11.8 is the required vs 12. Got everything working at 80fps! TensorRT is the juice. Thanks so much @dotsimulate !

Scott Hedstrom

ok, so after a bit of digging, looks like tensorrt.icudaengine version 10.0.1 doesnt have num_bindings, it has num_io_tensors... 8.6.1 seemed to have it, but I can't get it to let me install via pip

Scott Hedstrom

Basically the error is saying python doesnt have access to anything called pytorch_lightning and the software is trying to access it. I would try this: Go to the venv folder in terminal, type `source Scripts/activate` (without the `) This will make sure you are on the venv for the project... Then type `pip install pytorch_lighting` Try running it again...this will potentially lead down a list of similar errors and solutions, but will get ya started.

Scott Hedstrom

ok, finally got the venv installs working, had to add a few things but now when I run w/ tensorrt it says: tensorrt_bindings.tensorrt.ICudaEngine object has no attribute 'num_bindings' not sure where else to dig here as there isnt much coming back when I search for all that

Benlloyd Goldstein

a video on correct tensort implementation would be amazing!

Scott Hedstrom

Thanks for the awesome work and goodies!

Scott Hedstrom

For some reason I can't get tensorRT to boot...it says "no module named 'polygraphy'" in streamdiffusionTD/wrapper_td.py -> _load_model tried installing polygraphy myself in the venv, but no luck.

arunr6600 .

Hi, I am getting this error. Any fix? ...Loading ControlNet model: controlnet_sd15_depth_f16.safetensors Failed to load ControlNet model E:/Stable Diffusion/stable-diffusion-webui/extensions/sd-webui-controlnet/models/controlnet_sd15_depth_f16.safetensors using from Huggingface (from_pretrained) due to: It looks like the config file at 'E:/Stable Diffusion/stable-diffusion-webui/extensions/sd-webui-controlnet/models/controlnet_sd15_depth_f16.safetensors' is not a valid JSON file. Failed to load ControlNet model from local directory (from_single_file) E:/Stable Diffusion/stable-diffusion-webui/extensions/sd-webui-controlnet/models/controlnet_sd15_depth_f16.safetensors due to: 'time_embed.0.weight' Thank you

Benlloyd Goldstein

how would you go about applying motion loras to this ?

Gordon kummel

i havent had luck with XL either but control net should work dotsimulate was demoing it in his livestream interview for civitai which is on youtube, probably a matter of loading the right models.

Jasmine Rose Federer

Believe it only works with 1.5 models for now but please correct me if I'm wrong - I would also love to get XL working

Lucas

Hello there! I tried installing/reinstalling/deleting venv folder/ installing different version of python/... but I keep getting the following error: Failed to load SD model from both local and Hugging Face due to: No module named 'pytorch_lightning' Any idea on how to resolve this? Thanks a lot!

Latent_

I had to delete the venv folder but after that updating python and installing the dependencies from scratch it work!

aviv oz

hey i need help i can only use openai/turbo-sd model any other 1.5 or xl turbo xl etc. cant work? also non of my control net models are being accepted

Stella Speziali

Thank you for your reply. Why give the possibility to load a specific ControlNet model? I thought loading a specific controlNet model (eg depth, canny, normal, segmentation etc) in the controlNet tab of streamDiffusion would control the image generation like in Stable Diffusion 1111. But is the preprocessed image that affects the overall output. I get it now. Is there a way to generate the depth from a video in source in touchdesigner at 30fps? Is it even possible? I get generating the edge or normal in touchdeisgner but the depth comes only from the render node if the scene is 3D, right?

dotsimulate

The result from the model is always going to be the output from the diffusion process. The controlnet input must go in already preprocessed (in depth map form if using depth, or with edge lines if using softedge, etc).

Stella Speziali

Hello! Is there a way to see the result of the control net? as an output

dotsimulate

Link is available via any of the operators I’ve released in the about page (about me>discord)

Jasmine Rose Federer

heyy - could you please reshare the discord link? thanks

JuanJo Casallas Contreras

hey! I´m havin issues running it, i know CUDA is necessary but I have a nvidia 330 so i don´t know if the issues come from it

Arthur Deleu

"Failed to build tensorrt ERROR: Could not build wheels for tensorrt, which is required to install pyproject.toml-based projects", seems like a common error, would you have any workaround?

Arthur Deleu

Hey man, having issues with tensorrt...

kit lee

Hello, Thank you for providing amazing TOX! I would like to know how to change the output resolution? I try to get into [StreamDiffusion TD component] and change the [ndiout] but still doesn't work.

Lucas Chen

Thank you!

Joseph Armen

Hello and thank you! I've installed everything successfully and get a stream info output, but my StreamDiffusionTD is blank. The only error I see is: StreamDiffusion\streamdiffusionTD\main_sdtd.py:393: RuntimeWarning: invalid value encountered in cast processed_np = (processed_np * 255).astype(np.uint8)

Luis Guillermo Helfer

Hello Would you help me install LORA on your latest Tox from streamdiffusion? I get the following error " ERROR loading LoRA Models: Cannot load because down.weight expected shape tensor(..., device='meta', size=(32, 320)), but got torch.Size([32, 320, 1, 1]). If you want to instead overwrite randomly initialized weights, please make sure to pass both `low_cpu_mem_usage=False` and `ignore_mismatched_sizes=True`. For more information, see also: https://github.com/huggingface/diffusers/issues/1619#issuecomment-1345604389 as an example. "

Riccardo Fasoli

i'm using it with without any problem so i'd say you messed up something during the installation :)

We_ Make

thanks amazing work

orphée Cat

Raul Leal

I just downloaded the tox and followed the installation but when I try to start the stream the console opens and closes immediately. I see an error "no module named torchvision"

Dilip Ramírez

Any ideas on when are we getting NDI support back?

dotsimulate

it is the about section of each of my operators and I think in about me as well.

Noam F.

it worked !!!! Thank you!!

dotsimulate

this is because of model mismatch. you will want to use 1.5 controlnet model and 1.5 sd main moderl

dotsimulate

I should have a bit clearer info on how to dm or contact for support. I am planning on a video soon for a controlnet update. sd-turbo won't work with controlnet currently. but hoping to get that working soon as well. You need to use 1.5 main sd model and a 1.5 controlnet model at the moment.

Anton Thomsen

silly question - where is the link for discord? Am i blind?

GMM

I solved the problem by installing Python 3.10 64-bit instead of 32-bit. https://www.python.org/ftp/python/3.10.9/python-3.10.9-amd64.exe

Daniel Lozano

Same her

Jeremy Sanders

same issue here

Felix Reiterer

I have exactly the same issue! Did you manage to solve it yet?

Aratz Hernandez

Hi ! I saw your civitai stream .. great stuff thanks! Could you share with me the discord link?

Guglielmo Guarnera

I often touch 40fps but it doesn't seem to be using the gpu a 20 percent is using it parsec to remote the machine. when i install tensorcore it gives error but then if out of venv i install the correct version it seems to go. it gives me an error if i turn on cfg type it doesn't seem to work any controlnet so I wonder, did you put the page in without explaining which ones you can use because they don't work yet? I would need help but I didn't understand if I have to send a private message to Dotsimulate, if I have to go to discord, if I have to post every question publicly, I would really need feedback but I am not familiar with the rules. https://imgur.com/a/dotsimulate-streamdiffusion-krd-test-v1-1-4xw3Xvr

Alex England

I thought when I subscribed to your Patreon I was going to be in and out just to download the operator, but I've stayed for the constant updates. You really work hard on this and I love the new features in each update!

Andrea Tamburrini

Hi! I have this issue when i try to install tensorT: RuntimeError: ########################################################################################### The package you are trying to install is only a placeholder project on PyPI.org repository. This package is hosted on NVIDIA Python Package Index. This package can be installed as: ``` $ pip install nvidia-pyindex $ pip install nvidia-cudnn-cu116 ``` ###########################################################################################

Noam F.

Hi! First of all, thanks for your work!! I have a few questions. The first one: When I press the "Install (venv + all req)" button, I encounter a series of errors throughout the installation: Building wheel for n-vidia-pyindex (project.toml) ... error error: subprocess-exited-with-error Building wheel for n-vidia-pyindex (project.toml) did not run successfully. exit code: 1 /88 lines of output RuntimeError Traceback (most recent call last): File "" , line 1, in ModuleNotFoundError: No module named 'pip' ModuleNotFoundError: No module named 'setuptools' note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for nvidia-pyindex Failed to build nvidia-pyindex ERROR: Could not build wheels for nvidia-pyindex, which is required to install pyproject.toml-based projects Installing dependencies with pip from the activated virtual environment... Requirement already satisfied: pip in c:\users\flore\downloads\fran\x\streamdiffusion\venv\lib\site-packages (24.0) Looking in indexes: https://download.pytorch.org/whl/cu118 ERROR: Could not find a version that satisfies the requirement torch==2.1.0 (from versions: 2.2.0+cu118, 2.2.1+cu118, 2.2.2+cu118, 2.3.0+cu118) ERROR: No matching distribution found for torch==2.1.0 The second is that (despite the errors I get at the beginning) when I click on "Start Stream", a window opens with some text that I can't read because it closes immediately. In "Log Message" it says "started streaming using CMD" and in "stream Info" it says "not streaming". I hope you can help me. best regards!

Pieter van Dijk

Hi, I managed to load a Controlnet model: "Successfully loaded ControlNet model from local directory". But creating a model dropdown does not work - only a single file. When starting I get many error messages about specific files, under the title Process-3. And ultimately also a RuntimeError: "mat1 and mat2 shapes cannot by multiplied (154x2048 and 768x320)". I don't know exactly what to do with input 2 either. Can you show a working example? Kind regards, Pieter

Kelly Warner

This is fantastic! Where can I learn about installing Controlnet specifically for StreamDiffusionTD?

JJ JV

Ouuu, this is a big one here! Appreciate your hard work my dude 🙏🏽

João Silva

Everything works fine, the fps count its around 0,5 and by pc specs are a Intel Core i5-10300H, have 8g of ram and have a rtx 1650, but i have a problem in wich the screen becomes black or rainbow color and the model does not load up, Does someone knows how to fix it?

Rapid Carrot

I am using cuda 12.1 . I had a cuda version error which is fixed with dotsimulate's pull request on the streamdiffusion GitHub page. I also had a no module found error for cuda and onnxruntime , which can be fixed by activating the venv ( pulse in streamdiffusiontd didn't work for me ) and pip install cuda-python and onnxruntime. I don't think I had other errors on rttensors install. Thanks dotsimulate btw , nice work.