Actualiser README.md
This commit is contained in:
parent
1855c861c3
commit
ad917ff790
1 changed files with 140 additions and 15 deletions
155
README.md
155
README.md
|
|
@ -1,10 +1,9 @@
|
|||
# VideoStory
|
||||
|
||||
C'est une expérience avec Stable Diffusion 1.5, Llama 3 et du TTS qui permet la génération de "vidéo" (suite d'images) avec une histoire narrée. Il supporte potentiellement SD Video.
|
||||
Ce programme n'a pas vraiment pour vocation d'être utilisé.
|
||||
This is an experiment with Stable Diffusion 1.5, Llama 3, and TTS that allows for the generation of "video" (a sequence of images) with a narrated story. It potentially supports SD Video. This program is not really intended for practical use.
|
||||
|
||||
## Librairie
|
||||
Voici les dépendances:
|
||||
## Library
|
||||
Here are the dependencies:
|
||||
```
|
||||
re
|
||||
llama_cpp
|
||||
|
|
@ -15,19 +14,44 @@ os
|
|||
moviepy
|
||||
```
|
||||
|
||||
## Utilisation
|
||||
Dans le fichier `main.py`, ajoutez dans l'appel de main() le prompt.
|
||||
`SYSTEMPROMPTT` est le prompt système de Llama.
|
||||
`SDBAD` est le prompt négatif de Stable Diffusion.
|
||||
`SYSTEMPROMPTI` est le prompt système de Llama pour Stable Diffusion.
|
||||
## Usage
|
||||
In the `main.py` file, add the prompt in the call to main().
|
||||
`SYSTEMPROMPTT` is the system prompt for Llama.
|
||||
`SDBAD` is the negative prompt for Stable Diffusion.
|
||||
`SYSTEMPROMPTI` is the system prompt for Llama for Stable Diffusion.
|
||||
|
||||
`promptTtoI.txt` et `promptUtoT.txt` sont respectivement le prompt système de Stable Diffusion et celui de Llama.
|
||||
`promptTtoI.txt` and `promptUtoT.txt` are respectively the system prompt for Stable Diffusion and that for Llama.
|
||||
|
||||
Dans le fichier `gen.py`, dans les fonctions `loadllama()`, `loadtts()` et `loadsdxl()` vous devez ajouter vos models (fichiers locaux).
|
||||
In the `gen.py` file, in the functions `loadllama()`, `loadtts()`, and `loadsdxl()`, you need to add your models (local files).
|
||||
|
||||
Le programme se lance avec `main.py`.
|
||||
The program is launched with `main.py`.
|
||||
|
||||
## Fonctionnement
|
||||
## Why does this project suck
|
||||
|
||||
### Context Limitation
|
||||
|
||||
AIs are limited by the size of their context. Too much, and the AI goes crazy; not enough, and the output is worse than the usual trash. It is impossible to create a "real" long story fully with AI because:
|
||||
- If you generate it in one go, as in V1, the AI will create a pretty short story, and the further it goes, the more it becomes incoherent and repetitive.
|
||||
- If you generate it in multiple parts, as in V2, the story might be more coherent and better in the long term, but the overall quality will be lesser because when you rebuild the whole story into one big text, there are a lot of "artifacts."
|
||||
|
||||
### Self-Biasing Limitation
|
||||
|
||||
AIs self-bias themselves all the time because of their context. If there weren't any context, there wouldn't be any bias, but also no output. AI self-biasing is the same thing as human biasing but on a much larger scale. Everything biases AI toward its final output. The proof is that if you prompt the AI to generate a story about a cat, it will generate a story about a cat. However, this is also an issue because every word in its context is taken into account to generate the final output, along with all the "artifacts" it created along the way. For one artifact, ten more are generated, and the output rapidly becomes garbage. This is due to the fact that AIs are probabilistic machines, i.e., useless for tasks that require more than just probabilities.
|
||||
|
||||
This self-bias is really visible in V2 because, at each pass, the AI's context is cut and modified. This means that instead of having one AI with one context and one bias, we have multiple versions of the AI with different biases. This creates a LOT of artifacts, as they all have different "state of mind" and "goal." You could visualize the AI's bias as a vector made of all the tokens/n-grams in its context. While V1 only uses one context, with one vector in one direction, V2 uses multiple contexts with multiple vectors all pointing in "kind of the same direction" but still diverging.
|
||||
|
||||
### Conclusion
|
||||
|
||||
To correct the issue, you would need to write the text yourself multiple times with various small wording variations and then train the AI with them. Then you would have a well-written and longer story, and V2's bias would probably be better (i.e., pointing more in the same direction).
|
||||
Soo yeah, shocker: writing your own story is better than using an AI to generate them, even with the most sophisticated methods. The same goes for image and audio generation.
|
||||
|
||||
## Output exemple
|
||||
|
||||
https://uwo.nya.pub/forge/Joachim/VideoStory/src/branch/main/out.mp4
|
||||
|
||||
## Flow charts
|
||||
|
||||
### V1
|
||||
```mermaid
|
||||
flowchart TD;
|
||||
sd{{"Stable Diffusion"}}
|
||||
|
|
@ -67,6 +91,107 @@ img2 --> vd;
|
|||
img3 --> vd;
|
||||
```
|
||||
|
||||
## Exemple de sortie
|
||||
### V2 (Unpublished)
|
||||
|
||||
https://uwo.nya.pub/forge/Joachim/VideoStory/src/branch/main/out.mp4
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
state "Part 1" as p1
|
||||
state "Part 2" as p2
|
||||
state "Part N" as pN
|
||||
state "Gen Story p1" as Gp1
|
||||
state "Gen Story p2" as Gp2
|
||||
state "Gen Story pN" as GpN
|
||||
state "Summary 1" as S1
|
||||
state "Summary 2" as S2
|
||||
state "Summary N" as SN
|
||||
state "Prompt 1" as pt1
|
||||
state "Prompt 2" as pt2
|
||||
state "Prompt N" as ptN
|
||||
state "Gen illustration 1" as it1
|
||||
state "Gen illustration 2" as it2
|
||||
state "Gen illustration N" as itN
|
||||
state "Gen TTS 1" as tt1
|
||||
state "Gen TTS 2" as tt2
|
||||
state "Gen TTS N" as ttN
|
||||
state "Subtitle 1" as sub1
|
||||
state "Subtitle 2" as sub2
|
||||
state "Subtitle N" as subN
|
||||
state "Video part 1" as v1
|
||||
state "Video part 2" as v2
|
||||
state "Video part N" as vN
|
||||
state "Video Final" as vf
|
||||
World --> Base
|
||||
Description --> Base
|
||||
Name --> Base
|
||||
Base --> Master
|
||||
Master --> Player : Until number x of max interations is reached
|
||||
Player --> Master
|
||||
|
||||
Logs --> p1
|
||||
Logs --> p2
|
||||
Logs --> pN
|
||||
|
||||
p1 --> Gp1
|
||||
p2 --> Gp2
|
||||
pN --> GpN
|
||||
|
||||
Master --> Logs
|
||||
Player --> Logs
|
||||
|
||||
Gp1 --> S1
|
||||
Gp2 --> S2
|
||||
GpN --> SN
|
||||
|
||||
S1 --> pt1
|
||||
S2 --> pt2
|
||||
SN --> ptN
|
||||
|
||||
pt1 --> it1
|
||||
pt2 --> it2
|
||||
ptN --> itN
|
||||
|
||||
Gp1 --> tt1
|
||||
Gp2 --> tt2
|
||||
GpN --> ttN
|
||||
|
||||
Gp1 --> sub1
|
||||
Gp2 --> sub2
|
||||
GpN --> subN
|
||||
|
||||
it1 --> v1
|
||||
tt1 --> v1
|
||||
sub1 --> v1
|
||||
it2 --> v2
|
||||
tt2 --> v2
|
||||
sub2 --> v2
|
||||
itN --> vN
|
||||
ttN --> vN
|
||||
subN --> vN
|
||||
|
||||
v1 --> vf
|
||||
v2 --> vf
|
||||
vN --> vf
|
||||
|
||||
World: World name
|
||||
Description: World description/rules
|
||||
Name: Main actor's name
|
||||
Logs: Roleplay's logs
|
||||
Master: AI leading the game
|
||||
Player: AI choosing next state, with only current state context
|
||||
p1: Part 1 of logs
|
||||
p2: Part 2 of logs
|
||||
pN: Part N of logs
|
||||
Gp1: Story generated with Part 1
|
||||
Gp2: Story generated with Part 2
|
||||
GpN: Story generated with Part N
|
||||
Base: Base prompt for leading AI
|
||||
S1: Story summary
|
||||
S2: Story summary
|
||||
SN: Story summary
|
||||
sub1: Video's subtitles
|
||||
sub2: Video's subtitles
|
||||
subN: Video's subtitles
|
||||
pt1: Gen SD prompt with simplified story
|
||||
pt2: Gen SD prompt with simplified story
|
||||
ptN: Gen SD prompt with simplified story
|
||||
```
|
||||
Loading…
Add table
Reference in a new issue