Introduction

With LLMs, Prompt Engineering has been really crazy. I've been developing Miteiru with it for a while now.

I developed many features with it. Saved so many hours of porting the app into only minutes. I literally learned Cantonese because I did it myself. I'm so proud of myself.

Learning Japanese has never been this easy. I developed Miteiru to support my language learning, and it's become a tool I genuinely enjoy using for HOURS.

Other than that, I've been editing shorts using AI as well. By AI, I mean the process is done after so many iterations.

I will introduce you to MCP, which stands for Model Context Protocol. So there was this app called VectAPI, which is an MCP for CapCut. I've been trying to develop an app to help me edit videos, as it's the only thing that hinders my way to becoming a content creator. Editing takes too much of my time. Especially good editing.

First of all, I'd like to put in honorable mentions for having a good and reliable provider for a region like Hong Kong — which usually can't access LLMs.

• Higgsfield
• ChatboxAI
• OpenRouter
• Cartesia

https://www.youtube.com/@mikiraIRL/shorts You can take a look at the last 8 videos I posted here. AI edited all of them.

Anyway, the idea is that initially, there was an API based video editor that had things like:

void addImage(image, x, y, ..)
void addVideo(video, x, y, ..)

AI is good at programming.

AI is kinda good at text creativity

Is AI visually creative? No, that's why they always fail the SVG test.

But if you think about this, if I were to edit using this MCP, I'd like to give the creative direction to AI. AI needs to be able to construct all these commands while having to think about what to insert. But how does AI think creatively if this is the case? Won't that burn many of my tokens?

Then, I mapped the problems that I have. But before that, meet Necronival. I used his shorts as my goal, target, and model. Whatever you call it.

The Video Editor Journey

The Birth of AI Skills

The other day, I talked to Ammar, my Syrian friend. This is what I told him. "Hey Ammar, do you know why competitive programming is important? Do you know what distinguishes us, competitive programmer and vibe coder?" In the context that both of us are ex-competitive programmers. I told him this. When we are querying AIs, to sort these strings:

Banana
Avocado
Cherry
Mango
Durian
Apple

We don't. Instead, we ask AI to create a program that sorts strings, and we input it. Then I thought. What if we want to sort it by color? Use AI?

The answer is no. We ask AI to incrementally create a map of colors.

{
    "Red" : ["Cherry"],
    "Yellow" : ["Durian", "Banana"]
    ...
}

How many tokens would we burn if we were to ask AI to solve a question? Instead of asking it to create a machine that can solve the question?

https://claude.com/blog/skills. In October 2025 (less than a year ago), Anthropic innovated and created the SKILL.md for AI agent skills. They need something stable, something constrainable. Something to make their answer scoped, yet non-deterministic.

Amazing invention by Claude.

Back to Video Editor

Then I thought, okay. I need to make an AI skill. So I planned something called planning. I came up with these three types of edits.

• A-roll: Where the VTuber talks, with a background, and can add images, stickers, and reaction GIFs.
• B-roll: A video that explains why the VTuber is doing a voice-over.
• B-roll type 2: Blank background where the sticker plays to describe the scene.

Good enough? No, it's super expensive and slow. The AI needs to think about when to insert the images, how to get the images, and how to split the scene. AND the AI loses context because it keeps writing and calling the wrong API for the MCP.

JSON Schema

An amazing idea came to us. I queried: I need something that can make everything stable and expected. I asked him to create a JSON. The rules are simple, just create an example JSON, and code the polish.py or what we call the main video editor entry program as that. After 3-4 hours of video editing back and forth. The video is done!

But another problem comes: consistency was... a problem.

Consistency

I came to edit another video, but this time it relies too much on the old polish.py. It keeps peeking and copying everything from the old project. The edit is boring, too; the placement of images and stickers is inaccurate. I need something better.

JSON Validation comes as a savior here. I defined a proper schema.

Perfect, I thought. After this, the workflow improved a lot. AI is now improving and has better context because of this scene schema. They can think about the editing and the programming separately.

But another problem comes. Every time I create another edit. AI always forgets how to edit interestingly. The solution is a markdown called technique.MD, I put many video editing techniques so that AI could learn about editing techniques that I taught it for every edit iteration.

The Hallucination

I decided to delete older projects to make sure it's still consistent.

It hallucinates. It starts to create a thought that everything is Rujak, which is the title of my first project. Even though I already banished the project folder.

I asked it to De-Rujak-ify my project. It was already 3 AM when I was doing the edit. I was going crazy already.

Just so you know, rujak is a fruit salad dish with peanut sauce. I started to lose my sanity.

wait_why_does_cursor_keep_referencing_rujak.mp4

I stopped for a while. WTF am I doing? We can just generalize the polish.py. Stop AI from coding, and let it edit only using the JSON.

This idea works perfectly. After spending about !{money:70:usd} worth of tokens. I stopped and thought. I need a better way to maintain this.

• Add a coverage checker.
• Ask it to write unit tests until it reaches 95%
• Add a better validator for the JSON.
• Add a better communication between the polish.py to the capcut_cli.py
  • Added a greedy algorithm to reduce the amount of tracks (when you put images, initially it will just create a track for one media)
• Placement is tricky here. I added a verifier for it not to put 2 images of where one covers the other. I asked him to do a line sweep, of when there's a start visual event. It will check the current active visual, and do a rectangle area collission count. It will warn the edit if the placement is bad.
• Media can vary in size, create a script to probe a media, and do the math for the placement and scaling. When placing a media, extract the subtitle word per word, and time it.
• Preparing media: trim and normalize all sound effects and bgm.
• Create a fetcher to irasutoya and wikimedia common.

This is amazing. I make the "unstable" - edit, into something that is quantifiable!

Still, though, the view number is very sad. So I guess it all comes back to content, captivating moments, and creativity.