SyLC : MVC / frame-packing playback on Windows, built on edge264

[Symphoenix]

I've been reading this forum for years, mostly the Xvid and HDDVD threads. Close to everything I know about encoding, I learned here, including the fact that most of what I assumed was hard, wasn't, and most of what I assumed was easy, was. So I'd rather post this as "here is what I built, please tear into it" than as an announcement.

What it does


H.264 MVC (Stereo High, profile 128) from MKV, M2TS and SSIF
Frame packing output - 1920x2205 @ 24Hz on a custom resolution
SBS and TAB for standard displays
Direct3D 11 renderer, HDR (PQ) preserved
PGS subtitles, streamed
Real-time 2D-to-3D synthesis (depth estimation + GPU warp), optional
MV-HEVC export
Stereo casting to a Quest headset over USB or Wi-Fi
Windows only


What isn't mine

The decoder. MVC is standard H.264 Annex H and edge264 (Thibault Raffaillac, BSD 3-clause) already handles it natively. I did not write an MVC decoder, and I want that stated plainly up front, an earlier version of my own README overstated this and I'd rather correct it here than have it corrected for me.

My one contribution to edge264 was a fix (i did not post it): a fallback path in the quantization scaling matrix handling that produced horizontal banding on some streams (Gravity...).

Also not mine: libmpv (audio), GPAC and x265 as external tools, ONNX Runtime for the depth model.

What was actually hard

Not the decoding — the demuxing and the sync.


Interleaving base-view and dependent-view NAL units in the correct access-unit order, and keeping prefix NALs (type 14) contiguous with the base-view slice they belong to.
Matching the two views by PTS, which means normalising the PTS offset first: Blu-ray streams routinely start around 11s, MKV at arbitrary offsets. Any sync engine that skips this drifts.
Decoupling presentation from decoding. A ~100ms keyframe decode used to freeze the display once per GOP; a dedicated presenter thread and a ring buffer that absorbs the spike fixed it.
Concurrent file access. Running ffmpeg for thumbnails against the same file as the demuxer was a reliable crash on Windows for weeks before I understood what I was looking at.


If any of that sounds naive to people who solved it years ago, I'd honestly like to hear it.

Known limitations


Frame packing requires creating a 1920x2205 custom resolution by hand in the NVIDIA/AMD control panel. I haven't found a way around this.


This list is short because my test corpus is small, it's essentially my own shelf and one projector. That's the main thing I'd like to fix.

What I'm asking for

Test cases, mostly. Specifically:


Discs or rips that fail to open, desync, or show artefacts, the title and the container matter more than the log at this stage.
Anything with an unusual PTS layout or a dependent view that doesn't follow the pattern I've assumed.
Frame packing on displays and projectors other than mine.


https://github.com/5ymph0en1x/SyLC

Thanks ! Genuinely ! To the people whose posts I read while working this out. I'll take whatever criticism comes.

[Z2697]

Sadly, new displays don't have 3D support at all :(
I use to have one but that's a thing of the past...

Will there be more VR headset support?
I don't have one but my prediction is that any 3D-related thing in the future will happen in VR headset, not flat screen.

[Symphoenix]

3D didn't die, it eloped. It ran off with a projector and never wrote home.

Flat panels dropped it the moment marketing found a shinier word for the box: HDR, then 8K, then "AI". But DLP projectors quietly kept the flame burning : Hisense, Optoma, XGIMI, Valerion, JMGO all still do 3D, mostly because the DLP chip does it almost for free. So the format survives the way old love survives: nobody celebrates it, nobody quite lets go of it either.

Meanwhile Cameron is still out there defending it single-handedly like the last romantic in a world of pragmatists. Over half of Fire and Ash's opening gross was 3D. Not bad for a corpse.

As for headsets, you're probably right, and I find it a little sad. 3D was born as a shared thing: the whole family in ridiculous glasses, someone always sitting on a pair. In a headset it finally becomes perfect and completely solitary. Each eye gets its own screen at last, which is technically everything stereoscopy ever dreamed of… and you experience it alone, with a strap across your face.

If you ever miss the old feeling: a used DLP projector and two pairs of active glasses will bring it all back. The glasses are still ugly. That's part of the romance :)

[Z2697]

I'd prefer polarization tech! :)

[nevcairiel]

Over half of Fire and Ash's opening gross was 3D.

Probably also the only movie that was filmed in 3D in the last 10 years. Everything else is just fake.

[Symphoenix]

Let's try something different...

Real-time 2D -> 3D in SyLC v5.3.1 — the schematics, the measurements, and the one number I had to withdraw

Every film ever made was photographed in a world that had depth. The lens flattened the record; it did not delete the geometry. Occlusion, parallax, perspective, the fall of focus — the evidence survives in every frame. Reading that evidence back out at film cadence, on consumer hardware, while the same GPU decodes the film and draws two output surfaces, is the problem this post describes.

This is not a feature list and it is not a request. It is a statement of what was built, what it measures, and where it is still wrong — including one number I published, could not reproduce, and withdrew. It runs inside a free, open-source player, on any Direct3D 11 GPU. No licence, no activation, no telemetry.
https://github.com/5ymph0en1x/SyLC

1. One model, many surfaces, one hard deadline

Real-time 2D→3D is a scheduling problem before it is a machine-learning problem. Depth Anything V3 infers a depth map and the renderer warps the frame into a stereo pair; the model is the easy part. The answer is due on every frame, forever.

A single process-wide inference service owns the model. The embedded preview, the frame-packed window, both projector eyes and Cast attach to it; one surface holds a short renewable input leader lease and performs the GPU readback, everyone else consumes the same published map on its own device. Attaching a fifth surface costs no extra model work. The worker overlaps two stages deliberately: the GPU infers map N while the CPU runs boundary refinement, temporal fusion and geometry for map N-1, with optical flow farmed to a persistent thread pool.

And one principle above the rest: depth is stabilized in video time, not compute time. The two clocks are kept apart on purpose — a faster or slower GPU changes how often the depth refreshes, never how it behaves. Swapping the card must not change the geometry.

2. The 4K film that converted faster than the 1080p one

I begin with a bug rather than an architecture, because an architecture can be described in advance and a bug cannot.

A 4K HEVC master sustained 24 depth maps per second. The same film in 1080p H.264 managed 19.5. Same engine, same grid — and the 1080p file's GPU inference measured 2.3× faster (9 ms against 21 ms).

A component that is faster inside a system that is slower is never a throughput problem. It is a phase problem.

The depth engine was idle-waiting 45 % of every cycle. Its readback copy was issued but never submitted: a 1080p frame carries so little GPU work that the driver never accumulated enough to flush its own command buffer, so the copy was still unreadable a full frame later and no work was handed over at all — 41.7 ms lost, on 17–22 % of frames. The 4K path's heavier per-frame work flushed implicitly, which is precisely why the harder file never missed.

One Flush() after the copy: stalled hand-offs 436 in 1991 → 1, delivery 19.5 → 24.0 maps/s, cost 0.2 ms of lost command batching. Rollback SYLC_SYNTH3D_FLUSH=0.

3. Temporal stabilization

A raw per-frame depth map shimmers. Three mechanisms hold it still without smearing motion.

Per-pixel adaptive EMA — a moving pixel adapts instantly, a static surface keeps its history:
if (motion) {
mv = clamp(motion[i]);
motion_t = smoothstep(motion_low, motion_high, mv);
al = lerp(alpha_static, alpha_motion, motion_t);
}
ema_[i] = lerp(ema_[i], target, al * confidence);

Outlier snapping — a pixel leaving the established tone range must confirm the same sign across consecutive frames before it may snap, so model noise cannot manufacture a contour. Percentile normalization — depth has no absolute scale, so the map is stretched between tracked low/high percentiles, reserving headroom for the comfort knee.

The depth-cut gate above it is adaptive, and had to be: a fixed residual threshold fired roughly one false cut per second on busy content. A cut now needs the residual to exceed an absolute floor and a multiple of the ambient baseline — itself a video-time EMA of non-cut residuals. Rollback SYLC_SYNTH3D_DEPTHCUT_ADAPT=0.

4. Cuts: three observers, and the difference between a cut and a flash

A depth map is always a frame or two behind the picture it describes, so a shot change is a hard reset. Three independent observers report one: source histogram TV distance, depth residual (affine alignment error against the stabilized map), and a look-ahead scout analysing decoded frames far enough ahead of presentation to see the cut coming.

The subtlety: detecting a cut is not detecting a change. A flash and a hard cut spike the same distance measure, so a detector armed on a bare threshold stays safe against flashes only by keeping its bar high — and a genuine low-contrast cut below that bar is invisible to it. The scout therefore judges by shape, not magnitude: it arms on the exceedance and confirms on the return to calm, which a flash never produces, because the content after a flash is the content from before it.
if (is_cut && !previous_exceeded) arm;
if (armed && calm) publish;

And the advisory was still late. Every cut tore exactly one frame. Detection was never at fault and all temporal state was correctly re-primed — but the advisory carrying the news was refreshed by a 10 Hz timer while frames present at 24 fps, so it could stand frozen for 100 ms, still announcing a future cut on the first frame of a new shot. The renderer therefore re-opened up to 55 % of the disparity budget while filling disocclusions from a plate made entirely of the previous shot's pixels. It is now dead-reckoned against its own age. Rollback SYLC_LOOKAHEAD_DECAY=0.

5. The cyclopean lock — and the number I withdrew

synth3d.hlsl performs inverse warping: for every pixel of the synthesized eye we sample the 2D source through the disparity value, disocclusions filled from a flow-transported temporal background plate.

The v5.3.0 change is the cyclopean lock. A visible chroma contour used to cross its luma-owner threshold independently in each eye; at a 4:2:0 half-resolution phase boundary that could recolour a patch in one view only, while both luma warps stayed perfectly valid. The conjugate eye must now confirm reciprocal source ownership, visibility and the same luma-layer evidence before either chroma lock is admitted.

Now the part I would rather not write, which is the reason I am writing it.

An earlier draft claimed the worst-case chroma residual fell from 12.69 to 1.67 code values. That claim is withdrawn. The topology set behind it was never recorded and does not reproduce from anything in my tree. It is retracted in my own release notes, and I will not quote here a number my repository contradicts.

What the change actually does, on a recorded protocol (seeds 20260807..14, 640×360, against a committed baseline):

seed 20260812: paired chroma p95 1.285 → 1.002 (p99 12.30 → 10.47), stops failing the probe
five seeds: identical
seeds 20260810 / 20260814: worse by 0.03 and 0.07 code values
probe failures: 3/8 → 2/8; paired luma p95 identical on all eight


A smaller, duller, real result. I will take it over an impressive one I cannot reproduce.

Why the first number was wrong matters more than the number. The Lab writes 20–120 samples and leaves 97–98 % of the rivalry zone bit-identical, so a p95 over ~1500 samples is set by samples it never touched: when the written ones jump to the top of the ranking they displace the tail, and the p95 reports a rank shift as a degradation. Two correct shader fixes were scored as failures by that gate, and one was discarded on that evidence before being recovered by it. Amplitude is now measured only over the samples the Lab actually wrote.

With the measurement repaired, a real defect surfaced: the chroma pass converged a 4:2:0 sample toward an owner read from one corner of the 2×2 luma cell it covers. Reading the opposite corner merely relocates the error — measured — which proves no corner is right: the cell holds two owners, one chroma sample cannot represent both, and averaging the four corners would manufacture a third coordinate owned by neither. A straddling cell is now refused outright, discarding 36 harmful corrections for zero useful ones lost and lowering the worst single sample on three topologies (22.94 → 0.98 among them) while raising it on none.

One reservation, and not a minor one. The conjugate sample sits at destination_x + 2d, so the edge band where it falls off-screen is, at each pixel, as wide as that pixel's own disparity — widest exactly on near-field content entering or leaving frame, where a one-eye chroma error is most rivalrous. Raw is kept there for want of evidence; clamping into range would certify against a fabricated conjugate.

6. The sparse advective matte, and one ulp

MatAnyone2 at 720p measured 62–118 ms and often delivered 9–12 fps against a 23.976 fps request, so the automatic cap is now 512p — 49 % fewer model pixels than 720p, preserving 99.32 % binary IoU against the 640p reference (two different references, stated separately on purpose). Under 4K GPU contention even that fell to 2.4–4.3 fps at 313–420 ms inference, so the matte is transported rather than waited for: bidirectional luma flow between its source PTS and the displayed PTS, with photometric and forward/backward rejection acting on alpha inside an adaptive sparse band around the contour rather than over the whole frame. Interiors and background stay bit-exact outside that band. Confidence decays with the pipeline's own measured delivery latency and fails closed on missing evidence, so a stale silhouette is dropped, never frozen into the background.

A floating-point war story, for the right audience. The promotion rule was an inverse exponential solved for the grace period, and it does not round-trip in binary: it evaluated to 0.6800000000000002 — one ulp above the 0.68 threshold — at 45 of the integer horizons where that cap binds, admitting exactly the case the cap existed to exclude. Nudging the constant by an ulp moved the defect from "above" to "exactly at" rather than removing it. The invariant is now an age comparison against min(grace + margin, horizon) with a strict <, exact in binary at every horizon, no epsilon anywhere. Sweeping the 80–700 ms range at 0.25 ms granularity plus 20 000 random configurations: the old formulation fails 8468 times, the new one zero. The defect lived at horizons 420 and 462 — neither a boundary of anything, which is why hand-picked test points would never have found it.

7. Numbers — and what 41.7 ms actually means

You will see 41.7 ms throughout my documentation. It is not a benchmark result. 41.7 ms is 1/23.976 — the budget, not the cost. I would rather say so plainly than have someone here do the division and conclude I hoped nobody would.

Ryzen 9 5950X + RTX 4090, TensorRT, Quality preset, Scope grid, 23.976 fps source:
infer_ms 11.7 stab_ms 13.2
flow_ms 5.95 cycle_ms 41.7 (source-paced)
inwait_ms 20.95 map age 21.5 ms median
Half of every cycle is spent waiting for the next source observation; optimizing for more fps would not change output cadence here.

Same preset on a 59.94 fps source, where the budget stops binding: 42–44 fps sustained, cycle 23.9 ms, reswait 0.0 ms after warm-up — the inference result is ready when it is consumed, which is the proof that the overlap works. The ceiling there is CPU post-processing, not the model.

Real 4K playback, because a synthetic bench is not a promise. 3840×1608 Scope, Quality preset: 22.8 fps average, 24.0 peak against a 23.8 fps source ceiling, cycle 44.3 ms against a 42.0 ms interval, 35 % of the cycle idle. The depth engine is not the limiting stage on that content.

On Scope masters the black bars are detected and inference runs on a rectangular grid — 756×322 instead of 756×756, 57 % fewer depth pixels for roughly twice the depth rate, because black bars carry no depth worth computing.

Two caveats, stated before anyone has to ask. My preset table quotes 14.9 ms for Quality/TensorRT: that is a square-grid synthetic-pan bench and it understates real Scope playback by about 1.5× — a floor, never an expectation. And TensorRT is optional and opt-in, worth roughly 1.7–2× on supported cards; the default path is DirectML on any Direct3D 11 GPU, so expect less than every figure above.

8. The record

Every change above ships behind its own environment-variable rollback — SYLC_SYNTH3D_FLUSH, SYLC_LOOKAHEAD_DECAY, SYLC_MATANYONE2_AUTO_CAP, SYLC_SYNTH3D_DEPTHCUT_ADAPT — so that every claim on this page can be switched off and measured against. 191 tests run in about three seconds and need neither media nor a GPU.

The instrumentation ships enabled, and it has earned its place. When playback once froze with the audio playing on, two evenings went into dissecting this engine — which turned out to be the victim: a stalled depth worker and a stalled renderer wrote byte-identical logs, and every hypothesis built on them was unfalsifiable, until the one counter that could tell them apart was written. That counter ships too.

What is still wrong is written down next to what works. Two topologies stay known-red in the committed chroma baseline, with their diagnosis attached, so the matrix cannot be made green by deleting them. The edge band where the conjugate sample leaves the frame is uncertified, and stays raw. One published number was withdrawn.

That is the whole of it. Depth read back out of flat film, at film cadence, on the GPU you already have, given away with nothing asked in return.

The industry walked away from 3D. The depth never left the frame.

5ymph0en1x