---------------------------------------------------------

First name: Timothee
Last name: Soen
Email: fr-tsfr-teamrarv@thalesgroup.com
Company name: Thales
Subject: VideoFormat with Stride and Android Camera2 API
Description: Hello,

This will be a bit of a technical explanation, I will try to make it as clear as possible.

- We use the LiveSwitch Unity SDK
- We use a Augmented Reality provider (Wikitude) for our Android build, which is using the Camera on Android devices.
- We have access to the video frames Wikitude is filming, so we set-up a LiveSwitch custom VideoSource (and Sink on the other side).
- Wikitude gives us data about the VideoFormat of the frames, the width, the height, RowStride and PixelStride.

For now our application is NOT using the Android Camera2 API (https://developer.android.com/training/camera2). It is an option in the Wikitude parameters.t

Our application is working fine, video streaming works.
This is the data from Wikitude about the frames :
- ColorSpace : "YUV_420_888"
(https://developer.android.com/reference/android/graphics/ImageFormat#YUV_420_888)
- FrameWidth : 1280
- FrameHeight : 720
- RowStride for each 3 planes of the frame : 1280 ; 640 ; 640
- PixelStride for each 3 planes of the frame : 1 ; 1 ; 1
- Frame data length : 1382400 (I suppose this is == 1280 * 720 * 1.5)

In LiveSwitch, our custom source raises frames like this :
var dataBuffer = DataBuffer.Wrap(data);
var videoBuffer = new VideoBuffer(width, height, dataBuffer, VideoFormat.I420);
var videoFrame = new VideoFrame(videoBuffer);

According to https://wiki.videolan.org/YUV, I420 is a YUV_420 format.

Now, about our problem.

When we activate the Android Camera2 API (through the Wikitude option), we start to receive (or at least display) frames with wrong colors. Find attached a screenshot of a frame.

This is the data from Wikitude about the frames :
- ColorSpace : "YUV_420_888"
- FrameWidth : 1280
- FrameHeight : 720
- RowStride for each 3 planes of the frame : 1536 ; 1536 ; 1536
- PixelStride for each 3 planes of the frame : 1 ; 2 ; 2
- Frame data length : 2211070 (It looks like 1536 * 720 * 2 = 2211840)

It looks like something has changed about the VideoFormat.
I will also open a ticket on their side to get more details about the format.

I noticed their is a Stride property of the LiveSwitch VideoBuffer class.
I tried setting the Stride property to "1536" and it gave a result a bit better, but not quite (see attached picture).
There is also a "Strides" property which is an array, but i am not sure what it is used for, since the "get" of the Stride property returns only the first element of the array : ("return strides == null ? 0 : strides[0];")

I feel like I am on the right track but could not figure out how to resolve this.

Any suggestion would be appreciated.

Best regards,
Thales team
Steps to Reproduce: .
Product: LiveSwitch Client SDK
Product Version: 1.16
Category: Product issue


---------------------------------------------------------


Hi Thales Team,

It would be good to see what extra info Wikitude provides about the YUV format they're using. Something more specific than I420 could be helpful.

It seems setting the Stride value correctly has made the Y plane render correctly. However, the U and V planes are incorrect. Strides is the primary field.
Stride is just a wrapper of Strides to provide the first value in the array. You can set the Strides array and maybe this will have some improvement on the U and V planes. 
My concern is the pixel stride values. As we don't have a field for that, I'm not sure if we handle data well with greater than 1 pixel stride.


Rusty


---------------------------------------------------------