Start a new topic
Solved

Need more details about the FrameColorSpace format of Wikitude frames with Android Camera2 API

SDK  : Wikitude Unity Pro SDK 9.6.

Unity  : 2020.3.2f1


Hello,

Our application is retrieving frames given by Wikitude using the OnCameraFrameAvailable method.
Then we use a WebRTC library (LiveSwitch : https://www.liveswitch.io/) to send the frame across the network a another user.

LiveSwitch provides different video encodings : "NV12", "NV21", "YV12", "I420" ...

When the option "Enable Camera2 API" on the WikitudeCamera is disabled, the frames are correctly encoded, sent and displayed on the other device. The encoding we use in LiveSwitch is I420.

However, when we enable the Camera2 API on the WikitudeCamera, The frames' ColorSpace seems to change, and they are no longer encoded correctly. We have tried many different set-up with no success.

I have found this thread on your forums, but it is 6 years old now and it was not really clear what format was used at the end, and if the U and V planes were interleaved or not : https://support.wikitude.com/support/discussions/topics/5000082012

I have attached my discussion with the LiveSwitch support with more explanation and reproduction steps, please read it to get more details.

It would be great if you could give us more details about the exact format of the frames given by the WikitudeCamera, and maybe how we can use the RowStride and PixelStride, etc... Anything would help.

Best regards,
Thales team

txt

Hello,


I am writing some of our findings here as well as new questions for Wikitude support, hopefully it can help someone.


The camera produces 1280 * 720 frames.


When Wikitude.enableCamera2 is set to false :

- The frame "FrameColorSpace" given by Wikitude is "YUV_420_888" and consists of 3 planes : (Y, U, V)

- Width = 1280
- Height = 720
- RowStride = { 1280, 640, 640 }

- PixelStride = {1, 1, 1 }


This format is actually called "I420" according to https://wiki.videolan.org/YUV. It is by far the most common format. Android documentation can be found here : https://developer.android.com/reference/android/graphics/ImageFormat#YUV_420_888.
There is also the YV12 format, which has the same characteristics as I420, but instead of having Y plane, then U plane, then V plane, the order of the U and V planes is reversed.


Is there any way to know if the format is I420 or YV12 ?


When Wikitude.enableCamera2 is set to true:

- Width = 1280

- Height = 720

- RowStride = { 1536, 1536, 1536}

- PixelStride = {1, 2, 2 }


A PixelStride of 2 indicates that the format is a "NV" format, meaning a semi-planar format where U and V are interleaved like this : YYYYYYYY UVUV (NV12) or YYYYYYYY VUVU (NV21).

"For a 2×2 group of pixels, you have 4 Y samples and 1 U and 1 V sample"


Notice that here the RowStride is greater than the width, so the Y plane has a DataSize of 1536 * 720 instead of 1280 * 720.
Except the "padding" on the last row is absent, giving a DataSize of 1536 * 720 - (1536 - 1280) = 1536 * 720 - 256.


Note that for NV formats, the 2nd and 3rd planes are identical.

The DataSize of the U/V plane should be 1536 * 720 / 2 - 256 (because the padding on the last row is missing like for the Y plane).
However, the "Datasize" given by Wikitude was 1536 * 720 / 2 - 257 (== 552703).

We do not understand how it is possible to have an odd number for the Datasize.


However, there is no way to know if the format is NV12 or NV21.


Also, why does Wikitude gives us a "FrameColorSpace" of "YUV_420_888" and not directly "YUV_420_NV12" or "YUV_420_NV21" since the enum exists ?


An input from Wikitude would be greatly appreciated.


Best regards,

Hi,


The Wikitude SDK on Android uses the YUV_420_888 image format as it is the way the Android SDK retrieves the value to us. The only way you have to check this value is with the CameraFrame parameter that comes in the OnCameraFrameAvailable where you can check the ColorSpaces as follows: frame.ColorMetadata.ColorSpaces.


Since the value coming from Android will be YUV_420_888 it is true that it might be confusing to check, so I can share with you that we are using either YV12 or NV21. In the plugins samples, which I think are the samples you are looking for, we are using the NV21 format.


Best regards,

Aitor.

Hello,


Thank you for your answer.

If we are guaranteed that the format will always be YV12 or NV21 (so never I420 or NV12), that's fine for me.


Best regards,

Thales team

Login or Signup to post a comment