Wikitude
How can we help you today?
Need more details about the FrameColorSpace format of Wikitude frames with Android Camera2 API
Hello,
I am writing some of our findings here as well as new questions for Wikitude support, hopefully it can help someone.
The camera produces 1280 * 720 frames.
When Wikitude.enableCamera2 is set to false :
- The frame "FrameColorSpace" given by Wikitude is "YUV_420_888" and consists of 3 planes : (Y, U, V)
- Width = 1280
- Height = 720
- RowStride = { 1280, 640, 640 }
- PixelStride = {1, 1, 1 }
This format is actually called "I420" according to https://wiki.videolan.org/YUV. It is by far the most common format. Android documentation can be found here : https://developer.android.com/reference/android/graphics/ImageFormat#YUV_420_888.
There is also the YV12 format, which has the same characteristics as I420, but instead of having Y plane, then U plane, then V plane, the order of the U and V planes is reversed.
Is there any way to know if the format is I420 or YV12 ?
When Wikitude.enableCamera2 is set to true:
- Width = 1280
- Height = 720
- RowStride = { 1536, 1536, 1536}
- PixelStride = {1, 2, 2 }
A PixelStride of 2 indicates that the format is a "NV" format, meaning a semi-planar format where U and V are interleaved like this : YYYYYYYY UVUV (NV12) or YYYYYYYY VUVU (NV21).
"For a 2×2 group of pixels, you have 4 Y samples and 1 U and 1 V sample"
Notice that here the RowStride is greater than the width, so the Y plane has a DataSize of 1536 * 720 instead of 1280 * 720.
Except the "padding" on the last row is absent, giving a DataSize of 1536 * 720 - (1536 - 1280) = 1536 * 720 - 256.
Note that for NV formats, the 2nd and 3rd planes are identical.
The DataSize of the U/V plane should be 1536 * 720 / 2 - 256 (because the padding on the last row is missing like for the Y plane).
However, the "Datasize" given by Wikitude was 1536 * 720 / 2 - 257 (== 552703).
We do not understand how it is possible to have an odd number for the Datasize.
However, there is no way to know if the format is NV12 or NV21.
Also, why does Wikitude gives us a "FrameColorSpace" of "YUV_420_888" and not directly "YUV_420_NV12" or "YUV_420_NV21" since the enum exists ?
An input from Wikitude would be greatly appreciated.
Best regards,
Wikitude Technical Support
Hi,
The Wikitude SDK on Android uses the YUV_420_888 image format as it is the way the Android SDK retrieves the value to us. The only way you have to check this value is with the CameraFrame parameter that comes in the OnCameraFrameAvailable where you can check the ColorSpaces as follows: frame.ColorMetadata.ColorSpaces.
Since the value coming from Android will be YUV_420_888 it is true that it might be confusing to check, so I can share with you that we are using either YV12 or NV21. In the plugins samples, which I think are the samples you are looking for, we are using the NV21 format.
Best regards,
Aitor.
Hello,
Thank you for your answer.
If we are guaranteed that the format will always be YV12 or NV21 (so never I420 or NV12), that's fine for me.
Best regards,
Thales team
TeamThales RaRv
SDK : Wikitude Unity Pro SDK 9.6.
Unity : 2020.3.2f1