If I understand the documentation here (https://www.wikitude.com/external/doc/documentation/6.0/unity/pluginsapiunity.html) the new Camera2 interface the Frame returned in Unity with the YUV_420_NV21 colorspace is ImageFormat.YUV_420_888. The actual organization of this format is unknown across devices (the UV plane may or may not be interleaved). In order to reliably convert this format to NV21 I need to know the pixel stride which is available in Android when receiving Images through the camera preview. However, the Wikitude Frame object provides no such information. As a result, I have no reliable way of converting the Frame data into NV21. For reference, see https://bugs.chromium.org/p/libyuv/issues/detail?id=604. Do I fail to understand the problem or am I misunderstanding the Wikitude documentation?
The format will always be Yv12.
We don't do any conversion on the native side because we don't know how the data is going to be consumed and don't want to introduce overhead that might not be needed. Which is why we provide only a native pointer to the data. If you require the camera frame on the GPU, you can access the texture from the WikitudeCamera component directly and you don't need to go through plugins. If you need to do additional processing on the camera frame on the CPU, we recommend using a native plugin for that, like we do in our sample.
We will consider adding a helper function to convert to RGB if processing in C# is required.
Can you clarify what you mean by "The format will always be Yv12."?
Are you saying that the format will always be NV12?
If so then we are reporting that we are not seeing interleaved UV values.
If you are saying that the format will always be a 12 format (i.e. U followed by V) then this resolves our concern about the ordering.
And, you might consider changing the "colorspace" enumerator to something other than YUV_420_NV21.
Sorry I wasn't clear before. Since SDK 6.0 we changed it so that plugins always return 12 format on Android. In the next update, we will provide the stride information as well as fixing the enumerator to contain the proper value. It is expected that the values are not interleaved. I apologize for the inconvenience that this wasn't fixed properly in the initial release.
Sorry, but one final point of clarification.
The purpose of the original posting was to point out that developers have suggested that different devices may produce different results from the Camera2 interface (i.e. interleaved or non-interleaved UV).
Is it still the case (as your documentation states) that the result is the ImageFormat.YUV_420_888 format and therefore we will need the pixel stride information to determine whether it is interleaved?
Or is it "expected that the values are not interleaved" in which case we do not need the stride information?
Thank you for bringing this issue to our attention. You are right, this information is missing at the moment. We've added the strides information to the frame data and it will be available in the next update.
The Android specification for this format does not specify what order the U and V planes are ordered in. The documentation says it is "capable of describing any 4:2:0 chroma-subsampled planar or semiplanar buffer (but not fully interleaved)". This is not a significant issue when using the Java Image class because the U and V planes can be accessed directly (i.e. the 1 and 2 index of the planes is guaranteed to correspond to the U and V planes respectively). However, when the data has already been put into a single buffer if becomes impossible to know whether a UV interleaved plane consists of NV21 or NV12 ordering. We can live with having the stride information, we will cross our fingers that it will cover most situations, but the only practical way to do the conversion may be in native code utilizing the NEON chipset (as found in libyuv). I realize that the Camera2 interface has created this issue but it remains a performance, efficiency and reliability concern that ideally should not be imposed upon consumers of the data (especially in a non-native environment like C#).
I would recommend that you convert the image to NV21 in native code before passing the bytes to Unity.
I should clarify something from the comments above.
The format returned by Wikitude appears to be YV12 format which is V followed by U ( I stated incorrectly before).
This should not be confused with NV12 which is actually U followed by V or NV21 which is V followed by U (which is confusing indeed).