Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [january-dev] Reshaping and slicing RGBA image byte buffer for android screen capture

Hi Peter,

 

With a fresh mind, I’ve been able to make it work without the corrupted images.

Some refactoring of how I streamed the frames and adding a reentrant lock did the trick.

 

The image size doesn’t change, so how would I cache the dataset?

Can I push in a new byte[] ?  I tried with Dataset#set(java.lang.Object) but that didn’t work.

 

Kind regards,

Yannick

 

From: january-dev-bounces@xxxxxxxxxxx <january-dev-bounces@xxxxxxxxxxx> On Behalf Of Peter.Chang@xxxxxxxxxxxxx
Sent: woensdag 26 juni 2019 10:55
To: january developer discussions <january-dev@xxxxxxxxxxx>
Subject: Re: [january-dev] Reshaping and slicing RGBA image byte buffer for android screen capture

 

Hi Yannick,

 

Good to see it sort of works. Does the size of the image change?

 

If it’s a synchronization problem and you are latency sensitive then you can reduce some overhead by caching the dataset and the sliceND object.

 

Regards,

Peter

 

 

From: january-dev-bounces@xxxxxxxxxxx <january-dev-bounces@xxxxxxxxxxx> On Behalf Of Yannick De Visscher
Sent: 25 June 2019 17:41
To: january developer discussions <january-dev@xxxxxxxxxxx>
Cc: Erwin De Ley <erwin.de.ley@xxxxxxxxxxxxxxxx>
Subject: Re: [january-dev] Reshaping and slicing RGBA image byte buffer for android screen capture

 

Hi Peter,

 

Thanks! Your interpretations was correct.

 

It kind of works now with the following code:

 

private class ImageAvailableListener implements ImageReader.OnImageAvailableListener {
   
private ByteBuffer imageBuffer;

   
@Override
   
public void onImageAvailable(ImageReader reader) {
       
try (Image latestImage = imageReader.acquireLatestImage()) {
           
if (latestImage == null) {
               
return;
            }

           
final Image.Plane plane = latestImage.getPlanes()[0];
           
final ByteBuffer buffer = plane.getBuffer();

           
if (imageBuffer == null) {
               
// prepare a buffer for ABGR to ARGB conversion
               
imageBuffer = ByteBuffer.allocateDirect(buffer.capacity());
            }
           
// clear the buffer and reset to position 0
           
imageBuffer.clear();
           
imageBuffer.position(0);

           
int width = latestImage.getWidth();
           
int height = latestImage.getHeight();

            
int pixelStride = plane.getPixelStride();
           
int rowStride = plane.getRowStride();
           
int rowPadding = rowStride - (pixelStride * width);

           
// convert from ABGR to ARGB to get the correct color space for OpenTOK
            
RoboticChameleon.fromABGR().toARGB(buffer, rowStride, imageBuffer, rowStride, width, height);

           
int paddedWidth = width + (rowPadding / pixelStride);
           
if (paddedWidth > width) {
               
// we need to remove padding on the right side of this image (some devices show this behaviour in portrait mode)
               
imageBuffer.position(0);
               
int size = rowStride * height;
               
byte[] pixels = new byte[size];
               
imageBuffer.get(pixels, 0, size);

               
// create a compound dataset with itemsize = pixel stride = 4 (RGBA pixel) and reshape it
               
int fullWidth = rowStride / pixelStride;
                Dataset dataset = DatasetFactory.createFromObject(pixelStride, CompoundByteDataset.
class, pixels, height, fullWidth);

               
// slice to width and height
               
Dataset slicedDataset = dataset.getSlice(new SliceND(dataset.getShapeRef(), (Slice) null, new Slice(width)));

               
imageBuffer.clear();
               
imageBuffer.position(0);

               
// put the sliced byte array in the image buffer
               
imageBuffer.put(((CompoundByteDataset) slicedDataset).getData());
            }

           
frame = new Frame(imageBuffer, width, height);
           
newFrameAvailable = true;

        }
catch (Exception e) {
            Log.e(
LOG_TAG, "Error processing screen capture frame");
           
throw new RuntimeException("Error processing screen capture frame", e);
        }
    }
}

 

As long as the image doesn’t change too much, it’s fine.

There is still some distortion, but that’s ok.

 

cid:image002.jpg@01D52C04.664E0C60

 

But when something is happening/moving on screen, the image is not correct.

After a few frames it stabilizes again.

 

cid:image003.jpg@01D52C04.664E0C60

 

I think I need to build in some sort of synchronized locking on the imagebuffer, to avoid incorrect frames.

 

Thanks for the help!

 

Kind regards,

Yannick

 

From: january-dev-bounces@xxxxxxxxxxx <january-dev-bounces@xxxxxxxxxxx> On Behalf Of Peter.Chang@xxxxxxxxxxxxx
Sent: dinsdag 25 juni 2019 14:37
To: january developer discussions <january-dev@xxxxxxxxxxx>
Cc: Erwin De Ley <erwin.de.ley@xxxxxxxxxxxxxxxx>
Subject: Re: [january-dev] Reshaping and slicing RGBA image byte buffer for android screen capture

 

Hi Yannick,

 

Can you confirm my interpretation of your definitions is correct. I am assume row-major pixel storage.

 

Pixel stride = position difference (in byte buffer) between pixels (4, in this case)

Row stride = position difference between one row and the next (at same column)

Width = width of image

Height = height of image

Full width = (row stride /  pixel stride)

Size of buffer in bytes  = pixel stride * full width * height

 

Copying all the imageBuffer to a byte array then using data = "" CompoundByteDataset.class, height, full width)

To grab the left portion, take a slice with

 

slicedData = data.getSlice(new SliceND(data.getShapeRef(), null, new Slice(width)))

 

So the flattened bytes are in

slicedDataBytes = slicedData.getData()

 

Regards,

Peter

 

From: january-dev-bounces@xxxxxxxxxxx <january-dev-bounces@xxxxxxxxxxx> On Behalf Of Yannick De Visscher
Sent: 25 June 2019 10:52
To: january-dev@xxxxxxxxxxx
Cc: Erwin De Ley <erwin.de.ley@xxxxxxxxxxxxxxxx>
Subject: [january-dev] Reshaping and slicing RGBA image byte buffer for android screen capture

 

Hi,
 
Erwin contacted you earlier today about slicing an image represented in a bytebuffer (on android).
I’m trying to accomplish this with a CompoundByteDataset as suggested by Peter, but I’m having some problems.
 
Below is my current code. I’ve added some additional comments marked in yellow
Could someone point me in the right direction here?
 
 
private class ImageAvailableListener implements ImageReader.OnImageAvailableListener {
       
private ByteBuffer imageBuffer;

       
@Override
       
public void onImageAvailable(ImageReader reader) {
            
try (Image latestImage = imageReader.acquireLatestImage()) {
               
if (latestImage == null) {
                   
return;
                }

               
final Image.Plane plane = latestImage.getPlanes()[0];
               
final ByteBuffer buffer = plane.getBuffer();

               
if (imageBuffer == null) {
                   
// prepare a buffer for ABGR to ARGB conversion
                   
imageBuffer = ByteBuffer.allocateDirect(buffer.capacity());
                }
               
// clear the buffer and reset to position 0
               
imageBuffer.clear();
               
imageBuffer.position(0);

               
int width = latestImage.getWidth();
               
int height = latestImage.getHeight();

               
int pixelStride = plane.getPixelStride();
               
int rowStride = plane.getRowStride();
               
int rowPadding = rowStride - (pixelStride * width);

               
// convert from ABGR to ARGB to get the correct color space for OpenTOK
               
RoboticChameleon.fromABGR().toARGB(buffer, rowStride, imageBuffer, rowStride, width, height);

               
int paddedWidth = width + (rowPadding / pixelStride);
               
if (paddedWidth > width) {
                   
// we need to remove padding on the right side of this image (some devices show this behaviour in portrait mode)
                   
imageBuffer.position(0);
                   
int size = rowStride * height;    -> had to define a new byte array, since the byte array backed by the buffer had 7 extra bytes (which caused the reshaping to fail)
                   
byte[] pixels = new byte[size];
                   
imageBuffer.get(pixels, 0, size);

                   
// create a compound dataset with itemsize 4 (RGBA pixel) and reshape it
                   
Dataset dataset = DatasetFactory.createFromObject(4, CompoundByteDataset.class, pixels, rowStride / pixelStride, height);

                   
// slice to width and height
                   
Dataset slicedDataset = dataset.getSlice(new Slice(0, width), new Slice(0, height));   -> this takes a very long time

                   
imageBuffer.clear();
                   
imageBuffer.position(0);
                   
// convert the sliced dataset back to a byte array
                   
imageBuffer.put(((CompoundDataset) slicedDataset.flatten()).getByteArray());    -> not sure about this
                }

//                frame = new Frame(imageBuffer, paddedWidth, height);
               
frame = new Frame(imageBuffer, width, height);
               
newFrameAvailable
= true;

            }
catch (Exception e) {
                Log.e(
LOG_TAG, "Error processing screen capture frame");
            }
        }
    }
 
 
 
This is an example image, from which I want to remove the black bar on the right.
 
cid:image001.png@01D52B4E.94459810
 
 
Kind regards,
Yannick

 

 

-- 

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
 


Back to the top