Magnifying Glass AIR 2 application or how to communicate with a Java program from AIR

My favorite feature in Adobe AIR 2 is, by far, Native Processes: the ability to launch and control and communicate with a native process. It could be any executable on the machine where the AIR application is installed. I think this feature opens up a whole new range of AIR applications. When you add this feature to the ability to create socket servers, you have a powerful platform to build RIA applications for desktops.

Once I heard that this feature would make it in AIR 2 I was very excited. Why? Well, back in 2008 when we launched AIR 1.0, my fellow evangelist Serge Jespers created one of the coolest AIR applications for the AIR Tour. It was the smallest video player in the world. Basically it let you watch videos in the application icon from the Dock.

The application is extremely cool, but it has a small issue: it is too damn small to be able to see what’s going on. Being an engineer, I spent some time trying to find an engineering solution. Of course, I could have asked Serge to rewrite the application to make it bigger, but this wouldn’t have been an engineering solution. It would have been something that an accountant or manager would come up with. My solution is to build a second AIR application that can be used to magnify the video played inside the icon. This application would act like a digital magnifying glass.

With AIR 2 I’m finally able to implement the magnifying glass app pretty easily. Below you can see a screenshot of my application in action. It has two windows. The first window is the view port of the magnifying glass. You can see how many frames per second it processes, you can control the amount of zooming, and you can drag it around your screen. The second window displays the magnified image.

mg_1

The internals

How did I do it? The application has two main parts. One part is the AIR application itself. It renders the UI, controls the view port and the zoom factor, and scales the image.  The second part is a Java program that captures a screenshot of a portion of the screen. The Java program is controlled by the AIR application.

Using the NativeProcess and NativeProcessStartupInfo classes from AIR 2, you can launch an executable. In order to communicate with the executable you can use standard input and standard output. I wrote the Java program to output the bytes of the screenshot to standard output. It listens to standard input for commands, such as take a shot, set the viewport, or terminate the program. I compiled the Java program as an executable JAR file and placed in the AIR application root folder.

In order to capture the output of the Java program all you have to do is to register a listener on the NativeProcess instance for the standard output events. When you want to send commands you write bytes to the standardInput property of the same object. Here is a snippet of code, for the complete code have a look at the ScreenShotService class from the AIR application.

   1: private var nativeProcess:NativeProcess;
   2: private var npInfo:NativeProcessStartupInfo;
   3: //setting the arguments for starting the Java program
   4: var arg:Vector.<String> = new Vector.<String>;
   5: arg.push("-jar");
   6: arg.push(File.applicationDirectory.resolvePath("screenshot.jar").nativePath);
   7: arg.push("130");
   8: arg.push("100");
   9:
  10: npInfo = new NativeProcessStartupInfo();
  11: //setting the path to the native process
  12: npInfo.executable = new File("/Library/Java/Home/bin/java");
  13: npInfo.arguments = arg;
  14:
  15: nativeProcess = new NativeProcess();
  16: nativeProcess.addEventListener(ProgressEvent.STANDARD_OUTPUT_DATA, onStandardOutputData);
  17: //start the process
  18: nativeProcess.start(npInfo);
  19:
  20: /**
  21:  * Read the data from the standard ouput.
  22:  * Before reading a png, first you have to read the length of the image
  23:  */
  24: private function onStandardOutputData(e:ProgressEvent):void {
  25:     //reading the available bytes from the standard output buffer of the process
  26:     nativeProcess.standardOutput.readBytes(_processBuffer, _processBuffer.length, nativeProcess.standardOutput.bytesAvailable);
  27:     ...
  28: }
  29:
  30: //sending take command to the Java process
  31: nativeProcess.standardInput.writeMultiByte("take\n", "utf-8");

This is the relevant Java code (you can find the complete code inside the source folder of the application, ScreenShot.java):

   1: /**
   2:  * @param width of the screen capture
   3:  * @param height of the screen capture
   4:  * @param args
   5:  */
   6: public static void main(String[] args) {
   7:      if (args.length == 2) {
   8:          width = Integer.parseInt(args[0]);
   9:          height = Integer.parseInt(args[1]);
  10:      }
  11:
  12:     ScreenShot s = new ScreenShot();
  13:     BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
  14:     String text = "";
  15:     String[] tokens;
  16:
  17:     while (true) {
  18:         try {
  19:             text = in.readLine();
  20:             if (text.equals("take")) {
  21:                 s.capturePortion(x, y, width, height);
  22:             } else if (text.equals("terminate")) {
  23:                 return;
  24:             } else if (text.length() > 0) {
  25:                 tokens = text.split("\\|");
  26:                 if (tokens.length < 4)
  27:                     continue;
  28:                 x = Integer.parseInt(tokens[0]);
  29:                 y = Integer.parseInt(tokens[1]);
  30:                 width = Integer.parseInt(tokens[2]);
  31:                 height = Integer.parseInt(tokens[3]);
  32:             }
  33:         } catch (IOException e) {
  34:             System.err.println("Exception while reading the input. " + e);
  35:         }
  36:     }
  37: }
  38:
  39: /**
  40:  * Capture a portion of the screen
  41:  */
  42: public void capturePortion(int x, int y, int w, int h) {
  43:     try {
  44:         if (robot == null)
  45:             robot = new Robot();
  46:         BufferedImage img = robot.createScreenCapture(new Rectangle(x, y, w, h));
  47:         ByteArrayOutputStream output = new ByteArrayOutputStream();
  48:         ImageIO.write(img, imageType, output);
  49:
  50:         DataOutputStream dataOutputStream = new DataOutputStream(System.out);
  51:         //output the buffer size
  52:         dataOutputStream.writeInt(output.size());
  53:         //output the buffer
  54:         dataOutputStream.write(output.toByteArray());
  55:         dataOutputStream.flush();
  56:
  57:         output.close();
  58:     } catch (AWTException e) {
  59:         System.err.println("Exception while capturing screen. " + e);
  60:     } catch (IOException e) {
  61:         System.err.println("Exception while writting the image bytes. " + e);
  62:     }
  63: }

I am by no means a designer. Still, I think I managed to get a decent look of the main application window using Adobe Illustrator and Flash Catalyst. I created the design in Illustrator, and then using Flash Catalyst I transformed the graphics into a Flex application. And finally using Flash Builder 4 I added the logic.

Source code and native installers

You can download the Flex project from here, Mac installer from here, and Windows executable from here. This program requires Java 5 or newer and the Adobe AIR 2 runtime.

Things to know when working with Native Processes in AIR

In order to enable this feature you need to add the extendedDesktop profile to the application descriptor file. Add this tag as a child of the application tag:

   1: <supportedProfiles>extendedDesktop</supportedProfiles>

When using this feature you can’t package your application as an AIR file for distribution. You have to use the native installer. The easiest way to do this is to export for release from Flash Builder (you get the AIR file you normally use to distribute your application). And then you use adt at the command line to create the native installer. If you want a Mac installer you do it on a Mac, if you want a Windows installer you have to do it on a Windows. The command looks like this:

   1: adt -package -target native myApp.exe myApp.air

More on how to create native installers for AIR applications here (make sure you use the adt from AIR 2 and not one from an older version).

If you see an error like in the picture below when you install an application using the generated native installer, you should create a file named .airappinstall.log in your home folder. This log file can tell you what was wrong. In my case the error was “failed while validating native package: Error: Missing digested package file: .DS_Store starting cleanup of temporary files” (I fixed the problem by deleting the .DS_Store file from the source folder).

mg_2

Finally, you can check at runtime if the application has extended desktop capabilities by using this:

   1: if (NativeProcess.isSupported)
   2:     //extended desktop profile is available
   3: else
   4:     //extended desktop profile is not available

What’s next?

If you haven’t already, download the Adobe AIR 2 runtime and SDK and play with the new features. You can find a nice article about the new features from AIR 2 on Christian Cantrell’s blog.

I already have another idea: what about an AIR application that does screen sharing? Keep an eye out I might be able to pull it off!

PS. Many thanks to my friends Chicu and Raul from the Romanian AIR team for their help.

Later Update: My friend Benjamin Dobler created a nice screen recording application with AIR 2 (it captures the sound as well). Although for now the source code is not available, I still think it is worth having a look.

20 thoughts on “Magnifying Glass AIR 2 application or how to communicate with a Java program from AIR

  1. Thank you for a very good post on nativeprocess communication!
    By using your source code as a starting point I made an application that captures the displayed content on my secondary monitor (extended desktop, in windows) and displays this in a window on my primary monitor. It worked fine for a bit but after playing around for a few minutes I got this error message:

    Error: Error #2030: End of file was encountered.
    at flash.desktop::InboundPipe/readBytes()
    at test:ScreenShotService/onStandardOutputData()[C:\Users\DHL\Documents\Flex Builder 3\test\src\ScreenShotService.as:97]

    ScreenShotService.as:97 says:

    nativeProcess.standardOutput.readBytes(_processBuffer, _processBuffer.length, bLength – _processBuffer.length);

    Any clue on what might cause this and how to fix it?

  2. Hi, I get this error after starting on OS X 10.6.2.

    STANDARD_ERROR_DATA: [ProgressEvent type=”standardErrorData” bubbles=false cancelable=false eventPhase=2 bytesLoaded=121 bytesTotal=0]/n2010-01-18 08:40:05.944 java[35533:903] __CFServiceControllerBeginPBSLoadForLocalizations timed out while talking to pbs

    -ash

  3. @Ashley @Dag

    I have no idea why you get those errors. Maybe you can dig a little bit more by running the project in debug mode…

    I will ask around and if I find something I’ll be back.

  4. Pingback: AIR 2 Native Process with Java « bioRex21's blog

  5. Pingback: zero point nine » Blog Archive » Using NativeProcess in AIR 2 for screencaps

  6. Nice example! I am only curious about one thing, to call java you need to set the java executable path here:

    npInfo.executable = new File(“/Library/Java/Home/bin/java”);

    So… how can you be sure that path will work on every computer?

  7. You can’t actually. You can look for java or you can ask the user to provide the path to the java.

    On Mac OS usually there is symbolic link to the java so it is easy to guess where it is.

  8. I see, thanks for your answer! :)
    It’s just I am developing an application on Air using some functionalities I made in Java and I was looking for some way I can provide the java path without asking the user to provide it (you know some users really don’t know about what are you asking for and with some clicks put your app right into the trash).

    Thanks and congrats for the blog :)

  9. You could write a simple program in C++ that reads the the javahome variable from the registry. It would have to take OS variations into account.
    This program could then be executed by adobe AIR as a nativeapplication and report the path to Java back to AIR through stdout.

  10. It’s a clean solution to the problem. Thanks for sharing it with me!

  11. I’m amazed by the speed you get the images, but i can’t manage to run the app i get the “requirements” error but i have AIR2 and the latest Java, i’m on MAC, and created the file in my home directory and it didn’t help.. is there any way to see what the error is.

    Much appreciated.

  12. @NAZ

    You could try to run the project in debug mode; get the source code and import the project in Flash Builder and then start the project in debug mode.

    Also you could check that you have Java installed at this path: /Library/Java/Home/bin/java

    Regarding Java version, the minimum version required is 5.

  13. Pingback: 50 resources to get up to speed with the Flash Platform : Mihai Corlan

  14. Pingback: 有助于Flash平台开发提速的50个学习资源 « echo "RenYuan's Blog"

  15. Pingback: 有助于Flash平台开发提速的50个学习资源 – 新闻资讯 – 9RIA.com天地会 – 论坛 | 谱曲

  16. Pingback: Red5 Screen sharing success!.. using Adobe air and native process API - Adam Rensel | Red 5 Virtual Class Dev

Leave a Reply

Your email address will not be published. Required fields are marked *