Missing some pixel format conversions

Apr 20, 2013 at 12:38 PM
Hey,

I am missing some pixel format conversions in this library that are fairly important in my opinion. For instance, conversions from the float and fixed point formats to lower bit depths like RGB24 would be handy, e.g. just for showing the image in an image viewer. RGB48 to RGB24 seems to be missing as well. If I remember correctly, WIC did support these conversions. What's the reason these have been left out here?
Apr 29, 2013 at 10:54 PM
I'll second this. I'm writing a Quick Look plug in for OS X to add Jpeg-XR support. It's going good so far but the library bombs out with any image using an unusual colour format. Really this library really should be capable of decoding Jpeg-XR images of ANY colour format into AT LEAST RGB24 and RGBA32. Anyone consuming the library should not be expected to have to perform colour conversions on a raw pixel data just in case a user decides to try to open a RGB128Float image.

It's worth noting that for all intents and purposes this means for anyone using the library to displaying jpeg-xr images on the screen, this library is only a partial implementation of Jpeg-XR. If say, an image viewer were to use this library as is, it would only be able to open jpeg-xr files that are rgb24, rgba or rgb565 (assuming the UI library requires RGB24 bitmap sources). This would effectively mean that image viewer had an incomplete jpeg-xr implementation and therefore would be in fact liable for patent infringement by Microsoft as the community promise Jpeg-XR is covered by specifically excludes incomplete implementations of covered technologies. It's hard to encourage the adoption of this library in open source projects when it technically places them in legal jeopardy with Microsoft.

My understanding is that this was actually by design and indented to force anyone making Jpeg-XR implementations to implement the entire spec and not only parts they were interested in. This would mean people could create RGB64 Jpeg-XR images and know in confidence they could be opened by any Jpeg-XR software. Otherwise we could in theory have situations were only some Jpeg-XR libraries support anything but RGB24 and therefor nobody uses Jpeg-XR for anything else but RGB24 which defeats most of the whole point of the image spec.

I hope these extra colour conversions are added soon. It seems to have gone very quite around here.
Apr 30, 2013 at 3:00 PM
Do you accept code contributions to this library? If yes I can start adding some of the more important conversion routines that are still missing. I understand that format conversion is not strictly a part of the encoder and decoder and is independent of JPEG XR itself. But it will be needed in almost every software that will use this lib for showing an image or further processing.

@myutwo33: I wouldn't go so far as to say this is an incomplete implementation of the standard, there are pixel formats specified like the n-channel and CMYK formats that can't be shown on the screen anyway. So it could be pretty hard to tell if an implementation is complete or not.
Coordinator
Apr 30, 2013 at 4:23 PM
@christoph_hausner Yes! Absolutely we are accepting contributions. Please feel free to add the missing format conversions.
Apr 30, 2013 at 5:42 PM
@matthewu Is Microsoft planning on putting much more work into this library? I could add some of the colour format conversions. It looks like pretty simple work, but it looks like it would take a long time to make comprehensive colour conversions. I personally would hope this would be the kind of work Microsoft would want to do themselves as it relates to the basic functionality of the library. I understand these kinds of progress get limited resources assigned to them.

The library seems really nice sans the lack of some colour conversations, but the pace of development seems pretty far behind what Google are doing with webP. They appear to be doing libwebp check-ins on a near daily basis and in my own testing libwebp is between 60-160% faster than jxrlib in decoding images encoded at similar quality settings. I think Jpeg-XR is a great format especially for the broad colour format support and I can think of a few scenarios where it could be really useful in such games texture files (RGB565 etc) and video post production work and CGI animation (RGBA128Float). I might be interested in doing some porting work for these types of scenarios myself but it gets very difficult to work with unusual colour formats if jxrlib can't do colour conversion to RGBA32 for everything out of the box. It makes doing these kinds of ports a lot less appealing to me personally.

I could technically add these extra colour conversations myself, but it's my estimation at the moment that that would take more time than coding any individual plug-in. But more than that I feel this is really the kind of fix Microsoft as a company should want to be doing themselves and if Microsoft, the company that made the format, is bothered to put the resources into ensuring the jxrlib is as good as possible for developers, then why should I bother trying to support it with plug-in ports etc? Say, if Microsoft isn't interested in improving the jxrlib after the initial release, then that's not very encouraging for making other people wanting to work on it either for free.

I can understand Microsoft not wanting to add small features like maybe adding support for some obscure operating system, and allowing the community to add that kind of stuff instead, but the colour conversions thing effects almost any kind of usage of the library. Say I write a image viewer with jxrlib and the user tries to view a jxr file with it; "The file doesn't appear. Is the file broken? Is the program broken? Well the file is viewable in windows photo viewer so it can't be the file... I guess the program is broken then." What's happened though is the user unknowingly tried to open a .jxr file with a CMYK colour format. It will work in Windows photo viewer but jxrlib won't convert it to RGBA32 for my image viewer to display it.

I'm not trying to be mean here, and I'm making a few assumptions about how jxrlib will be maintained going forward and the library does look like really great work imo. But it's just my personal feeling that Microsoft isn't jumping to add features to jxrlib as soon as two developers point out lacking features that could be added with a very modest amount of resources, and that does make me feel like the Microsoft isn't trying as hard as I feel they should.
May 11, 2013 at 10:55 PM
I just pushed code to my fork to allow conversion from most pixel formats to the corresponding 8bits per channel format, that is, all grayscale formats to 8bppGray, RGB formats to 24bppRGB and RGBA formats to 32bppRGBA. Comments? I am aware that applications will need to perform a second conversion if they only support 24bppRGB or 24bppBGR, for example, but I didn't want to add the same routines again and again and blow up the code size.
May 12, 2013 at 11:30 AM
That code looks really great. There's a lot of work in there :).

About converting to 24bppBGR, how frequently is this pixel format required in the real world? I can think of quite a few places where it get's used, but is it more common than RGB? If it's significantly rarer I think it would be acceptable for now to just let a second conversion be performed.

I was thinking about this during the week about what would be the best way to go about improving the pixel format conversions. I think the first priority would be to make it possible to convert all JPEG XR supported pixel formats to rgb24 and rgba32. This looks like a great step towards that and most of that work seems to be done now. If I may I'd recommend going further and allowing all black and white image formats to be convertible to rgb24 simply by adding conversion methods that convert from say gray4 -> gray8 -> rgb24. In this case only new conversion method gray8_rgb24 is needed. This requires two colour conversions, but I think this is an acceptable compromise for two reasons: it keeps the code size down and we can assume situations where someone will be loading a gray4 JPEG XR image and loading it into rgb24 and not gray8 will be rare. Similarly I think it would be useful to be able to load images without alpha channels to rgba32 and ones with alpha channels to rgb24. I think this could be added relatively easily by adding rgb24_rgba32 and rgba32_rgb24 methods, and then say to load a rgb48 image to rgba32 would be done by rgb48 -> rgb24 -> rgba32 etc. I think two conversions is justifiable here as it should be considered a rare event that someone will be trying to load an image with an alpha channel into an image format that does not support one. Generally I think it should be considered that anyone converting obscure pixel formats into other non intuitive pixel formats should not expect the same performance as converting to more conventional and expected pixel formats. Any thoughts on this approach?

Other colour conversions that are missing are CYMK to rgb24 and N-Component to rgb24. I'll be happy to add these conversions myself. The N-Component one sounds a bit unintuitive, but Windows Photo Viewer can open N-Component JPEG XR images just fine. If the N-Component image has 3 channels it assumes they are RGB, if it has 4 channels are more it assumes the first 4 are CYMK and the last 4, if present, seem to be red, green, blue and beige. I'll have to look into that some more..

I'll be happy to add the other conversions I mentioned as well, if there's no objection to that approach. Most of the work that was making me feel nauseous now seems to be done :).

on a somewhat related note I have a question about parallelisation.. I'm very new to C still and I'm not sure how difficult this kind of stuff is to use, but from what I can tell there's no multithreading code in the jxrlib right now. I wouldn't dare try adding multithreading support to the jxr decoding myself, but I imagine multithreading the colour format conversion could make for a decent performance boost in this part of the library and I think should be relatively easy... Would this be prohibitively difficult to consider implementing in the future?
May 12, 2013 at 2:45 PM
Not sure about 24bppBGR, as far as I know it's commonly used in GDI under Windows and in the BMP format, for example.

About CMYK and n-channel formats, I assumed Windows would not display them, but since you say it does it obviously makes sense to include conversions for these, too. This is what the JPEG XR Standard says about n-channel interpretation:
There is no inherent description of the color context for n-channel data, so when using an n-channel pixel format, it is recommended that an ICC profile should always be included (when applicable). However, if an ICC profile is not present and no other indication of the color interpretation is available by other means, the following default color context assumptions are recommended:
– When n = 3, the three channels should be assumed to be red, green, and blue (RGB) encoded using the sRGB color space.
– When n > 3, the first four channels should be assumed to be cyan, yellow, magenta and black (CMYK) encoded according to CGATS/ SWOP TR003 2007 CMYK. Any additional channels should be discarded.

I am no C expert either but I think adding parallelization is a bad idea for a codec library that aims to be as platform-independent as possible. There is no standardized support for threading in ANSI C so you'd have to employ OpenMP or similar libraries, which means applications using jxrlib would have a dependency on that. I haven't done any benchmarks but I suppose format conversion is very fast compared to the actual decoding. And the current conversion routines do not look to be optimized anyway (mine included).
May 13, 2013 at 10:18 AM
mmm. In that case it's probably ideal to have every format be convertible direct to 24bbpBGR? That adds massively to the amount of code though..

That stuff in the specification about CMYK and N-Channel pixel formats in interesting. I guess the thing to do will be to try follow that. I had a quick look at how to do CMYK to RGB conversions and it doesn't look like there's any easy way to do that. Basically the only way to do it that is considered in any way accurate is using colour profiles like CGATS/ SWOP TR003 2007 CMYK. This would add a lot of complexity to the code. I could just do a pretty dumb conversion for now and simply warn anyone using the library that we don't do good CMYK to RGB conversions. This wouldn't be an issue in applications like Photoshop that will load the CMYK stream directly and convert it to RGB themselves. It would be unfortunate if someone who saved a CYMK jpeg xr file using the Photoshop plug-in wasn't able to it with remotely accurate colours in quick view on mac os x using my plug-in, or any other jxrlib based image viewer. I'm really not sure what the right thing to do here is.
May 14, 2013 at 2:32 PM
Edited May 14, 2013 at 2:33 PM
I think we can safely ignore CMYK and n-channel formats for now, as far as I know they have been included for special printing scenarios only and I don't think it's a good idea to add half-baked conversions to the library.

Maybe a compiler directive could be an option for choosing whether all formats should be convertible to RGB24 or BGR24, but that would still double the source code size. Let's leave it as it is for now. I guess you and me are the only consumers of this library anyway so far.
May 17, 2013 at 6:17 PM
Edited May 17, 2013 at 6:28 PM
Agreed that CMYK should probably be ignored. Probably simplest if applications using the library are asked to take responsibility of converting between colour modes and providing the right colour profiles to do so.

I've had a look at what kind of work it would take to make it possible for every suitable format to be converted into RGB24, RGBA32, BGR24 and BGRA32. Just to clarify the approach I'm thinking of, each applicable pixel format should be converted to it's 8-bpc variant. After that it can be converted directly to RGB24, BGR24, RGBA32 or BGRA32 . Another conversion could be performed to convert a pixel format with alpha to one without, or visa versa, so a final conversion might be RGBA32 -> RGB24.

I made a graph to try and get an idea of what kind of work this would require. Bear with me because this graph is confusing to look at... the rows of the grid are named by their RGB equivalent, so the RGB equivalent of RGBA64 is RGB48. Basically the same number of bits per channel. This is confusing but if I ordered rows by bits per channel only things would get confusing when I has to enter formats like RGB565. To me this seemed like the best way for to draw it out for my own purposes.

White boxes represent a colour mode that exists as an equivalent to the RGB mode of the row. So B&W has only one white square which is RGB "3-bit" which is equivalent in B&W to "1-bit per channel", confused yet? The rows with yellow coloured titles are rows that are equivalent to RGB pixel formats that have empty bytes for alignment purposes such as RGB64. There's no equivalent to this kind of format in pixel formats with an alpha channel or 4 channels, so I coloured the cells in these rows black to make that clear to myself. So now that no one is confused... ahem...

The black arrows on the grid mark out conversions needed. The vertical arrows all represents conversions to the 8-bit equivalent of the relevant colour mode. Down further, the horizontal lines represent conversions from the 8-bit version of a colour mode to RGB24, BGR24, RGBA32, BGR32. Down below the thin red line are some conversion to convert from noalpha->alpha and via versa.

Finally the arrows coloured green represent the ones now covered by you and Microsoft. Nice to see many of them already now covered. Rows were the RGB pixel format title is yellow are rows that are equivalent to RGB pixel formats that have empty bytes for alignment purposes such as RGB64.

Please take note of the fact I did not colour BGR555 -> BGR24 and others in that area because.... the jxrlib library goes not contain a pixel format for BGR555, or BGR565 or BGR101010. Instead jxrlib uses RGB version of these, not BGR. The JPEG XR spec only contains BGR version of these colour formats and not RGB. I presume this is a bug or oversight in the library? Maybe these conversions work fine and are just miss-named though.

Image
May 20, 2013 at 5:40 PM
Edited May 20, 2013 at 6:58 PM
Huh, just found out that the WIC implementation in Windows can't even display 32bppRGBA files (no thumbnails and Photo Viewer can't open them either and fails with a misleading error message "No image found.". IE fails too.). The pixel format needs to be 32bppBGRA to work. And for black-white images the black-is-one bit is ignored, too. This is bad, my GIMP plugin currently makes use of both 32bppRGBA and the bit field.. would have to implement extra conversions as a workaround. Exotic formats like CMYK and n-channel formats are displayed but simple RGBA fails. A bit funny, isn't it?

Edit: Apparently jxrlib supports some pixel formats that are not part of the official standard. These are 32bppRGB, 32bppRGBA, 32bppPRGB and 96bppRGBFloat. They were probably only included for in-memory image data and not as the final format in the jxr file. I propose we add a check that forces consumers of this library not to encode in these formats directly.
Developer
May 23, 2013 at 2:39 PM
Right, WIC for some reason doesn't handle those pixel formats (32bppRGB, 32bppRGBA, 32bppPRGB and 96bppRGBFloat). The Photoshop plugin checks this outside the library but it would be a good idea to put the check in the library as you suggest.