A alternative to MrSID graphics format

[Duane Evenson]

I'm running linux and the MrSID viewer sucks -- besides having to open
up a separate viewer.
I also want to be able to store document images in the same format, and
not pay $3000 for a MrSID image encoder.
Right now I'm struggling with bloated tiff and pnm files.
Does anyone know of a comparable, open source, lossless file format?
TIA

[Denis Beauregard]

On Sat, 19 Mar 2005 07:26:42 GMT, Duane Evenson <[email protected]> wrote
in soc.genealogy.computing:

I'm running linux and the MrSID viewer sucks -- besides having to open
up a separate viewer.
I also want to be able to store document images in the same format, and
not pay $3000 for a MrSID image encoder.
Right now I'm struggling with bloated tiff and pnm files.
Does anyone know of a comparable, open source, lossless file format?

I don't understand your need. There are already many file
viewers available under Linux. What about using them ?


Denis

[David Harper]

Duane Evenson wrote:

I'm running linux and the MrSID viewer sucks -- besides having to
open up a separate viewer. I also want to be able to store document
images in the same format, and not pay $3000 for a MrSID image
encoder. Right now I'm struggling with bloated tiff and pnm files.
Does anyone know of a comparable, open source, lossless file format?

JPEG is the standard format for photographs, though it is not lossless.

PNG is the standard for most other applications, such as icons and line
art. It *is* lossless but it uses compression techniques to shrink the
file size right down. It's the patent-free replacement for GIF.

If you're looking for a flexible and powerful image viewer and
converter, use the Gimp. It's part of most standard Linux distros, and
even if you don't have it, it's open source so you can get it from
http://www.gimp.org

Hope this helps.

David Harper
Cambridge, England

[Duane Evenson]

JPEG is fairly good format using the Discrete Cosine Transform, but it
is lossy -- it creates artifacts along its block's borders. It also
takes time to decode for VERY large images because its compression and
decomression funcion is linear (double the dimensions = quadruple the
time).
BTW by large, I mean greater than 4000 pixels per dimension and in color.

MrSID (Multi-resolution Seamless Image Database) is a wavelet-based
compression scheme. This is the next generation after JPEG. In fact the
newer JPEG2000 format uses wavelet-based compression. Instead of
encoding small blocks, they encode the entire picture as a whole, thus
eliminating the artifacts along the edges of the blocks.

Archiving sites, such as "Library and Archives Canada" and "National
Library of Scotland" are using MrSID to store their census records and
maps respectively. It seems suitable, often a person wants to zoom in on
a census or historical map to read the name but still keep the entire image.

Here's a comparison in sizes of one particular file:
3484 x 2663 x 8bpp (greyscale)
sid 917,992
tiff 9,291,714
tiff.bz2 6,326,589
jpeg 1,592,501 (lossy, of course)
jp2 5,088,468 (lossless)
jp2 3,236,897 (lossy)
gif 8,981,277

Actually, in researching this answer, I discovered that jpeg2000 is also
wavelet based. I tried it out, but the above numbers show that
lossless jp2 isn't much better than bzip'ed tiff. Because the image is
256 color greyscale, I tried gif as well, just to see. I'd like to see
the numbers for a really big color image -- add to todo list. None of
the above are close to sid.

What I'm looking for is a file format that is:
1) open and non-proprietary - to allow linux hacks
2) compresses well - ala above numbers
3) lossless - so I can make permanent archives of my own documents

I realize that I've run on. However, as my first post was too terse, I
thought I'd expand upon it in my reply.

Duane Evenson
Edmonton, Alberta, Canada


David Harper wrote:

Duane Evenson wrote:

I'm running linux and the MrSID viewer sucks -- besides having to
open up a separate viewer. I also want to be able to store document
images in the same format, and not pay $3000 for a MrSID image
encoder. Right now I'm struggling with bloated tiff and pnm files.
Does anyone know of a comparable, open source, lossless file format?


JPEG is the standard format for photographs, though it is not lossless.

PNG is the standard for most other applications, such as icons and line
art. It *is* lossless but it uses compression techniques to shrink the
file size right down. It's the patent-free replacement for GIF.

If you're looking for a flexible and powerful image viewer and
converter, use the Gimp. It's part of most standard Linux distros, and
even if you don't have it, it's open source so you can get it from
http://www.gimp.org

Hope this helps.

David Harper
Cambridge, England

[David Harper]

Duane Everson wrote:

Here's a comparison in sizes of one particular file:
3484 x 2663 x 8bpp (greyscale)
sid 917,992
tiff 9,291,714
tiff.bz2 6,326,589
jpeg 1,592,501 (lossy, of course)
jp2 5,088,468 (lossless)
jp2 3,236,897 (lossy)
gif 8,981,277

So the raw image array is 9,275,229 bytes and the MrSID format
compresses it by a factor of more than 10. Is this *really* lossless
compression? I find it hard to believe that information isn't being
lost, even though you may not be able to perceive it.

Given that MrSID and JPEG2000 both use wavelets, I'd expect both formats
to compress the image losslessly to roughly the same size, yet
apparently MrSID still achieves a factor 5 improvement over JPEG2000.

I used to deal with large image files when I was a professional
astronomer, so I've experimented with image compression in the past.

What were the characteristics of the image in your example file? Was it
a document which has high contrast between the background and the
writing? In this case, perhaps the MrSID encoder is smart enough to
recognise that 99.99% of the information in the image is held in the
writing, which occupies a relatively small number of pixels in the raw
image, and it can selectively increase the image quality in those
regions and decrease it when it's encoding the "white" background.

Alas, the LizardTech web site is very coy about the details of MrSID. At
least, I didn't find any useful technical information there.

David Harper
Cambridge, England

[Doug McDonald]

David Harper wrote:

Alas, the LizardTech web site is very coy about the details of MrSID. At
least, I didn't find any useful technical information there.

The compression used in the version of MrSID used by Ancestry
for the census is exceedingly bad. It is quite obviously
a very lossy compression scheme.

There are versions of JPEG that are lossless and
still compress quite well.

Doug McDonald

[Dale DePriest]

Doug McDonald wrote:

David Harper wrote:


Alas, the LizardTech web site is very coy about the details of MrSID.
At least, I didn't find any useful technical information there.



The compression used in the version of MrSID used by Ancestry
for the census is exceedingly bad. It is quite obviously
a very lossy compression scheme.

There are versions of JPEG that are lossless and
still compress quite well.

Doug McDonald

The only version of Jpeg that is truly lossless is gray scale.

Dale
--
_ _ Dale DePriest
/`) _ // http://users.cwnet.com/dalede
o/_/ (_(_X_(` For GPS and GPS/PDAs

[Dennis Lee Bieber]

On Sun, 20 Mar 2005 15:41:36 -0800, Dale DePriest
<[email protected]> declaimed the following in
soc.genealogy.computing:

The only version of Jpeg that is truly lossless is gray scale.

From what I know, even grey-scale is lossy in JPEG, given that

part of the "compression" is to throw out higher frequency terms in the
polynomial generated from the picture data.

I believe I've read of a "JPEG" format that isn't lossy
(JPEG2000?) -- but it also isn't compatible with "regular" JPEG decoding
logic.

--

==============================================================
[email protected] | Wulfraed Dennis Lee Bieber KD6MOG
[email protected] | Bestiaria Support Staff
==============================================================
Home Page: <http://www.dm.net/~wulfraed/
Overflow Page: <http://wlfraed.home.netcom.com/

[Richard Cleavelad]

Dennis Lee Bieber wrote:

On Sun, 20 Mar 2005 15:41:36 -0800, Dale DePriest
[email protected]> declaimed the following in
soc.genealogy.computing:



The only version of Jpeg that is truly lossless is gray scale.


From what I know, even grey-scale is lossy in JPEG....

The image editing software I use - LView and IrfanView - each have an
option for JPG saves that can be set for various degrees of compression,
all the way from about 10 to 100 - the latter being no compression. I
suspect other image editors also have such an option - one just has to
look for it - the ones I use don't make it very plainly visible;
IrfanView brings it up as a separate window when a JPG save is chosen.

[Dennis Lee Bieber]

On Mon, 21 Mar 2005 14:09:52 GMT, Richard Cleavelad
<[email protected]> declaimed the following in
soc.genealogy.computing:

The image editing software I use - LView and IrfanView - each have an
option for JPG saves that can be set for various degrees of compression,
all the way from about 10 to 100 - the latter being no compression. I

From the Photoshop help file:

PS> A JPEG image is automatically decompressed when opened. A higher
PS> level of compression results in lower image quality, and a lower
level
PS> of compression results in better image quality. In most cases, the
PS> Maximum quality option produces a result indistinguishable from the
PS> original.

"Indistinguishable" is not the same as "identical".

--

==============================================================
[email protected] | Wulfraed Dennis Lee Bieber KD6MOG
[email protected] | Bestiaria Support Staff
==============================================================
Home Page: <http://www.dm.net/~wulfraed/
Overflow Page: <http://wlfraed.home.netcom.com/

[Richard Cleavelad]

Dennis Lee Bieber wrote:

On Mon, 21 Mar 2005 14:09:52 GMT, Richard Cleavelad
[email protected]> declaimed the following in
soc.genealogy.computing:



The image editing software I use - LView and IrfanView - each have an
option for JPG saves that can be set for various degrees of compression,
all the way from about 10 to 100 - the latter being no compression. I


From the Photoshop help file:

PS> A JPEG image is automatically decompressed when opened. A higher
PS> level of compression results in lower image quality, and a lower
level
PS> of compression results in better image quality. In most cases, the
PS> Maximum quality option produces a result indistinguishable from the
PS> original.

"Indistinguishable" is not the same as "identical".

Sounds like an opportunity for some tests. If you make some, please let
us know the results.

I made some tests with pictures using LView, and was astonished that
zooming in on 75% compression pictures they were "indistinguishable"
from the original bmp from a scanner. Did it several times because I
couldn't believe it.

I'm convinced that, for practical purposes, saving with 75% compression
is adequate (for pictures - line drawings are different).

Dick

[singhals]

Dennis Lee Bieber wrote:

On Mon, 21 Mar 2005 14:09:52 GMT, Richard Cleavelad
[email protected]> declaimed the following in
soc.genealogy.computing:



The image editing software I use - LView and IrfanView - each have an
option for JPG saves that can be set for various degrees of compression,
all the way from about 10 to 100 - the latter being no compression. I


From the Photoshop help file:

PS> A JPEG image is automatically decompressed when opened. A higher
PS> level of compression results in lower image quality, and a lower
level
PS> of compression results in better image quality. In most cases, the
PS> Maximum quality option produces a result indistinguishable from the
PS> original.

"Indistinguishable" is not the same as "identical".

While that's unarguably true, is it relevant? (g) I mean, no copy is
ever going to be identical with the original, because if it were, you'd
have two originals?

Cheryl

[David Harper]

Richard Cleavelad wrote:

I made some tests with pictures using LView, and was astonished that
zooming in on 75% compression pictures they were "indistinguishable"
from the original bmp from a scanner. Did it several times because I
couldn't believe it.

I'm convinced that, for practical purposes, saving with 75%
compression is adequate (for pictures - line drawings are different).

Line drawings suffer from conversion to JPEG because sharp edges become
fuzzy at anything more than minimal levels of compression.

I suspect that digitised images of hand-written documents such as vital
records also suffer badly from over-zealous JPEG compression, especially
those that are already difficult to read because of faded ink.

Slight digression ...

If you've been following the NASA Cassini mission to Saturn, you'll have
been impressed by the amazing pictures that it's sending back (go to
http://ciclops.org/ if you haven't seen the pictures!). The cameras on
Cassini can send back pictures in raw form i.e. exactly as they were
digitised, or compressed using a JPEG-like method.

The scientists who are interested in looking at the very fine detail in
Saturn's rings, or in hunting for hitherto-unknown small moons, insist
on getting the raw images, because the JPEG-like compression introduces
artefacts into the images which can hide or mimic the very details that
they are looking for.

David Harper
Cambridge, England

[Dennis Lee Bieber]

On Mon, 21 Mar 2005 17:33:35 -0500, singhals <[email protected]>
declaimed the following in soc.genealogy.computing:

While that's unarguably true, is it relevant? (g) I mean, no copy is
ever going to be identical with the original, because if it were, you'd
have two originals?

Uh.. If you make a copy of a digital file (without processing

the file by some other application) you DO get an identical digital
file...

Or do you get random noise in your emails? Do you get random
changes in your genealogy data? [disk drives work in multiples of
512bytes for all data transfers -- writing a 50 byte comment in a data
field requires reading the entire 512 bytes, changing 50, and then
writing the 512 byte block](Oh, and if you're running W9x with a large
drive, the OS itself is working in chunks that might be all of 8Kbytes
or larger -- each change is making a copy, just putting it in the same
spot of the disk).

I suspect not, even though most modems perform some sort of compression
of packets when sending.

JPEG looks at blocks of the image, converts them to an equation
(I've been referring to it as a polynomial, but it may be some format),
then throws away the higher frequency terms of the equation. At maximum
compression (this is a reductio ad absurdum example, perhaps), a
checkerboard of black/white squares will have only one term left --
middle grey. No white or black will appear in the saved file.

TIFF, PNG, BMP do /not/ throw out any data. Loading such into an
application, and saving it as a new file, creates a file with the very
same data (may have different overhead, but the raw pixel data is
identical).

Now, making a SCAN of a paper document /will/ result in some
loss -- but that loss takes place in the conversion from analog (light
reflection from paper) to digital... It is not in the saving of the
digital data, unless you save in JPEG.

A grey-scale GIF also does not lose any details, since it holds
256 possible shades, and 8-bit grey scale is only 256 shades. Color data
has to lose colors to save as GIF, unless the original has less than 256
total color/shades. Line art (blue-prints, for example) work great as
GIF.

--

==============================================================
[email protected] | Wulfraed Dennis Lee Bieber KD6MOG
[email protected] | Bestiaria Support Staff
==============================================================
Home Page: <http://www.dm.net/~wulfraed/
Overflow Page: <http://wlfraed.home.netcom.com/

[singhals]

Dennis Lee Bieber wrote:

Do you get random
changes in your genealogy data?

You have clearly forgotten my battle with the genie programs I used for
my book. (g)

Cheryl

[Doug McDonald]

David Harper wrote:

If you've been following the NASA Cassini mission to Saturn, you'll have
been impressed by the amazing pictures that it's sending back (go to
http://ciclops.org/ if you haven't seen the pictures!). The cameras on
Cassini can send back pictures in raw form i.e. exactly as they were
digitised, or compressed using a JPEG-like method.

They probably send the uncompressed images as something like a JEPG
image with the numerical difference between the decompressed jpeg and
the original sent as something like pkzip lossless compression.

Doug McDonald

[David Harper]

Doug McDonald wrote:

They probably send the uncompressed images as something like a JEPG
image with the numerical difference between the decompressed jpeg and
the original sent as something like pkzip lossless compression.

No, the uncompressed images are sent using Huffman encoding, which
achieves a 2:1 compression ratio in the best cases. It's lossless, so
the image that arrives back at Earth is exactly what the camera saw.

Full details are in a paper in volume 115 of Space Science Reviews, if
you're interested.

But I digress somewhat from genealogy

David Harper
Cambridge, England

[Doug McDonald]

David Harper wrote:

Doug McDonald wrote:

They probably send the uncompressed images as something like a JEPG
image with the numerical difference between the decompressed jpeg and
the original sent as something like pkzip lossless compression.


No, the uncompressed images are sent using Huffman encoding, which
achieves a 2:1 compression ratio in the best cases. It's lossless, so
the image that arrives back at Earth is exactly what the camera saw.

I am somewhat surprised! I would certainly not do that, assuming that
the spacecraft has the memory and power to do otherwise. Using JPEG
on the image, uncompressing, subtracting the uncompressed and
original image, and doing Huffman on the difference would likely
result in a smaller file. The final result back on Earth
would, of course, be identical, barring transmission mistakes.

Doug McDonakld

[David Harper]

Doug McDonald wrote:

I am somewhat surprised! I would certainly not do that, assuming that
the spacecraft has the memory and power to do otherwise.

Ah, but the on-board memory is limited, as is the bandwidth between the
spacecraft and Earth.

Using JPEG on the image, uncompressing, subtracting the uncompressed
and original image, and doing Huffman on the difference would likely
result in a smaller file.

Many of the images have large regions of dark sky, upon which Huffman is
quite effective.

David Harper
Cambridge, England

[Dale DePriest]

Richard Cleavelad wrote:

Dennis Lee Bieber wrote:

On Sun, 20 Mar 2005 15:41:36 -0800, Dale DePriest
[email protected]> declaimed the following in
soc.genealogy.computing:



The only version of Jpeg that is truly lossless is gray scale.


From what I know, even grey-scale is lossy in JPEG....


The image editing software I use - LView and IrfanView - each have an
option for JPG saves that can be set for various degrees of compression,
all the way from about 10 to 100 - the latter being no compression. I
suspect other image editors also have such an option - one just has to
look for it - the ones I use don't make it very plainly visible;
IrfanView brings it up as a separate window when a JPG save is chosen.

While these do affect compression the settings are called Quality
settings. Even at 100 percent quality there are some losses, just not as
many. Take the same image and safe it, load it, resave it, load it, etc.
about a 100 times and you will easily see the degradation. This is why
Infanview has a special rotate mode for jpg images to be able to rotate
the image without resaving it via the nor jpg save.

Dale
--
_ _ Dale DePriest
/`) _ // http://users.cwnet.com/dalede
o/_/ (_(_X_(` For GPS and GPS/PDAs

[Ralph Page]

"Dale DePriest" <[email protected]> wrote in message
news:[email protected]...

Richard Cleavelad wrote:

Dennis Lee Bieber wrote:

On Sun, 20 Mar 2005 15:41:36 -0800, Dale DePriest
[email protected]> declaimed the following in
soc.genealogy.computing:



The only version of Jpeg that is truly lossless is gray scale.


From what I know, even grey-scale is lossy in JPEG....


The image editing software I use - LView and IrfanView - each have
an option for JPG saves that can be set for various degrees of
compression, all the way from about 10 to 100 - the latter being no
compression. I suspect other image editors also have such an
option - one just has to look for it - the ones I use don't make it
very plainly visible; IrfanView brings it up as a separate window
when a JPG save is chosen.

While these do affect compression the settings are called Quality
settings. Even at 100 percent quality there are some losses, just
not as many. Take the same image and safe it, load it, resave it,
load it, etc. about a 100 times and you will easily see the
degradation. This is why Infanview has a special rotate mode for jpg
images to be able to rotate the image without resaving it via the
nor jpg save.

Dale

See:
http://www.hpl.hp.com/loco/

It appears that there is a lossless 'flavor' of JPEG developed by HP,
JPEG-LG, and the webpage refers to an 'old lossless JPEG' that was
basically just huffman coding with some twist.

-Ralph Page

[Dale DePriest]

Ralph Page wrote:

"Dale DePriest" <[email protected]> wrote in message
news:[email protected]...


Richard Cleavelad wrote:


Dennis Lee Bieber wrote:


On Sun, 20 Mar 2005 15:41:36 -0800, Dale DePriest
[email protected]> declaimed the following in
soc.genealogy.computing:




The only version of Jpeg that is truly lossless is gray scale.


From what I know, even grey-scale is lossy in JPEG....


The image editing software I use - LView and IrfanView - each have
an option for JPG saves that can be set for various degrees of
compression, all the way from about 10 to 100 - the latter being no
compression. I suspect other image editors also have such an
option - one just has to look for it - the ones I use don't make it
very plainly visible; IrfanView brings it up as a separate window
when a JPG save is chosen.

While these do affect compression the settings are called Quality
settings. Even at 100 percent quality there are some losses, just
not as many. Take the same image and safe it, load it, resave it,
load it, etc. about a 100 times and you will easily see the
degradation. This is why Infanview has a special rotate mode for jpg
images to be able to rotate the image without resaving it via the
nor jpg save.

Dale


See:
http://www.hpl.hp.com/loco/

It appears that there is a lossless 'flavor' of JPEG developed by HP,
JPEG-LG, and the webpage refers to an 'old lossless JPEG' that was
basically just huffman coding with some twist.

-Ralph Page

Certainly true but not jpeg files! JPEG is a group - not a format. The
ones with the .jpg extension cannot do lossless. The group defined some
other formats that were lossless but .jpg isn't it.

Dale

--
_ _ Dale DePriest
/`) _ // http://users.cwnet.com/dalede
o/_/ (_(_X_(` For GPS and GPS/PDAs