Category Archives: Linux

Debian, ProFTPD, FTPS, TLS, SSL, and SSL23_GET_SERVER_HELLO:unknown protocol

Recently I needed to test against an FTPS server. No big deal, I thought to myself, I’ll just set one up real quick. Boy did I end up having a hard time with that. Not because the task was actually hard but because there’s a bit of a general haziness about the whole idea of what FTPS is. More on that later.

The first thing I did was setup my Debian ProFTPD server via the included /etc/proftpd/tls.conf. Restarted ProFTPD, and then tried curl -v -v -k ‘ftps://localhost’ which immediately resulted in the following error

* About to connect() to localhost port 990 (#0)
*   Trying 127.0.0.1... Connection refused
* couldn't connect to host
* Closing connection #0
curl: (7) couldn't connect to host

* About to connect() to localhost port 990 (#0)

* Trying 127.0.0.1... Connection refused

* couldn't connect to host

* Closing connection #0

curl: (7) couldn't connect to host

Oh, right, It’s listening on port 21 not port 990… curl -v -v -k ftps://localhost:21/ which gave me this error

* About to connect() to localhost port 21 (#0)
*   Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 21 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
* Closing connection #0
curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

* About to connect() to localhost port 21 (#0)

* Trying 127.0.0.1... connected

* Connected to localhost (127.0.0.1) port 21 (#0)

* successfully set certificate verify locations:

* CAfile: none

CApath: /etc/ssl/certs

* SSLv3, TLS handshake, Client hello (1):

* error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

* Closing connection #0

curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

Believe it, or not, I got stuck here for more than an entire day. Which is kind of embarrassing. I googled the hell out of this issue, and got lots of advice which centered about generating appropriate certs, and using “openssl s_client -connect 127.0.0.1:21″ to test (which resulted in, essentially, the same error: “14996:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:607:” )

With the help of a friend from work we found what I had been overlooking. You see FTPS can mean one of two very different things.

FTPS can mean FTP with explicit SSL. This is where you connect to FTP, then give a command to encrypt the session after the initial plaintext connection has been established.

FTPS can also mean FTP with implicit SSL. This is where you connect to the ftp server and the connection is encrypted before any commands are sent (this is like having HTTP on port 80 and HTTPS on port 443, except using 21 and 990 for FTP.)

The two types of FTPS are not compatible with one another. Apparently FTPS/Implicit is no longer a part of the standard, but still “around” and “supported” by “things”. And curl thinks you mean this when you give it a url of ftps://something. FTPS/Implicit is also the kind of stream that “openssl s_client -connect 127.0.0.1:21″ would test. FTPS/Implicit is not the configuration setup by /etc/proftpd/tls.conf. Which is why my testing failed, frustratingly, for so long.

Since ProFTPD uses FTPS/Explicit by default… how do you test? With very similar commands to the ones I used previously (lending to the confusion…)

openssl s_client -connect 127.0.0.1:21 -starttls ftp

1	openssl s_client -connect 127.0.0.1:21 -starttls ftp

curl -v -v -k --ftp-ssl ftp://localhost:21/

1	curl -v -v -k --ftp-ssl ftp://localhost:21/

Ok. Now I’m able to setup and test an FTP/E server. What if I also need to setup and test an FTP/I server too? Thats pretty simple. in ProFTPD 1.3.3rc2, the mod_tls module was enhanced to support implicit FTPS via the UseImplicitSSL TLSOption. So by adding “TLSOption UseImplicitSSL” on an appropriately new version of ProFTPD and mod_tls you can have a server that works with “curl -v -v -k ftps://localhost:21/” and “openssl s_client -connect 127.0.0.1:21″

I hope that this saves someone else the headaches that going through all of this gave me. Had I read through the ProFTPD TLS howto carefully, instead of just searching for what I thought I needed, I would have solved this all much more quickly.

AirPort ♡ Free IPV6 From TunnelBroker.net

Leave a reply

After signing up for a (free) account on tunnelbroker.net and creating a (free) tunnel with my ipv4 address as the endpoint I was able to easily configure my AirPort Extreme. View your tunnel, then click on “Example Configurations” and then “Apple Airport.”

In TCP/IP prefs for my MacBooks Network/AirPort Preferences I have “Configure IPv6″ set to “Automatically” Then BOOM “ping6 en.blog.wordpress.com” works just fine.

IPv6 without needing my ISPs support and it didn’t cost me an extra dime. Happy World IPv6 Day

I approve, Mr. Kovács, I approve

Leave a reply

This list… Know it… Live it… Love it…

Via

Linux AFP server going slow?

2 Replies

Recently I found that it was taking appauling amounts of time to transfer files to/from my debian file server setup with netatalks AFP. I upgraded to the latest version to no avail, but this trick helped out quite a lot — http://www.netafp.com/tn003-slow-afp-read-performance-90/

Incidentally I believe, unscientifically, that it also helped my OnLive game latency quite a lot (on OSX)

video of a simple stress test of elockd

Leave a reply

in case anyone is curious… elockd-in-action

elockd

1 Reply

I’ve been working on an erlang port of my php locking daemon in erlang (which is a more appropriate language for this kind of thing.) And I have it all tricked out (ok partially tricked out but hey it’s my first real erlang project and i’ve only spent 2 afternoons on it.)

The api is completely the same between the two (read: simple), and it works great (in my tests.) It supports both exclusive and shared locks, orphaning on disconnect works great for both, stats are working, it’s all running under a supervisor to restart anything that stops, I *think* i’ve done the code well enough that hot code swapping should work as expected. I know there’s a lot of “how an erlang application is packaged” stuff that I don’t know yet.

If i had to describe in a one-liner what this does i would say that lockd is MySQL named locks meets Memcached.

I’m kind of annoyed, however, that “erl -s esuper” doesn’t run the stupid thing, I have to actually run esuper:start(). to get it going. I’ll have to figure that out. You would think that running some precompiled beams/modules/functions/args would be super easy from the outside a-la-init-script, and it probably is but I’m missing something.

Comments on the code are welcome. It’s a pretty cool little thing — my [lack of] mad erlang skills aside.

When I’m ready I’ll be testing it in production, and putting it in a public repo.

Using PHP and OpenSSH with username/password auth

Leave a reply

It turns out that this is actually a tricky problem. It’s super easy to use the OpenSSH command line stuff via PHP when you have key based authentication set up, but it’s not at all easy to use when you want to go the user/pass route. This is for a couple of reasons:

First you cannot specify the password on the command line. Second you cannot use the php process controls directly to give the password (well this isn’t 100% true, if you want to recompile your PHP binary with pty support then you probably could bypass everything I’m about to say and just use proc_open straight). And there’s a third reason that I’ll get to in a bit.

OpenSSH supports getting a password from an executable program via the SSH_ASKPASS environment variable — with two notable gotchas. First this only works if you also specify a DISPLAY environment variable, and second it does NOT work if the controlling process has a tty or pty.

The code below works… but if you just paste it into a script file and run it directly with php ./myscript.php it fails. Why?

function ssh_user_pass_port_forward( $hostname, $username, $password, $localport, $remotehost, $remoteport ) {
	$descriptorspec = array(
	        0 => array("pipe", "r"),  // stdin is a pipe that the child will read from
	        1 => array("pipe", "w"),  // stdout is a pipe that the child will write to
	        2 => array("pipe", "w")   // stderr is a file to write to
	);
	$script = tempnam( '/tmp/', 'askpass-');
	file_put_contents( 
		$script, 
		"#!/bin/bash
echo -n ".escapeshellarg( $password ) 
	);
	chmod( $script, 0755 );
	$env = array( 
		'DISPLAY' => '0', 
		'SSH_ASKPASS' => $script 
	);
	$forward = sprintf(
		"%d:%s:%d",
		$localport,
		$remotehost,
		$remoteport
	);
	$command = sprintf(
		"/usr/bin/ssh -n -o StrictHostKeyChecking=no -l %s -L %s -N %s",
		escapeshellarg( $username ),
		escapeshellarg( $forward ),
		escapeshellarg( $hostname )
	);
	return proc_open( $command, $descriptorspec, $pipes, getcwd(), $env);
}

function ssh_user_pass_port_forward( $hostname, $username, $password, $localport, $remotehost, $remoteport ) {

$descriptorspec = array(

0 => array("pipe", "r"), // stdin is a pipe that the child will read from

1 => array("pipe", "w"), // stdout is a pipe that the child will write to

2 => array("pipe", "w") // stderr is a file to write to

);

$script = tempnam( '/tmp/', 'askpass-');

file_put_contents(

$script,

"#!/bin/bash

echo -n ".escapeshellarg( $password )

);

chmod( $script, 0755 );

$env = array(

'DISPLAY' => '0',

'SSH_ASKPASS' => $script

);

$forward = sprintf(

"%d:%s:%d",

$localport,

$remotehost,

$remoteport

);

$command = sprintf(

"/usr/bin/ssh -n -o StrictHostKeyChecking=no -l %s -L %s -N %s",

escapeshellarg( $username ),

escapeshellarg( $forward ),

escapeshellarg( $hostname )

);

return proc_open( $command, $descriptorspec, $pipes, getcwd(), $env);

}

The answer is that you are running it, and you’re running it from a shell which has a tty or pty attached, and the script inherits those and OpenSSH sees this and then ignores the SSH_ASKPASS variable completely. The trick to testing/using this code is to execute it without running it from a terminal. If you execute the command below (roughly) it looks like it hangs. but if you open another terminal you should be able to use the port forward just created.

ssh myserver.com “php /path/to/myscript.php”

This is because since you’re sshing for the sole purpose of running that command and not using a shell the system doesn’t allocate you a pty like it would normally. I’m guessing that executing it via an http request would do something similar.

Since I previously posted some hard-won advice for working with ssh2_* in php I thought I would share this equally tricky bit that I’d also figured out in the process.

Remember that there is ALWAYS more than one way to skin a problem, and every problem can be skinned with enough effort.

Again, this is itch-scratch-ware, YMMV, this is meant as a starting point on a journey to a solution and not a drop-in-works-everywhere bit of code. It’s just the hard stuff. The useful stuff is still up to you

PHP SSH2 code

6 Replies

I’ve had a need to use the PHP SSH2 PECL recently (working on making a product, at work, more efficient) And thought I would share some of the preliminary code. You can find it here: vpssh.phps

The most interesting thing is not vpssh_core or it’s exec (though it’s good code) the really interesting thing is the vpssh_tunnel class and the accompanying examples at the top of the file. This really shows some advanced usage of ssh2_tunnel that you can’t really find anywhere else.

It’s just the beginnings of some useful code, but it’s probably a huge jumping off point for anyone seriously looking into the ssh2 pecl functionality. Oh, it also works with both password and key based authentication.

This code is less than 12 hours old, and it works for me so far, YMMV. Feedback welcome … or not… Whatever. Hope it helps someone out. And I hope it helps me out later when I need this kind of thing again.

php debugging the really really hard way

Leave a reply

If you’re ever in a situation where something is only happening intermittently, and only on a live server, and only while it’s under load… Lets say its not generating any error_log or stderr output, and you cant run it manually to reproduce… (we’ve all been in this situation) How do you get any debugging output at all?

Step 1: add this to the top of your entry point php file

if ( $_SERVER['REMOTE_ADDR'] == '127.0.0.1' ) {
       error_log( ' :: ' . getmypid() );
       sleep( 10 );
}

if ( $_SERVER['REMOTE_ADDR'] == '127.0.0.1' ) {

error_log( ' :: ' . getmypid() );

sleep( 10 );

}

Step 2: use curl on the localhost to make the request

Step 3: (this assumes your error log is /tmp/php-error-output) run the following command in a second (root) terminal window

strace -p $(tail -n 1000 /tmp/php-error-output | grep ' :: ' | tail -n 1 | sed -r s/'^.+ :: '//g) -s 10240 2>&1

1	strace -p $(tail -n 1000 /tmp/php-error-output \| grep ' :: ' \| tail -n 1 \| sed -r s/'^.+ :: '//g) -s 10240 2>&1

Good luck…

Quality Time With Your JPEGs

6 Replies

When working with user provided images in PHP you run into a problem. Lets say that you want to generate thumbnails of uploaded JPEGs for users. This is a fairly common use case where you would employ PHP and GD (the most prevalent php image extension.) But when you generate the new, smaller, image what quality setting do you use? If your quality setting is too low then the image is distorted unacceptably. Likewise if your quality setting is too high then you produce a dimensionally smaller image with a larger size in bytes than the original. So what do you do when you want to satisfy all cases? Well the obvious answer is that you should use the same JPEG quality setting that the image had when it was uploading. Now… Using PHP and GD tell me how you accomplish this.

Go ahead

I’ll wait

You can’t, can you? If you’re really sneaky you might be thinking about just pulling the data out of the binary stream, and if you’re a linux nut you’re probably sitting there muttering “just use identify (a la imagemagick)”. Of course if you try that under heavy traffic you’ll soon discover that it kills your servers. Just for reference I’ll share that code with you (not everyone is serving 2g/sec in dynamic image traffic after all). We already have the binary data in ram in the $rval variable, in case you were curious.

// define how we will deal with stdin, stdout, and stderr 
$descriptorspec = array( 0 => array( "pipe", "r" ), 1 => array( "pipe", "w" ), 2 => array( "pipe", "a" ) );
// run identify, -verbose required, - means read file from stdin
$process = proc_open( IDENTIFY . ' -verbose -', $descriptorspec, $pipes );
if ( false !== $process ) {
	// pipe the image data through it to its stdin
	fwrite( $pipes[0], $rval );
	// close stdin to allow identify to process
	fclose( $pipes[0] );
	// read the results of the program execution
	$results = stream_get_contents($pipes[1]);
	// clean up open file handles
	fclose( $pipes[1] );
	fclose( $pipes[2] );
	// Unix return value of 0 means success
	if ( 0 == proc_close($process) ) {
		// pull out the image quality from identify output
		if ( preg_match( '#Quality: ([0-9]+)[^0-9]#', $results, $m ) )
			$origin_quality = intval($m[1]);
		// detect when something goes wrong above
		if ( $origin_quality == 0 )
			$origin_quality = 100;
	}
}

// define how we will deal with stdin, stdout, and stderr

$descriptorspec = array( 0 => array( "pipe", "r" ), 1 => array( "pipe", "w" ), 2 => array( "pipe", "a" ) );

// run identify, -verbose required, - means read file from stdin

$process = proc_open( IDENTIFY . ' -verbose -', $descriptorspec, $pipes );

if ( false !== $process ) {

// pipe the image data through it to its stdin

fwrite( $pipes[0], $rval );

// close stdin to allow identify to process

fclose( $pipes[0] );

// read the results of the program execution

$results = stream_get_contents($pipes[1]);

// clean up open file handles

fclose( $pipes[1] );

fclose( $pipes[2] );

// Unix return value of 0 means success

if ( 0 == proc_close($process) ) {

// pull out the image quality from identify output

if ( preg_match( '#Quality: ([0-9]+)[^0-9]#', $results, $m ) )

$origin_quality = intval($m[1]);

// detect when something goes wrong above

if ( $origin_quality == 0 )

$origin_quality = 100;

}

So, since this consumes too much of our available resources (especially ram and cpu usage since identify fully decompresses and reads the image into a raw state for processing… hundreds of MB of ram, which you can limit but then it becomes unbearably slow…) that’s out. What now? If you’re really extra sneaky you’re thinking that you should be able to read the setting out of the binary data… there should be a header after all, right? Well… Yes there is a header but “quality” is not a “setting” its more a measure of how compressed the image is… which isn’t exactly recorded either… at least… not as an integer value. The compression matrix used to preform JPEGs lossy compression *IS* stored in a header.. and it turns out this is what the ImageMagick code uses to give us that quality setting. So I set out to reproduce this in PHP.

I know… I’m a masochist.

Thanks to some serious google-fu (and possibly a note in a php online doc manual relating to IPC, I don’t remember which of the two led me to the package first) I found that there’s already some code out there which deals with the nasty bits of reading raw JPEG headers (though not doing what I want with the header that I want) in the PHP JPEG Metadata Toolkit And the instructions for evaluating the header we can then pull is in the ImageMagick source code (coders/jpeg.c)

When we put the two together and modify it a bit to suite our needs (i.e. reading from the in-memory buffer, pulling just the right header from the jpeg file, etc) we get this code… and finally the ability to call $quality = get_jpeg_quality( $rval ); In my test (yea just one or two… very scientific like), this over 100 times faster than using the proc_open and executable method, uses less ram (a lot lot lot lot lot less ram) and generally just doesn’t suck as much.

< ?php

function get_jpeg_header_data( &$buff, $want=null ) { 
	$data = buffer_read( $buff, 2, true ); // Read the first two characters
	// Check that the first two characters are 0xFF 0xDA  (SOI - Start of image)
	if ( $data != "\xFF\xD8" ) {
		// No SOI (FF D8) at start of file - This probably isn't a JPEG file - close file and return;
		return false;
	}
	$data = buffer_read( $buff, 2 ); // Read the third character
	// Check that the third character is 0xFF (Start of first segment header)
	if ( $data{0} != "\xFF" ) {
		// NO FF found - close file and return - JPEG is probably corrupted
		return false;
	}
	// Cycle through the file until, one of: 
	//   1) an EOI (End of image) marker is hit,
	//   2) we have hit the compressed image data (no more headers are allowed after data)
	//   3) or end of file is hit
	$hit_compressed_image_data = FALSE;
	while ( ( $data{1} != "\xD9" ) && ( !$hit_compressed_image_data) && ( $data != '' ) ) { 
		// Found a segment to look at.
		// Check that the segment marker is not a Restart marker - restart markers don't have size or data after them
		if (  ( ord($data{1}) < 0xD0 ) || ( ord($data{1}) > 0xD7 ) ) {
			// Segment isn't a Restart marker
			$sizestr = buffer_read( $buff, 2 ); // Read the next two bytes (size)
			$decodedsize = unpack ("nsize", $sizestr); // convert the size bytes to an integer
			// Read the segment data with length indicated by the previously read size
			$segdata = buffer_read( $buff, $decodedsize['size'] - 2 );
			// Store the segment information in the output array
			if ( !$want || $want == ord($data{1}) ) {
				$headerdata[] = (object)array(  
					"SegType" => ord($data{1}),
					"SegName" => $GLOBALS[ "JPEG_Segment_Names" ][ ord($data{1}) ],
					"SegDesc" => $GLOBALS[ "JPEG_Segment_Descriptions" ][ ord($data{1}) ],
					"SegData" => $segdata 
				);
			}
		}
		// If this is a SOS (Start Of Scan) segment, then there is no more header data - the compressed image data follows
		if ( $data{1} == "\xDA" ) {
			$hit_compressed_image_data = true;
		} else {
			// Not an SOS - Read the next two bytes - should be the segment marker for the next segment
			$data = buffer_read( $buff, 2 );
			// Check that the first byte of the two is 0xFF as it should be for a marker
			if ( $data{0} != "\xFF" ) {
				// NO FF found - close file and return - JPEG is probably corrupted
				return false;
			}
		}
	}
	return $headerdata;
}

function buffer_read( &$buff, $len, $new=false ) {
	static $pointer = 0;
	if ( $new )
		$pointer = 0;
	if ( $pointer + $len > strlen( $buff ) ) {
		$len = strlen( $buff ) - $pointer;
		if ( $len < 1 )
			return null;
	}
	$data = substr( $buff, $pointer, $len );
	$pointer += $len;
	return $data;
}

// The names of the JPEG segment markers, indexed by their marker number
$GLOBALS[ "JPEG_Segment_Names" ] = array(
	0xC0 =>  "SOF0",  0xC1 =>  "SOF1",  0xC2 =>  "SOF2",  0xC3 =>  "SOF4",
	0xC5 =>  "SOF5",  0xC6 =>  "SOF6",  0xC7 =>  "SOF7",  0xC8 =>  "JPG",
	0xC9 =>  "SOF9",  0xCA =>  "SOF10", 0xCB =>  "SOF11", 0xCD =>  "SOF13",
	0xCE =>  "SOF14", 0xCF =>  "SOF15",
	0xC4 =>  "DHT",   0xCC =>  "DAC",
	0xD0 =>  "RST0",  0xD1 =>  "RST1",  0xD2 =>  "RST2",  0xD3 =>  "RST3",
	0xD4 =>  "RST4",  0xD5 =>  "RST5",  0xD6 =>  "RST6",  0xD7 =>  "RST7",
	0xD8 =>  "SOI",   0xD9 =>  "EOI",   0xDA =>  "SOS",   0xDB =>  "DQT",
	0xDC =>  "DNL",   0xDD =>  "DRI",   0xDE =>  "DHP",   0xDF =>  "EXP",
	0xE0 =>  "APP0",  0xE1 =>  "APP1",  0xE2 =>  "APP2",  0xE3 =>  "APP3",
	0xE4 =>  "APP4",  0xE5 =>  "APP5",  0xE6 =>  "APP6",  0xE7 =>  "APP7",
	0xE8 =>  "APP8",  0xE9 =>  "APP9",  0xEA =>  "APP10", 0xEB =>  "APP11",
	0xEC =>  "APP12", 0xED =>  "APP13", 0xEE =>  "APP14", 0xEF =>  "APP15",
	0xF0 =>  "JPG0",  0xF1 =>  "JPG1",  0xF2 =>  "JPG2",  0xF3 =>  "JPG3",
	0xF4 =>  "JPG4",  0xF5 =>  "JPG5",  0xF6 =>  "JPG6",  0xF7 =>  "JPG7",
	0xF8 =>  "JPG8",  0xF9 =>  "JPG9",  0xFA =>  "JPG10", 0xFB =>  "JPG11",
	0xFC =>  "JPG12", 0xFD =>  "JPG13",
	0xFE =>  "COM",   0x01 =>  "TEM",   0x02 =>  "RES",
);


// The descriptions of the JPEG segment markers, indexed by their marker number
$GLOBALS[ "JPEG_Segment_Descriptions" ] = array(
	/* JIF Marker byte pairs in JPEG Interchange Format sequence */
	0xC0 => "Start Of Frame (SOF) Huffman  - Baseline DCT",
	0xC1 =>  "Start Of Frame (SOF) Huffman  - Extended sequential DCT",
	0xC2 =>  "Start Of Frame Huffman  - Progressive DCT (SOF2)",
	0xC3 =>  "Start Of Frame Huffman  - Spatial (sequential) lossless (SOF3)",
	0xC5 =>  "Start Of Frame Huffman  - Differential sequential DCT (SOF5)",
	0xC6 =>  "Start Of Frame Huffman  - Differential progressive DCT (SOF6)",
	0xC7 =>  "Start Of Frame Huffman  - Differential spatial (SOF7)",
	0xC8 =>  "Start Of Frame Arithmetic - Reserved for JPEG extensions (JPG)",
	0xC9 =>  "Start Of Frame Arithmetic - Extended sequential DCT (SOF9)",
	0xCA =>  "Start Of Frame Arithmetic - Progressive DCT (SOF10)",
	0xCB =>  "Start Of Frame Arithmetic - Spatial (sequential) lossless (SOF11)",
	0xCD =>  "Start Of Frame Arithmetic - Differential sequential DCT (SOF13)",
	0xCE =>  "Start Of Frame Arithmetic - Differential progressive DCT (SOF14)",
	0xCF =>  "Start Of Frame Arithmetic - Differential spatial (SOF15)",
	0xC4 =>  "Define Huffman Table(s) (DHT)",
	0xCC =>  "Define Arithmetic coding conditioning(s) (DAC)",
	0xD0 =>  "Restart with modulo 8 count 0 (RST0)",
	0xD1 =>  "Restart with modulo 8 count 1 (RST1)",
	0xD2 =>  "Restart with modulo 8 count 2 (RST2)",
	0xD3 =>  "Restart with modulo 8 count 3 (RST3)",
	0xD4 =>  "Restart with modulo 8 count 4 (RST4)",
	0xD5 =>  "Restart with modulo 8 count 5 (RST5)",
	0xD6 =>  "Restart with modulo 8 count 6 (RST6)",
	0xD7 =>  "Restart with modulo 8 count 7 (RST7)",
	0xD8 =>  "Start of Image (SOI)",
	0xD9 =>  "End of Image (EOI)",
	0xDA =>  "Start of Scan (SOS)",
	0xDB =>  "Define quantization Table(s) (DQT)",
	0xDC =>  "Define Number of Lines (DNL)",
	0xDD =>  "Define Restart Interval (DRI)",
	0xDE =>  "Define Hierarchical progression (DHP)",
	0xDF =>  "Expand Reference Component(s) (EXP)",
	0xE0 =>  "Application Field 0 (APP0) - usually JFIF or JFXX",
	0xE1 =>  "Application Field 1 (APP1) - usually EXIF or XMP/RDF",
	0xE2 =>  "Application Field 2 (APP2) - usually Flashpix",
	0xE3 =>  "Application Field 3 (APP3)",
	0xE4 =>  "Application Field 4 (APP4)",
	0xE5 =>  "Application Field 5 (APP5)",
	0xE6 =>  "Application Field 6 (APP6)",
	0xE7 =>  "Application Field 7 (APP7)",
	0xE8 =>  "Application Field 8 (APP8)",
	0xE9 =>  "Application Field 9 (APP9)",
	0xEA =>  "Application Field 10 (APP10)",
	0xEB =>  "Application Field 11 (APP11)",
	0xEC =>  "Application Field 12 (APP12) - usually [picture info]",
	0xED =>  "Application Field 13 (APP13) - usually photoshop IRB / IPTC",
	0xEE =>  "Application Field 14 (APP14)",
	0xEF =>  "Application Field 15 (APP15)",
	0xF0 =>  "Reserved for JPEG extensions (JPG0)",
	0xF1 =>  "Reserved for JPEG extensions (JPG1)",
	0xF2 =>  "Reserved for JPEG extensions (JPG2)",
	0xF3 =>  "Reserved for JPEG extensions (JPG3)",
	0xF4 =>  "Reserved for JPEG extensions (JPG4)",
	0xF5 =>  "Reserved for JPEG extensions (JPG5)",
	0xF6 =>  "Reserved for JPEG extensions (JPG6)",
	0xF7 =>  "Reserved for JPEG extensions (JPG7)",
	0xF8 =>  "Reserved for JPEG extensions (JPG8)",
	0xF9 =>  "Reserved for JPEG extensions (JPG9)",
	0xFA =>  "Reserved for JPEG extensions (JPG10)",
	0xFB =>  "Reserved for JPEG extensions (JPG11)",
	0xFC =>  "Reserved for JPEG extensions (JPG12)",
	0xFD =>  "Reserved for JPEG extensions (JPG13)",
	0xFE =>  "Comment (COM)",
	0x01 =>  "For temp private use arith code (TEM)",
	0x02 =>  "Reserved (RES)",
);

/**
 * Most of this function is taken directly from the source code for imagemagick
 * and how it handles identify -verbose $filename to present you with a Quality
 * number.  This number should be considered approximate.  It's essentially
 * based upon the numbers used to perform compression on the original image
 * data... 
 *
 * See: http://www.obrador.com/essentialjpeg/headerinfo.htm
 * See: http://www.impulseadventure.com/photo/jpeg-quantization.html
 * See: http://www.impulseadventure.com/photo/jpeg-huffman-coding.html
 */
function get_jpeg_quality( &$buff ) {
	$tables = array(
		'multi' => array(
			'hash' => array(
				 1020, 1015,  932,  848,  780,  735,  702,  679,  660,  645,
				  632,  623,  613,  607,  600,  594,  589,  585,  581,  571,
				  555,  542,  529,  514,  494,  474,  457,  439,  424,  410,
				  397,  386,  373,  364,  351,  341,  334,  324,  317,  309,
				  299,  294,  287,  279,  274,  267,  262,  257,  251,  247,
				  243,  237,  232,  227,  222,  217,  213,  207,  202,  198,
				  192,  188,  183,  177,  173,  168,  163,  157,  153,  148,
				  143,  139,  132,  128,  125,  119,  115,  108,  104,   99,
				   94,   90,   84,   79,   74,   70,   64,   59,   55,   49,
				   45,   40,   34,   30,   25,   20,   15,   11,    6,    4,
					0	
			), // hash
			'sums' => array (
				 32640, 32635, 32266, 31495, 30665, 29804, 29146, 28599, 28104,
				 27670, 27225, 26725, 26210, 25716, 25240, 24789, 24373, 23946,
				 23572, 22846, 21801, 20842, 19949, 19121, 18386, 17651, 16998,
				 16349, 15800, 15247, 14783, 14321, 13859, 13535, 13081, 12702,
				 12423, 12056, 11779, 11513, 11135, 10955, 10676, 10392, 10208,
				  9928,  9747,  9564,  9369,  9193,  9017,  8822,  8639,  8458,
				  8270,  8084,  7896,  7710,  7527,  7347,  7156,  6977,  6788,
				  6607,  6422,  6236,  6054,  5867,  5684,  5495,  5305,  5128,
				  4945,  4751,  4638,  4442,  4248,  4065,  3888,  3698,  3509,
				  3326,  3139,  2957,  2775,  2586,  2405,  2216,  2037,  1846,
				  1666,  1483,  1297,  1109,   927,   735,   554,   375,   201,
				   128,     0
			 ), // sums
		), // multi
		'single' => array(
			'hash' => array(
               510,  505,  422,  380,  355,  338,  326,  318,  311,  305,
               300,  297,  293,  291,  288,  286,  284,  283,  281,  280,
               279,  278,  277,  273,  262,  251,  243,  233,  225,  218,
               211,  205,  198,  193,  186,  181,  177,  172,  168,  164,
               158,  156,  152,  148,  145,  142,  139,  136,  133,  131,
               129,  126,  123,  120,  118,  115,  113,  110,  107,  105,
               102,  100,   97,   94,   92,   89,   87,   83,   81,   79,
                76,   74,   70,   68,   66,   63,   61,   57,   55,   52,
                50,   48,   44,   42,   39,   37,   34,   31,   29,   26,
                24,   21,   18,   16,   13,   11,    8,    6,    3,    2,
                 0
			), // hash
			'sums' => array(
               16320, 16315, 15946, 15277, 14655, 14073, 13623, 13230, 12859,
               12560, 12240, 11861, 11456, 11081, 10714, 10360, 10027,  9679,
                9368,  9056,  8680,  8331,  7995,  7668,  7376,  7084,  6823,
                6562,  6345,  6125,  5939,  5756,  5571,  5421,  5240,  5086,
                4976,  4829,  4719,  4616,  4463,  4393,  4280,  4166,  4092,
                3980,  3909,  3835,  3755,  3688,  3621,  3541,  3467,  3396,
                3323,  3247,  3170,  3096,  3021,  2952,  2874,  2804,  2727,
                2657,  2583,  2509,  2437,  2362,  2290,  2211,  2136,  2068,
                1996,  1915,  1858,  1773,  1692,  1620,  1552,  1477,  1398,
                1326,  1251,  1179,  1109,  1031,   961,   884,   814,   736,
                 667,   592,   518,   441,   369,   292,   221,   151,    86,
                  64,     0
			), // sums
		), // single
	); // tables
	
	$headers = get_jpeg_header_data( $buff, 0xDB );
	if ( !is_array( $headers ) || !count( $headers ) )
		return 100;
	$header = $headers[0];	
	$quality = 0;
	if ( strlen($header->SegData) > 128 ) {
		$entry = array( 0 => array(), 1 => array() );
		foreach ( str_split( substr( $header->SegData, 1, 64) ) as $chr ) 
			$entry[0][] = ord($chr);
		foreach ( str_split( substr( $header->SegData, -64) ) as $chr )
			$entry[1][] = ord($chr);
		$sum = array_sum( $entry[0] ) + array_sum( $entry[1] );
		$qvalue = $entry[0][2] + $entry[0][53] + $entry[1][0] + $entry[1][63];
		$table = "multi";
	} else if ( strlen($header->SegData) > 64 ) {
		$entry = array( 0 => array() );
		foreach ( str_split( substr( $header->SegData, 1, 64) ) as $chr )
			$entry[0][] = ord($chr);
		$sum = array_sum( $entry[0] );
		$qvalue = $entry[0][2] + $entry[0][53];
		$table = "single";
	} else {
		return 100; // go with a safe value
	}
	for( $i = 0; $i < 100; $i++ ) {
		if ( ( $qvalue < $tables[$table]['hash'][$i] ) && ( $sum < $tables[$table]['sums'][$i] ) )
			continue;
		// error_log( "$table >> $qvalue && $sum >> " . ( $i + 1 ) );
		return $i + 1;
	}
	return 100; // go with a safe value
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

< ?php

function get_jpeg_header_data( &$buff, $want=null ) {

$data = buffer_read( $buff, 2, true ); // Read the first two characters

// Check that the first two characters are 0xFF 0xDA (SOI - Start of image)

if ( $data != "\xFF\xD8" ) {

// No SOI (FF D8) at start of file - This probably isn't a JPEG file - close file and return;

return false;

}

$data = buffer_read( $buff, 2 ); // Read the third character

// Check that the third character is 0xFF (Start of first segment header)

if ( $data{0} != "\xFF" ) {

// NO FF found - close file and return - JPEG is probably corrupted

return false;

}

// Cycle through the file until, one of:

// 1) an EOI (End of image) marker is hit,

// 2) we have hit the compressed image data (no more headers are allowed after data)

// 3) or end of file is hit

$hit_compressed_image_data = FALSE;

while ( ( $data{1} != "\xD9" ) && ( !$hit_compressed_image_data) && ( $data != '' ) ) {

// Found a segment to look at.

// Check that the segment marker is not a Restart marker - restart markers don't have size or data after them

if ( ( ord($data{1}) < 0xD0 ) || ( ord($data{1}) > 0xD7 ) ) {

// Segment isn't a Restart marker

$sizestr = buffer_read( $buff, 2 ); // Read the next two bytes (size)

$decodedsize = unpack ("nsize", $sizestr); // convert the size bytes to an integer

// Read the segment data with length indicated by the previously read size

$segdata = buffer_read( $buff, $decodedsize['size'] - 2 );

// Store the segment information in the output array

if ( !$want || $want == ord($data{1}) ) {

$headerdata[] = (object)array(

"SegType" => ord($data{1}),

"SegName" => $GLOBALS[ "JPEG_Segment_Names" ][ ord($data{1}) ],

"SegDesc" => $GLOBALS[ "JPEG_Segment_Descriptions" ][ ord($data{1}) ],

"SegData" => $segdata

);

}

// If this is a SOS (Start Of Scan) segment, then there is no more header data - the compressed image data follows

if ( $data{1} == "\xDA" ) {

$hit_compressed_image_data = true;

} else {

// Not an SOS - Read the next two bytes - should be the segment marker for the next segment

$data = buffer_read( $buff, 2 );

// Check that the first byte of the two is 0xFF as it should be for a marker

if ( $data{0} != "\xFF" ) {

// NO FF found - close file and return - JPEG is probably corrupted

return false;

}

return $headerdata;

}

function buffer_read( &$buff, $len, $new=false ) {

static $pointer = 0;

if ( $new )

$pointer = 0;

if ( $pointer + $len > strlen( $buff ) ) {

$len = strlen( $buff ) - $pointer;

if ( $len < 1 )

return null;

}

$data = substr( $buff, $pointer, $len );

$pointer += $len;

return $data;

}

// The names of the JPEG segment markers, indexed by their marker number

$GLOBALS[ "JPEG_Segment_Names" ] = array(

0xC0 => "SOF0", 0xC1 => "SOF1", 0xC2 => "SOF2", 0xC3 => "SOF4",

0xC5 => "SOF5", 0xC6 => "SOF6", 0xC7 => "SOF7", 0xC8 => "JPG",

0xC9 => "SOF9", 0xCA => "SOF10", 0xCB => "SOF11", 0xCD => "SOF13",

0xCE => "SOF14", 0xCF => "SOF15",

0xC4 => "DHT", 0xCC => "DAC",

0xD0 => "RST0", 0xD1 => "RST1", 0xD2 => "RST2", 0xD3 => "RST3",

0xD4 => "RST4", 0xD5 => "RST5", 0xD6 => "RST6", 0xD7 => "RST7",

0xD8 => "SOI", 0xD9 => "EOI", 0xDA => "SOS", 0xDB => "DQT",

0xDC => "DNL", 0xDD => "DRI", 0xDE => "DHP", 0xDF => "EXP",

0xE0 => "APP0", 0xE1 => "APP1", 0xE2 => "APP2", 0xE3 => "APP3",

0xE4 => "APP4", 0xE5 => "APP5", 0xE6 => "APP6", 0xE7 => "APP7",

0xE8 => "APP8", 0xE9 => "APP9", 0xEA => "APP10", 0xEB => "APP11",

0xEC => "APP12", 0xED => "APP13", 0xEE => "APP14", 0xEF => "APP15",

0xF0 => "JPG0", 0xF1 => "JPG1", 0xF2 => "JPG2", 0xF3 => "JPG3",

0xF4 => "JPG4", 0xF5 => "JPG5", 0xF6 => "JPG6", 0xF7 => "JPG7",

0xF8 => "JPG8", 0xF9 => "JPG9", 0xFA => "JPG10", 0xFB => "JPG11",

0xFC => "JPG12", 0xFD => "JPG13",

0xFE => "COM", 0x01 => "TEM", 0x02 => "RES",

);

// The descriptions of the JPEG segment markers, indexed by their marker number

$GLOBALS[ "JPEG_Segment_Descriptions" ] = array(

/* JIF Marker byte pairs in JPEG Interchange Format sequence */

0xC0 => "Start Of Frame (SOF) Huffman - Baseline DCT",

0xC1 => "Start Of Frame (SOF) Huffman - Extended sequential DCT",

0xC2 => "Start Of Frame Huffman - Progressive DCT (SOF2)",

0xC3 => "Start Of Frame Huffman - Spatial (sequential) lossless (SOF3)",

0xC5 => "Start Of Frame Huffman - Differential sequential DCT (SOF5)",

0xC6 => "Start Of Frame Huffman - Differential progressive DCT (SOF6)",

0xC7 => "Start Of Frame Huffman - Differential spatial (SOF7)",

0xC8 => "Start Of Frame Arithmetic - Reserved for JPEG extensions (JPG)",

0xC9 => "Start Of Frame Arithmetic - Extended sequential DCT (SOF9)",

0xCA => "Start Of Frame Arithmetic - Progressive DCT (SOF10)",

0xCB => "Start Of Frame Arithmetic - Spatial (sequential) lossless (SOF11)",

0xCD => "Start Of Frame Arithmetic - Differential sequential DCT (SOF13)",

0xCE => "Start Of Frame Arithmetic - Differential progressive DCT (SOF14)",

0xCF => "Start Of Frame Arithmetic - Differential spatial (SOF15)",

0xC4 => "Define Huffman Table(s) (DHT)",

0xCC => "Define Arithmetic coding conditioning(s) (DAC)",

0xD0 => "Restart with modulo 8 count 0 (RST0)",

0xD1 => "Restart with modulo 8 count 1 (RST1)",

0xD2 => "Restart with modulo 8 count 2 (RST2)",

0xD3 => "Restart with modulo 8 count 3 (RST3)",

0xD4 => "Restart with modulo 8 count 4 (RST4)",

0xD5 => "Restart with modulo 8 count 5 (RST5)",

0xD6 => "Restart with modulo 8 count 6 (RST6)",

0xD7 => "Restart with modulo 8 count 7 (RST7)",

0xD8 => "Start of Image (SOI)",

0xD9 => "End of Image (EOI)",

0xDA => "Start of Scan (SOS)",

0xDB => "Define quantization Table(s) (DQT)",

0xDC => "Define Number of Lines (DNL)",

0xDD => "Define Restart Interval (DRI)",

0xDE => "Define Hierarchical progression (DHP)",

0xDF => "Expand Reference Component(s) (EXP)",

0xE0 => "Application Field 0 (APP0) - usually JFIF or JFXX",

0xE1 => "Application Field 1 (APP1) - usually EXIF or XMP/RDF",

0xE2 => "Application Field 2 (APP2) - usually Flashpix",

0xE3 => "Application Field 3 (APP3)",

0xE4 => "Application Field 4 (APP4)",

0xE5 => "Application Field 5 (APP5)",

0xE6 => "Application Field 6 (APP6)",

0xE7 => "Application Field 7 (APP7)",

0xE8 => "Application Field 8 (APP8)",

0xE9 => "Application Field 9 (APP9)",

0xEA => "Application Field 10 (APP10)",

0xEB => "Application Field 11 (APP11)",

0xEC => "Application Field 12 (APP12) - usually [picture info]",

0xED => "Application Field 13 (APP13) - usually photoshop IRB / IPTC",

0xEE => "Application Field 14 (APP14)",

0xEF => "Application Field 15 (APP15)",

0xF0 => "Reserved for JPEG extensions (JPG0)",

0xF1 => "Reserved for JPEG extensions (JPG1)",

0xF2 => "Reserved for JPEG extensions (JPG2)",

0xF3 => "Reserved for JPEG extensions (JPG3)",

0xF4 => "Reserved for JPEG extensions (JPG4)",

0xF5 => "Reserved for JPEG extensions (JPG5)",

0xF6 => "Reserved for JPEG extensions (JPG6)",

0xF7 => "Reserved for JPEG extensions (JPG7)",

0xF8 => "Reserved for JPEG extensions (JPG8)",

0xF9 => "Reserved for JPEG extensions (JPG9)",

0xFA => "Reserved for JPEG extensions (JPG10)",

0xFB => "Reserved for JPEG extensions (JPG11)",

0xFC => "Reserved for JPEG extensions (JPG12)",

0xFD => "Reserved for JPEG extensions (JPG13)",

0xFE => "Comment (COM)",

0x01 => "For temp private use arith code (TEM)",

0x02 => "Reserved (RES)",

);

/**

* Most of this function is taken directly from the source code for imagemagick

* and how it handles identify -verbose $filename to present you with a Quality

* number. This number should be considered approximate. It's essentially

* based upon the numbers used to perform compression on the original image

* data...

* See: http://www.obrador.com/essentialjpeg/headerinfo.htm

* See: http://www.impulseadventure.com/photo/jpeg-quantization.html

* See: http://www.impulseadventure.com/photo/jpeg-huffman-coding.html

function get_jpeg_quality( &$buff ) {

$tables = array(

'multi' => array(

'hash' => array(

1020, 1015, 932, 848, 780, 735, 702, 679, 660, 645,

632, 623, 613, 607, 600, 594, 589, 585, 581, 571,

555, 542, 529, 514, 494, 474, 457, 439, 424, 410,

397, 386, 373, 364, 351, 341, 334, 324, 317, 309,

299, 294, 287, 279, 274, 267, 262, 257, 251, 247,

243, 237, 232, 227, 222, 217, 213, 207, 202, 198,

192, 188, 183, 177, 173, 168, 163, 157, 153, 148,

143, 139, 132, 128, 125, 119, 115, 108, 104, 99,

94, 90, 84, 79, 74, 70, 64, 59, 55, 49,

45, 40, 34, 30, 25, 20, 15, 11, 6, 4,

), // hash

'sums' => array (

32640, 32635, 32266, 31495, 30665, 29804, 29146, 28599, 28104,

27670, 27225, 26725, 26210, 25716, 25240, 24789, 24373, 23946,

23572, 22846, 21801, 20842, 19949, 19121, 18386, 17651, 16998,

16349, 15800, 15247, 14783, 14321, 13859, 13535, 13081, 12702,

12423, 12056, 11779, 11513, 11135, 10955, 10676, 10392, 10208,

9928, 9747, 9564, 9369, 9193, 9017, 8822, 8639, 8458,

8270, 8084, 7896, 7710, 7527, 7347, 7156, 6977, 6788,

6607, 6422, 6236, 6054, 5867, 5684, 5495, 5305, 5128,

4945, 4751, 4638, 4442, 4248, 4065, 3888, 3698, 3509,

3326, 3139, 2957, 2775, 2586, 2405, 2216, 2037, 1846,

1666, 1483, 1297, 1109, 927, 735, 554, 375, 201,

128, 0

), // sums

), // multi

'single' => array(

'hash' => array(

510, 505, 422, 380, 355, 338, 326, 318, 311, 305,

300, 297, 293, 291, 288, 286, 284, 283, 281, 280,

279, 278, 277, 273, 262, 251, 243, 233, 225, 218,

211, 205, 198, 193, 186, 181, 177, 172, 168, 164,

158, 156, 152, 148, 145, 142, 139, 136, 133, 131,

129, 126, 123, 120, 118, 115, 113, 110, 107, 105,

102, 100, 97, 94, 92, 89, 87, 83, 81, 79,

76, 74, 70, 68, 66, 63, 61, 57, 55, 52,

50, 48, 44, 42, 39, 37, 34, 31, 29, 26,

24, 21, 18, 16, 13, 11, 8, 6, 3, 2,

), // hash

'sums' => array(

16320, 16315, 15946, 15277, 14655, 14073, 13623, 13230, 12859,

12560, 12240, 11861, 11456, 11081, 10714, 10360, 10027, 9679,

9368, 9056, 8680, 8331, 7995, 7668, 7376, 7084, 6823,

6562, 6345, 6125, 5939, 5756, 5571, 5421, 5240, 5086,

4976, 4829, 4719, 4616, 4463, 4393, 4280, 4166, 4092,

3980, 3909, 3835, 3755, 3688, 3621, 3541, 3467, 3396,

3323, 3247, 3170, 3096, 3021, 2952, 2874, 2804, 2727,

2657, 2583, 2509, 2437, 2362, 2290, 2211, 2136, 2068,

1996, 1915, 1858, 1773, 1692, 1620, 1552, 1477, 1398,

1326, 1251, 1179, 1109, 1031, 961, 884, 814, 736,

667, 592, 518, 441, 369, 292, 221, 151, 86,

64, 0

), // sums

), // single

); // tables

$headers = get_jpeg_header_data( $buff, 0xDB );

if ( !is_array( $headers ) || !count( $headers ) )

return 100;

$header = $headers[0];

$quality = 0;

if ( strlen($header->SegData) > 128 ) {

$entry = array( 0 => array(), 1 => array() );

foreach ( str_split( substr( $header->SegData, 1, 64) ) as $chr )

$entry[0][] = ord($chr);

foreach ( str_split( substr( $header->SegData, -64) ) as $chr )

$entry[1][] = ord($chr);

$sum = array_sum( $entry[0] ) + array_sum( $entry[1] );

$qvalue = $entry[0][2] + $entry[0][53] + $entry[1][0] + $entry[1][63];

$table = "multi";

} else if ( strlen($header->SegData) > 64 ) {

$entry = array( 0 => array() );

foreach ( str_split( substr( $header->SegData, 1, 64) ) as $chr )

$entry[0][] = ord($chr);

$sum = array_sum( $entry[0] );

$qvalue = $entry[0][2] + $entry[0][53];

$table = "single";

} else {

return 100; // go with a safe value

}

for( $i = 0; $i < 100; $i++ ) {

if ( ( $qvalue < $tables[$table]['hash'][$i] ) && ( $sum < $tables[$table]['sums'][$i] ) )

continue;

// error_log( "$table >> $qvalue && $sum >> " . ( $i + 1 ) );

return $i + 1;

}

return 100; // go with a safe value

}

Right… serious suckage. Which is why I’m sharing it here so that you don’t have to go through all that trouble. You can just steal my stolen code. Aren’t GPL compatible licenses fun?

Ok, finally, you… in the back… stop jumping up and down screaming about the Imagick PECL extension… In my testing getCompressionQuality() didnt work and getImageCompression() was unavailable (though I hear there is a newer version of the extension now… YMMV)

CodeWord: Apokalyptik

The random things that spew forth from my brain…

Category Archives: Linux

Debian, ProFTPD, FTPS, TLS, SSL, and SSL23_GET_SERVER_HELLO:unknown protocol

AirPort ♡ Free IPV6 From TunnelBroker.net

I approve, Mr. Kovács, I approve

Linux AFP server going slow?

video of a simple stress test of elockd

elockd

Using PHP and OpenSSH with username/password auth

PHP SSH2 code

php debugging the really really hard way

Quality Time With Your JPEGs