DivShare, Day 1 (raw commentary)

I began looking at divshare a few days ago as a way to stor, save, and share my personal photo collection.  The idea of auto-galleries, unlimited space, flash video, and possible FTP access was… enticing.  But it’s tough to tell how something like this is going to work on a large scale…

So… after messing around with a free divshare account for a while I decided it was more worth my while to pay 10 bucks for a pro account and get FTP access than to try and use mechanize (or something similar) to hack out my own makeshift API.  Now I have about… Oh… 8,000 files I want to upload… So… doing that 10 at a time was just _NOT_ going to happen…

After paying for a pro account I was *immediately* granted FTP access, no waiting. And for that I was grateful.  Since I take photos at 6MP, and thats WAY too large for most online uses I have a shell script which automagically creates 5%, 10%, and 25% or original sized thumbnails.  This meant that I had an expansive set of files I could upload and only take a couple of hours doing it (5% thumbs end up being less than 200Mb.)  This, I thought, would be an excellent test of their interfaces.

So an-ftping-i-a-go.  Upload all my files into a sub directory (005). Visit the dash. nothing. Visit the ftp-upload-page to recheck… maybe I did something wrong. AND WHAM! an 8,000 check box form to accept ftp uploaded files… ugh.  Thankfully they’re all checked by default.  I let it load (for a long while) and hit submit… and wait… and wait… and wait.  Then the server side connection times out a while later.  Fair enough. Check my dash… about 1500 of the 8,000 photos were imported… I’m going to have to do this 6 times. Annoying, but doable.  Hit the second submit, and pop open another browser to look at my dash.  And divshare did *nothing* with my folder name… that wasnt translated to a “virtual” folder at all. tsk tsk.

So I need to put about 1500 photo, manually, into an 005 folder… and then I realize… I have to do this 20 files at a time… with no way to just show files that are not currently in a folder.

… uh no …

Ok, so I open up one of the photos that I DID put into the 005 folder, and it did, in fact, make them into a “gallery” of sorts. It made a thumbnail , and displayed all 3,000 photos side by side in something similar to an iframe… no rows. just one row… 3,000 columns… and waiting as my browser requests each… and every… thumb… from divshare. Wonderful.  The gallery controls are simple enough an iframe with a scrollbar at the bottom, a next photo link, and a previous photo link.  And all 3 controls make you loose your place in the iframe when you use them…

Now dont get me wrong. You get what you pay for. But hey… I did pay this time ;)  The service is excellent for what it does. And my use case was a bit extreme. Still I hope that they address these issues that I’ve pointed out.  I’d really like to continue using them, and if they can make my pohoto process easier I’ll gladly keep paying them $10/mo

Thats

  1. Don’t ignore what pro users are telling you when they upload
  2. Process large-accepts in the background, let me know I need to come back later
  3. Negative searching (folder == nil)
  4. Mass file controls (Iether items/page, or all-items-in-view (folder == nil))
  5. Give me a gallery a non-broadband user can use (1500 thumbs in one sitting tastes bad, more filling)
  6. Don’t undo what I’ve done in the gallery every click.  Finding your place among 8,000 photos is tedious to do once

And I know I sound like I’m just complaining. And I am. But this is web 2.0 feedback baby. Ignore my grouchiness, and (If I’m lucky) take my suggestions and run with them asap.  The photo/files market is very very far from cornered!

Sockets and PHP — Part II: Working with our connection

WooWOO! We’ve connected! “Uhh, now what?” Now we log in. This will be your first and most important step to actually WORKING with (rather then creating) socket connections.

Step 7: Working with our connection…

Now, unfortunately I can’t just JUMP into logging on without first figuring out whether we can. Depending on the service YOU are trying to connect to, this may be accomplished in a number of ways, in accordance with how the server has been set up. With our FTP server, we can read (buried DEEP within the RFCs) that every command given should evoke a response from the server; When I say “hello” to the server, it’s appropriate for the server to acknowledge me. FTP does this by giving you number codes. Anything starting with a 1, 2, or 3 (we’re going to lump all of those into one category here for the sake of brevity) means that the last command was completed successfully–OR that the server is waiting for something more. A response of 4 or 5 is BAD JUJU–we don’t want those.

We’ll build our cmd_status code based on those assumptions above

<?php

function cmd_status() {
// first we make sure the socket is set properly
$this->control_socket_reset();
// We’ll get ALL of the data from the control sockets buffer
while ( $data=fgets($this->contsock, 9999) ) {
// And from the buffer we will find the reply we want
// (in this case we are only concerned with the VERY
// first number of the VERY last line)
$reply=substr($data, 0, 1);
}
if (
$reply == 1 || $reply == 2 || $reply == 3 ) {
// Let’s give our socket some more coffee
$this->control_socket_reset();
// we make sure the socket WILL
// work properly in the future…
// (get used to redundancy… it
// can be a little work in the ‘now’
// but it saves a lot of work in
// the future

// Now let’s return TRUE
return(1);
} else {
// return FALSE, because our command didn’t do ANYTHING of any use
// at all…
return(0);
}
}

?>

Step 8: Logging In (FINALLY!)

If all has gone well so far, we are now ready to take our first steps! Let’s log in!

We’ll make our login() function like this:

<?php

function login($username=‘anonymous’, $password=[email protected]) {
$this->username=$username; // again, we tuck our variables away
$this->password=$password; // for safe keeping…
fputs($this->contsock, ‘USER ‘.$this->username.chr(10));
// WHAT IN THE HELL WAS THAT?! Well, our FTP RFC says that to
// log in we need to type ‘LOGIN <username> <carriage return>’
// which is exactly what we just did. We used the concatenation
// operator (http://www.php.net/manual/en/language.operators.string.php)
// to put those three things together (the chr() function lets you
// convert an ASCII code into its character (see http://www.asciitable.com/
// for an explanation of what ASCII is…))
if ( $this->cmd_status() ) {
// IF the command returned a GOOD code…
fputs($this->contsock, ‘PASS ‘.$this->password.chr(10));
if (
$this->cmd_status() ) {
return(
1);
} else {
return(-
1); // -1 means ‘bad password’
}
}

else {
return(0); // 0 means ‘bad username’
}
}

?>

AGAIN, it’s important to TEST TEST TEST TEST! So, let’s modify our testing code (down at the bottom of the page) again…

<?php

$ftp = new ftp;
if (
$ftp->control_socket(‘ftp.cdrom.com’, 21) ) {
echo
‘We have successfully connected!<br>’;
flush();
$test=$ftp->login();
if (
$test == 1 ) {
echo
‘We have successfully logged in!<br>’;
flush();
}

else if ( $test == 0 ) {
die(
‘Bad username!<br>’);
}

else if ( $test == –1 ) {
die(
‘Bad password!<br>’);
}
}

else {
die(‘We have failed to connect!<br>’);
}

?>

SUCCESS! We’ve successfully logged into our FTP! Congratulations–we’re almost half way there! And I decree it time to go see a movie, or go get a rental, have some popcorn, or drink some water and let loose. This stuff is hard to grasp, and if you don’t reward yourself, then you’ll have a hard time pushing on into the more exciting areas of the language. So soak it up. If you understood everything so far, then you’re WAY ahead of the curve!

Step 9: Our data connection (passive mode)

If you haven’t already got the idea of how FTP data is transmitted (and you probably don’t–you didn’t do the RFC homework I gave you, did you?!) you’ll need to understand it now, or you’ll be completely and utterly lost when we begin this phase of the project. By the way, be ready for a headache, because this is, arguably, the most difficult part of the tutorial to grasp in its entirety.

Passive mode was invented to get around firewalls. This is how it works:

a) you tell the server “enable passive mode”
b) the server says, “ok, connect to this ip and this port, and wait”
c) you connect, and wait
d) you give a command (like “LIST”) on the control port
e) you receive the data output from your list command–which you gave on the control port–from the passive data connection you just made
f) you close your passive connection (because you have to re-enable passive mode and connect when you need it again).

If you got all of that, then good–you’ve officially learned more than most people will ever know about how they get files! 🙂 If you didn’t catch it, try reading this: (which gave me MY inspiration http://neworder.box.sk/newsread.php?newsid=4056).

Here’s our data_socket() function, which is almost identical to control_socket() except that it does not keep track of the hostname and port number, because those may (the port number WILL) change each time you use passive(), and you have to use passive() every time you use your data connection 🙂

<?php

function control_socket_reset() {
if (
$this->contsock ) {
socket_set_timeout($this->contsock, 1);
// Reset (or set) the time-
// out on our control socket
return(1);
} else {
return(
0);
}
}

?>
And here’s one of our most complicated functions… passive()

<?php

function passive() {
// First we are going to ‘clear’ our buffer
while ( $tmp=fgets($this->contsock, 9999) ) {
unset(
$tmp);
}
$this->control_socket_reset();
fputs($this->contsock, ‘pasv’.chr(10));
// We should get a line which looks like this when we give this command
// ‘227 Entering Passive Mode (206,100,24,34,217,120)’
while ( $tmp=fgets($this->contsock, 512) ) {
$data.=$tmp;
}
// We need to get everything from in between ( and ), so let’s
// make those into something we can work with. We’ll probably
// not see this ‘{|}’ in that line, EVER, so we’ll use that…
$t=str_replace(‘(‘, ‘{|}’, $data);
$t=str_replace(‘)’, ‘{|}’, $t);
// Now we should have a line which looks like this:
// ‘227 Entering Passive Mode {|}206,100,24,34,217,120{|}’
// and we can ‘explode’ it into an array using that string
$te=explode(‘{|}’, $t);
// This should give us:
// $te[0] == ‘227 Entering Passive Mode’
// $te[1] == ‘206,100,24,34,217,120’
// $te[2] == ”
// That wasn’t so bad, was it?
$tem=explode(‘,’, $te[1]);
// This should give us another array which looks like this:
// $tem[0] == 206
// $tem[1] == 100
// $tem[2] == 24
// $tem[3] == 34
// $tem[4] == 217
// $tem[5] == 120
// See http://neworder.box.sk/newsread.php?newsid=4056 for an
// explanation of the next 2 steps…
$port=( ( $tem[4] * 256 ) + $tem[5] );
// You NEED to use the ip address you got with the pasv command… and here’s
// why: do an nslookup on ftp.simtel.net, and you get MANY (15 at the time
// I wrote this article) ip addresses associated with this one host name.
// When you connect to the NAME ‘ftp.simtel.net’ you get a random single ip
// from the host name. And if you tried to do it again, chances are
// that you’d get a different ip the second time. And since only one
// server (the ip that you connected to originally) is expecting you to
// connect on the passive port, you’ll find yourself knee deep in problems
// like self doubt and low self esteem!
$host=$tem[0].‘.’.$tem[1].‘.’.$tem[2].‘.’.$tem[3];
if (
$this->data_socket($host, $port) ) {
return(
1);
} else {
return(
0);
}
}
?>

And once again, like good little boys and girls, we need to test our new configuration!

<?php

$ftp=new ftp;
if (
$ftp->control_socket(‘ftp.cdrom.com’, 21) ) {
echo
‘We have successfully connected!<br>’;
flush();
$test=$ftp->login();
if (
$test == 1 ) {
echo
‘We have successfully logged in!<br>’;
flush();
if (
$ftp->passive() ) {
echo
‘We have successfully entered PASSIVE mode!<br>’;
} else {
die(
‘Passive mode could NOT be initiated!<br>’);
}
} else if (
$test == 0 ) {
die(
‘Bad username!<br>’);
} else if (
$test == –1 ) {
die(
‘Bad password!<br>’);
}
} else {
die(
‘We have failed to connect!<br>’);
}

?>

Congratulations! If you’ve gotten this fat, and understood everything that’s gone on, you’re almost a php-sockets guru! Only a little ways to go, and we’ll have a functional application using one of the least understood function sets in php!

Its nap time! Go to bed! You’ve worked hard. Come back in the morning with a clear mind, and a fresh view on life and a cup of coffee, and we can start out your day by getting to the bottom of this application and these functions!
Sleep well? GREAT! Now let’s finish everything up, and get done with this whole ordeal!

We’ll start with our directory listing. Now, here’s where most people cringe in fear at the thought of having to work with FTP, and all of those pesky socket connections, but its ALSO where YOU learn that it isn’t that difficult at all!

<?php

function ftp_list() {
fputs($this->contsock, ‘LIST’.chr(10));
// Let the FTP server know you
// are interested in a directory
// listing.
if ( $this->cmd_status() ) {
// IF the list command went through OK, then collect our data into
// a nicely packaged array
while ( $data=fgets($this->datasock, 9999) ) {
$return[]=$data;
}
// Close the data socket (because its not useful anymore. Another
// directory listing would require another passive connection.
fclose($this->datasock);
// Return our data! THE HOLY GRAIL! WE’VE MADE IT!
return($return);
} else {
// DOH!
return(0);
}
}

?>

We’ll finish everything off by being courteous to the site owners, and logging out properly…

<?php

function logout() {
// We’ll give the Micro-Crap way of logging out, and the
// handy-dandy-always-setting-standards-and-coming-out-ahead
// unix way of saying goodbye to the server…
fputs($this->contsock, ‘QUIT’.chr(10).‘BYE’.chr(10));
// Then we’ll close the control connection
fclose($this->contsock);
// And since we don’t care one way or another, we assume that
// the function succeeded!
// YAY! THE FINISH LINE!
return(1);
}

?>

And we’ll again modify our test code, and make sure that our now fully functioning, handy dandy FTP directory listing software is fully functional before we approach marketing with an idea for a great new product 😉

<?php

$ftp=new ftp;
if (
$ftp->control_socket(‘ftp.cdrom.com’, 21) ) {
echo
‘We have successfully connected!<br>’;
flush();
$test=$ftp->login();
if (
$test == 1 ) {
echo
‘We have successfully logged in!<br>’;
flush();
if (
$ftp->passive() ) {
echo
‘We have successfully entered PASSIVE mode!<br>’;
$listing=$ftp->ftp_list();
if (
is_array($listing) ) {
echo
‘<pre>’;
foreach (
$listing as $num => $file ) {
echo
‘# ‘.$num.‘ ‘.$file.‘<br>’;
}
echo
‘</pre>’;
$ftp->logout();
} else {
die(
‘So close, yet so far… no dir list!<br>’);
}
} else {
die(
‘Passive mode could NOT be initiated!<br>’);
}
} else if (
$test == 0 ) {
die(
‘Bad username!<br>’);
} else if (
$test == –1 ) {
die(
‘Bad password!<br>’);
}
} else {
die(
‘We have failed to connect!<br>’);
}

?>

WooWOO! Congratulations! We have a winner! And your prize? Well… er… um… a directory listing? OK, maybe this isn’t the MOST useful socket code ever (seeing as how there are already FTP functions in php). But in reading this article, you’ve accomplished several fundamentally important things in regards to your php coding abilities. You’ve seen how you can use object oriented programming to bundle everything into a neat package–which would have otherwise only been a spaghetti-string-consortium of functions calling one another. You’ve seen how to effectively use sockets and work with data streams (MULTIPLE data streams no less). And you’ve officially let me rant, rave, and talk, more than anyone should ever have had to bear… Congratulations (I think 😉

-Demitrious

Home page: http://www.apokalyptik.com/
I’m also available for *nix and php consulting, e-mail [email protected] if interested. Cheers!

Poor Man’s Debugger

Debugging is essentially of a way of life. Debugging keeps you sane, and provides you with peace of mind. Debugging makes it so that you don’t have to pull your hair out when something does not work. Debugging is the most direct way of finding problems in your code, and it’s exactly what I hope to help you learn to do right now.

It seems to me that, at some point in every programmer’s life, they finally get realistic, and say “That’s it… I’m not going to second guess my logic anymore. I’m going to make this stupid thing debug itself!”. I would guess that this was first said long ago, shortly after the advent of the punch card, when thousands of cards made up a simple TicTacToe computer game. I don’t think that it’s really something that programmers ever pass down to their disciples either. Throughout time it’s been something that each programmer would stumble on in their own time, and would be a milestone in their logical development as a programmer.

Building a program is a lot like playing with Legos (TM). When I was little I would build towers and castles and anything else my little imagination could come up with. Inevitably, some of them fell down, but I could easily learn from my mistakes. I could see which way the tower fell, and tell where each block was stuck together. I’m sure that most of us don’t have the capacity to imagine our programs as such – not with the detail needed to debug, at least. And that is why, when our programs fail, we are left wading through a murky puddle of half-thought-out logic, incomplete ideas, and caffeine stained notes from 4 months ago. So we have to find a way to debug.

Going about debugging a program is, actually, simpler then one might imagine. OK, to be fair, there are complicated ways of debugging but, since this is a tutorial for beginners, we’ll stick to the boneheaded basics (not that there is anything wrong with the boneheaded basics, mind you!).

Methods for debugging your code follow a couple of very distinct paths, each with its pro’s and its con’s. We have the poor man’s Debug-By-Echo, the poor man’s Debug-By- Logfile, and the poor man’s Debug-By-errorTrap. These, by the way, are just my names for the methods, not standard programming names.

Short descriptions of the three are as follows (I’ve started abbreviating them for the sake of brevity.):

DBE (Debug-By-Echo) is as simple as debugging methods come. All you are doing is spouting off information which, hopefully, is meaningful to the programmer.

DBL (Debug-By-Logging) is basically the same thing as DBE, except that it writes to a file.

DBT (Debug-By-errorTrap) is the most complicated of the three. It displays only relevant information based on checking the data we DO have versus the data that we SHOULD have. It can be the most useful, but takes more time and thought to implement.

DBE (Debug-By-Echo)

DBE is probably the simplest of the debugging methods (which often makes it the most effective!). DBE is an extremely easily implemented method which can tell you EXACTLY where something went wrong. It’s done by simply echoing some text between the lines, including the variables you are currently working with. Here’s an example:

<?php

function doStuff($num) {
if (
is_numeric($num) ) {
echo
‘[1]: Doung stuff ‘ . $num . ‘ times<br>’;
}
else {
die (
‘[1]: doStuff() called with a non integer<br>’);
}

echo ‘[1]: Begining execution loop<br>’;

$count = 0;

while ( $count < $num ) {
$count++;
echo
‘[1]: Stuff, loop #’ . $count . ‘<br>’;
// do more stuff later on…
}

echo ‘[1]: Finished execution loop<br>’;
Return
True;
}

echo ‘[0]: Begining program execution<br>’;
doStuff(7);
echo
‘[0]: Ending program execution<br>’;

?>

This example fully illustrates how to take advantage of DBE. We have a marker representing where we are in the program. In this instance I used [0] to signify that we were at the root of the code. [1] means that we are inside function one. And if I had more functions they would be [2], [3], [4], and so on. This allows us to quickly reference which part of the code is being processed.

We also output the values of all of the variables that are being worked within our loop inside dostuff(). Now, if we had a simple flaw in our logic and our while loop was not going through enough iterations, or things were happening in the wrong order, this output makes most problems easy to track down.
DBL (Debug-By-Logging)

DBL is much the same, except that instead of echo’ing, you log the information to a file (with simple fopen() and fputs() statements. The beauty of this method is that you can keep permanent records, and compare their behavior.

Let’s say something started happening in version 1.2 of a script you were writing, but did not happen in 1.1. Using DBL method you can simply look at the log files created when debugging, and find where the two versions started to disagree. This is not always true for all circumstances, but can be a very useful tool when used properly.

Now, if you are writing code to be distributed, it is sometimes desirable to allow the user to change a simple variable to turn debugging output on or off. You might have them e-mail the output you (or let you see it) which will help troubleshoot problems that other people are having with your code… you can do that quite simply like this:

<?php

// configuration Variables
// set to ‘0’ to disable debugging messages
$debug = 1;
$logfile = ‘./program.log’;

function logentry($line) {
global
$logfile;
// open our log file
$fp = @fopen($logfile, ‘a’);

//check to make sure it’s useable now…
if ( $fp ) {
// if it is – write our line, close the file,
// exit the function we’re also going to time
// stamp is so that we can see when something
// happened (useful if the program is being
// looked at for lack of speed)
fputs($fp, ‘[‘.time().‘] ‘ . $line . chr(10));
fclose($fp);
return
True;
}

else {
// if not, then something is wrong…
die(‘FATAL ERROR: COULD NOT OPEN LOG FILE!’ . chr(10));
}
}

function doStuff($num) {
global
$logfile, $debug;
if (
is_numeric($num) ) {
if (
$debug == 1 ) {
logentry(‘[1]: Doung stuff ‘ . $num . ‘ times’);
}
}
else {
logentry(‘[1]: doStuff() called with a non integer’);
die();
}
if (
$debug == 1 ) {
logentry(‘[1]: Beginning execution loop’);
}

$count = 0;

while ( $count < $num ) {
$count++;

if ( $debug == 1 ) {
logentry(‘[1]: Stuff, loop #’ .$count);
}
// do more stuff later on…
}

if ( $debug == 1 ) {
logentry(‘[1]: Finished execution loop’);
}

Return True;
}

if ( $debug == 1 ) {
logentry(‘[0]: Beginning program execution’);
}

doStuff(7);

if ( $debug == 1 ) {
logentry(‘[0]: Ending program execution’);
}

?>

It adds quite a few lines to your code, but it’s an excellent tool when you need to troubleshoot. You might consider removing the debugging from the final version to speed things up if performance is too much of an issue.

DBT (Debug-By-errorTrap)

DBT is the least common, and most complex type of debugging. It is similar to DBE (or DBL if you are logging), but incorporates a bit of intelligence – which is nice. Let’s say you have the following code:

<?php
$data
=@file('http://blah.com/index.shtml');
foreach (
$data as $line ) {
echo
$line;
}
?>

Now, file() returns you an array if successful, and foreach() expects an array, but what if you call a url or file that does not exist, or that you cannot open? Then you get an error saying “Warning: Invalid argument supplied for foreach() in /whatever/file/it/could/happen.php on line 4″. This is all fine and good, but if you’re interested in making more sense out of things, or just want to make the error look better for a client, or maybe you expect this to fail sometimes, this is when you use error trapping, like so:

<?php
$data
=@file('http://blah.com/index.shtml');
if (
is_array($data) ) {
foreach (
$data as $line ) {
echo
$line;
}
}
else {
echo
'File Not Found!';
}
?>

In this example we have thought all of the possibilities through, and provided the program with a GRACEFUL way of handling any problems. I’ve included this as a debugging technique, but realistically it’s just nothing but good programming practice. If something can go wrong, then you – as a programmer – should expect that it will, and provide for that in DBT.

In a perfect world, debugging would not be necessary (because programs would run correctly the first time) and my refrigerator would never run out of Pepsi for me to drink! If you insist on living in this world, plan on going bald a lot sooner, and renting a room with padded walls, because finding a problem without a little bit of troubleshooting is bound to drive even the most learned ‘guru’ insane!

-Demitrious

Home page: http://www.apokalyptik.com/
I’m also available for *nix and php consulting, e-mail [email protected] if interested. Cheers!

Most people wont care…

Us web 2.0 and web 3.0 people have a hard time caring about the things that normal people care about. And we have a hard time believing that people don’t care about the things that we do.  In short we’re a large group of very detached individuals who are, more or less, free to form ideas into substance in the vacuum of our own creation.

I often have a hard time coming to grips with this concept myself. WHAT DO YOU MEAN nobody will care about this idea?! It’s great.  But after a while chewing on that, I’ll grudgingly admit that while it may be a great idea… Almost nobody will care.

So when I saw, a few days ago, a bit of a fuss being kicked up over google wanting your browsing history. I surprised myself by offhandedly thinking: “nobody but us cares.” And I still think that.  As a matter of fact I think that in a utilitarian sense most everybody will embrace the idea.

The problem is in search.  Google has taken keyword search straight to the edge.  And now people are hungering for the next search. Search 4.5 beta.  And that’s relevancy.  I’m a dog lover (I have 3 large dogs) so let me give you an example from my world.

Lets assume I just got a new pupy and she’s SUPER submissive. Peeing all over, shakes, just scared.  If I go to google and type “submissive bitch”… I don’t get what I was looking for.  Now if google has my browser history and sees that I frequent the Chazhound Dog Forums now google has the information necessary to determine that I’m not looking for sex, but in fact dog related topics.

This is why, not only will they not care but, most people will embrace giving google more data.  Sure I care. You care. But lets not fool ourselves into thinking that everybody else cares too 🙂

company dinner

Well I just had a great meeting with my new co workers. I must say that I’m a lucky guy to be working with such an awesome group of interesting people. Just the idreas exchanged at dinner make it all worth while 🙂