I have both:
PSP (slim) and DS (lite).

Theres a noticeable difference in size in terms of the DS/DS Lite upgrade.
And there is a noticeable difference in weight in terms of the PSP/PSP Slim upgrade.

In terms of picture quality, the DS can't touch the PSP (granted, the PSP is used for movie viewing as well).

I find that the DS is a great machine for arcade/old school nintendo games as well as newer jap games that are being rought out on the market.
It focuses more on a great GAMING experience.

The PSP also offers a rewarding gaming experience, what with good sound quality (via headphones only - the regular speakers sound too tinny), ergonomic design that helps you play for hours without noticing any muscle strain and the visuals are outstanding.
If you have a PS3, you can also utilise the PSP as a wireless controller. I've tried it out a couple of times but prefer using the bigger, wireless dualshock controllers when im using the console.

All in all, you need to decide whether you're after a next gen handheld GAMING device OR you want that with something more.