Sep 12, 2005 04:35 pm not sure, not a bad idea though...
question about HRC member music
Posted on Sep 12, 2005 03:53 pm
fortymile
Member Since: Jan 18, 2003
db--any idea what the total megabytage of the member music on the site here is?
i keep fantasizing about a site feature that would let you select and download like handfuls of songs at a time instead of one by one. so that i could get these member songs into my ipod and really listen to them.
[ Back to Top ]
Sep 12, 2005 09:35 pm not today...a little bit of research and a lot of code would need to go into it...it is a good idea. It'll go on the "look into it when there is time" list.
Sep 12, 2005 10:29 pm the tricky part is getting songs to zip up into one download, essentially getting files to zip up on the fly server-side, then deleting itself after download is complete...it can be done, but not sure how in PHP. The quicker solution would be to have a "songs allowed for download" list in the music section and download one by one in that one place, cuz zipping up a ton of files could be one very LARGE download.
Sep 13, 2005 04:15 pm yes i was aware of these problems. the whole idea, though, is to eliminate the need to sit there dowloading them one by one.
but if that can't be done, then the page of those available to be downloaded is a good idea i guess. at least that way we could be sure we were getting them all.
or
what if you had a permanent zip file of every song that had been made available for download by current members. any time someone adds a new download-enabled song to thier profile, it's added to that zip file automatically. when people let thier memberships expire or make a song unavailable for download, those/that song is automatically removed from the zip file.
this zip file wouldnt get used very often; at first, yes, while current members upload the HRC catalogue to thier mp3 players, but after that, only new members would be tapping it. us people who already have the full set would graduate to adding songs one by one. so it might not be a huge bandwidth drain ultimately.
it could be an added incentive to join the site--the member mp3 bundle. cant get it if youre not a member.
Sep 13, 2005 05:35 pm Well, having just one zip of all available downloads wouldn't be sexy, what WOULD be sexy wouldbe a list of songs with checkboxes, check the ones you want to download and it zips just those up and gives them to you...I, for example, wouldn't want to download them, people that would would click a button to check all checkboxes and boom, all of them come down.
Also, storing a zip of all the music would take up twice the disk space, and the music on HRC is probably 80% of the overall disk space used, so it would impact that substantially.
Sep 13, 2005 07:49 pm How about keeping a queue of songs to download instead of lumping them all into 1 download?
You could queue up 5 or 10 songs and when 1 finished, the next would begin. No compression, so it'd take longer to download (unless you zip each one before sending it), but we could select several songs and come back later instead of having to select a new one every few minutes.
Eh, just a thought anyways. I'm not the guy that gets stuck with the coding :)
Sep 13, 2005 07:53 pm I don't believe I could, from a script, control client downloading insofar as beginning one when the previous ends. Also, zipping or taring the songs wouldn't really be for "compression" as audio and video zipped up generally doesn't make much difference in size, sometimes it actually winds up larger, the reason for zipping or taring would be more for the convenience of a single download over multiple downloads.
Sep 13, 2005 10:25 pm yeah, db, i see what you mean. the checkbox screen was how the idea initially appeared to me, too. i didnt think about zipping, but i see how it would work, if you could code it. that'd be awesome, were it possible.
Sep 14, 2005 06:14 am oh, I know that much is possible,just not sure how yet, never done it in PHP...I'll look into it at some point.
Sep 14, 2005 09:51 am Hey db, easiest way:
through php use exec or shell_exec and call something like:
tar -cf archive.tar file1 file2 file3
then
gzip archive.tar
Or just write a bash script.
Maybe cron it daily and remove the previous file, as it'll eat up lots of processor and HDD resources every time it runs (don't suggest running it every time someone downloads, that'd be crazy load).
W.
Sep 14, 2005 09:53 am Yeah, I know I can do it in shell...but I don't have shell right now...never needed it.
Sep 14, 2005 09:54 am Oh, but that won't cover the check box issue... Not possible with current technology unless you create a client side application.
W.
Sep 14, 2005 09:55 am Don't need shell dB, if you do it through PHP. I'm sure you have access to your crontab too, through a plesk interface or something.
W.
Sep 14, 2005 09:57 am Plesk blows, but regardless, yes I have crontab access, but, if I can't do it with selectable songs to download I won't do it at all...I am not zipping up the entire list and make it an all or nothing thing, that's lame...and a waste of space...if I can't do it customizable to the user I won't do it...and it is possible, I have seen it, just never done...
Sep 14, 2005 10:45 am Sure if you combine the files on the fly (tar them), but you'll load down the server a ton, same way I showed above, but list out the files they selected. It's not possible to chain the individual files for download.
Tar doesn't really compress the files, it just combines them, so if you skipped the gzip'n, it'd help, but still gonna be some load issues...
The client (end user) then needs a client side application to untar the file. Hence the requirement of a client side application... If they know how to do it manually, and it doesn't matter to you if they don't (they can bloody well learn ;)) then you're on your way, it's not tough at all..
You'd also need a small mechanism to handle the download, so that when it completed, you could delete the temporary tar.
W.
Sep 14, 2005 10:46 am Oh and yes, I agree, plesk blows. Straight command line all the way for me.
W.
Sep 14, 2005 10:56 am Hey, Waldo, if I have your attention for a minute, I gotta ask, have you been seeing much of this email header injection in PHP? I have a couple clients that for some reason, just filtering the user-input header info (name and email) for new line and carriage returns doesn't seem to be stopping it, I have been reading and researching like mad, and for some reason a couple sites simply are not stopping getting injected...any other tricks you are aware of to stop the header injection?
This seems to be a trend right now for spammers.
Sep 14, 2005 08:29 pm There's so many different types of 'injection' and 'hijacking', typically the mail stuff is done by hijacking form processing.
I'd really have to see the code to guess what is happening, and possibly some raw dumps of server vars and get/post vars.
I haven't got hit by it in years, but it's also taken me nearly a decade to develop the form processing technology I use...
W.
Sep 14, 2005 09:21 pm securephp.damonkohler.com...ail_Injectionis the injection issues I am dealing with, but as of a few minutes ago, it appears I have them wrapped up...had some users trying like hell to hack a script and it's been solid...kinda weird...my form processing I use has been rock solid for a long time (3 years or so) then in the last week this flurry of crap came from about 10 sites...
spammers and hackers suck ***.
Sep 14, 2005 09:44 pm Yeah, I don't use the php mail function, wrote my own SMTP engine, and stringently parse all input passed to every field (following all RFC standards), so that sort of injection would never work.
I also use replay blocking, and use unique identifiers on critical apps which prevent form hijacking (i.e., people placing a form on their own server, pointing to anothers processing script, or passing raw data into the processing from anywhere els), not to mention a bunch of challenge/response apps and good ol' md5 digests.
Be happy to share the odd regex with you to help filter out that stuff but keep RFC compliant. Tough these days, as there's more than one RFC for nearly everything now...
W.
Sep 14, 2005 11:23 pm that would be awesome to see. regex has never been a real strong point with me, every time I need to write some I have to re-learn the code...
Sep 14, 2005 11:43 pm You ready for this one? To validate an e-mail address: (according to RFC 2822 and RFC 1035)
^[-a-z0-9!#$%&\'*+/=?^_`{|}~]+(\.[-a-z0-9!#$%&\'*+/=?^_`{|}~]+)*@(([a-z]([-a-z0-9]*[a-z0-9]+)?){1,63}\.)+([a-z]([-a-z0-9]*[a-z0-9]+)?){2,63}$
Next I check to see if the domain actually resolves to an IP (i.e., it actually exists).
I then sometimes check the host to see if has a valid MX record to ensure mail will get through. Not always reliable though, as not everyone sets up MX records properly.
W.
Sep 14, 2005 11:46 pm Hmm... just noted your posting on this page is double stripping slashes... What's displayed above isn't valid...
Got 'code' BB tags that force a section to be displayed as is?
W.
Sep 15, 2005 06:16 am hmm, yeah, well, I have the email addrress code, and, I get it in email alerts :-)
Sep 15, 2005 09:08 am Right! lol, the e-mails that aren't stripped slashed ;) Just remember to strip out the slashes only once on them!
W.
Sep 15, 2005 09:09 am btw, let me know if there is a specific sort of validation you are looking for, like cc nums, sin no's, etc... I got a wack...
W.
Sep 15, 2005 09:13 am damn, thats right, I never fixed the slashes thing on this forum, that was a different script...
What I am essentially trying to do is strip any additional header injection from the fields where the user inserts his name and email address, which I use to concatinate the "from" info for the email...so what I was doing was removing line breaks & carriage returns, ya know slash n and slash r...it seems to have worked on the most widely used script I have for processing, but some little rogue forms I have scattered around various clients don't seem to work...so I am trying to error out any that has bcc: or cc: in them...
if that makes sense...
Sep 15, 2005 09:19 am The validation above will fail if someone attempts that sort of injection, as an injected address would be invalid by current RFC standards. Use it on all e-mail addresses submitted and you are okay. (there is plenty of 'bad' regexes out there that don't work well, the one above seems to)
In the name (if you are using it in the email 'from' field), just don't allow any other characters other than alpha-numeric and spaces. You could use something like (winged:)
^[a-zA-Z0-9\ ]+$
W.
Sep 15, 2005 09:22 am True, thats should be an easy validation to write...even if the one you have above doesn't work...good idea...I just wonder about foreign names and such with weird characters (non-enlgish ones)...for different character sets and stuff...
I may email you today for the regex above if that's alright, just to make sure I get it right, the whole damn magic quotes can make problems even if I THINK it's right...
Sep 15, 2005 09:53 am Foregin/non-english sets can be accounted for by using translation, just convert the character into its closest representation in the english set.
As far as e-mail addresses, non-english characters are not allowed period, or they're using non-standard mail, meaning you couldn't e-mail them anyway. Most things that appear in the headers are governed by similar rules, as are those that most injection techniques exploit. So if you follow the RFC's and use a little common sense, you are generally 100% safe (until someone finds that variable you forgot to initialize properly! lol).
Tight code takes a ton of research, and that, has to be the worst part for most. For me, I do more dreaming than code (even though I write 2-3,000 lines a day), so the research comes naturally.
W.
Sep 15, 2005 09:59 am Yeah, a few years back when first learning it was all code with no security thoughts or planning. Since then, I code less, plan more and get more actually accomplished overall with less problems.
As a whole I have had very, very few security related issues on any of my sites, like said said, common sense generally prevails...every now and then something catches me off guard...I had never dealt with this form of injection before.
I find it interesting to see how people do this stuff...
Sep 15, 2005 11:06 am I've got the advantage of having started with code nearly 17 years ago...
It's actually better if you encounter it vs. research it I think. Kinda like putting your hand on the stove... You remember it alot better from that point on!!
W.
Sep 15, 2005 11:08 am agreed completely...the hard lessons are the ones best remembered...
Sep 15, 2005 07:06 pm Hmmm, well, I got ^[a-zA-Z-. ]+$ for the name thing, so it also allows hyphens and period (cuz of hyphenated names and things like Jr. and Sr.)
^[_\.0-9a-z-]+@([0-9a-z][0-9a-z-]+\.)+[a-z]{2,6}$ seems to work well for email address...it took every email format I threw at it and errored when I expected...
Both still error on line breaks, colons, and other junk...
Sep 15, 2005 07:07 pm hey, it even looks like the regex came out in the forum properly...
Sep 16, 2005 09:14 am You'll have to make sure to use eregi (case insensitive) with the one email ereg you used there or an address such as [email protected] would fail.
Also, if that . that follows the + in the 2nd have is not escaped (slashed) properly it'll allow ANY character in that position (including a carriage return) which can be exploited. The above also does not allow for IP addresses, and a small variaety of domains (allowed by RFC standards). It also allows an infinite amount of characters before and after the @ symbol, leaving you open for other sorts of attacks...
I'd be weary of trusting it!
W.
Sep 16, 2005 09:16 am interesting, well, it worked from what I tested, I'll try yours too, the IP thing is a good point.
Sep 16, 2005 09:42 am Don't forget the one I sent above requires a few escapes that got trimmed when I posted it, if ya e-mail me, I'll shoot you a reply with it in plain text to ensure nothing is left out.
W.
Sep 16, 2005 09:45 am will do, comin at ya...
I wish I knew this regex stuff better...times like this I learn it better, then I don't use it for weeks/months and I forget it all.
Sep 17, 2005 12:24 am It's nowhere near as easy as any scripting language, simply 'cause it's not following any standard spoken or written language.
Once you get used to what each symbol means, it becomes just like reading a few lines of code in one string... But it doesn't stick in the head anywhere near as easy.
W.
|
Related Forum Topics: |
If you would like to participate in the forum discussions, feel free to register for your free membership.