Error downloading videos - Another example

[WT]

Sky,

You asked for detailed info when an error occurred downloading videos. I got truncated every time I tried to download wp-bts-0010 just now. Look in the log on 29 Sep 2009 at time 00h32m40s BST + or - 5 seconds, and it failed within a second of starting.

Just as an experiment I then tried downloading by RIGHT clicking - and it worked! Have you switched back to RIGHT click instead of LEFT click without saying anything on the pages?

[skymouse]

Unfortunately my earliest entries at this moment start at about 5 a.m. on the morning of 29 Sep, so this time around we've missed locating any entry by just a few hours. On other occasions, I may hvae been able to catch this (I just happened to be out all day, and it may be that the entry was there earlier on if I'd been around to see it. Sorry about that.)

I also had another look through the logs over the period that are available right now, and of the several hundred thousand visits during this time no abnormal entries have been logged (only the usual things like 404's, etc.) In particular, I was looking for Apache restarts and the like. However, that in itself is not conclusive, because if an individual apache child process dies unexpectedly, I suspect that is not necessarily recorded. I will have a chat with my web host about it, and see ask them to find out what is happening. As yes, other members have not reported a similar problem, but that doesn't mean it isn't happening to anyone else, as I've learnt from past experience that problems (even ones that stop customers using the service they paid for) often go unreported (strange but true).

Left- and right-clicks *shouldn't* affect the problem you are getting, as the link now just goes to the video file. There's no jiggery pokery mediating the download. However, the MIME type is set to Application/octet-stream in the HTTP header, and disposition: attachment. Some old versions of IE are expected to have trouble with that due to some bugs that made IE do strange things (the precise analysis of which even now remains a matter of slight, though useless, controversy), but the symptoms were different from what you described, and I think in your case the session is being interupted; I don't think it's a browser bug. However, there is an outside chance that the problem is triggered in some browsers specifically in a left- or right-click situation, so just in case that is so, please report any more findings you may have in regard to that.

SM

[WT]

Sky,

I will log if/when the problem occurs again. I am using IE 8 fully patched with no download manager, so an old browser isn't part of the equation. Next time this occurs, I will then try downloading in Firefox to see if that also has the problem.

I think your diagnosis that the session is being interrupted is exactly right - that matches the symptom of starting normally and ending abnormally but with no error message within a second perfectly.

Other sources of the problem could, of course, be my computer, broadband connection or ISP. However, no downloads from any other source have had this problem, whereas it has happened before on your (ISP's) servers with exactly these symptoms more than once so, on the large balance of probability, the problem is likely to be at your (ISP's) end.

[WT]

Well, that didn't take long to investigate. Happened again just now with wp-0542.mpg and... just for the fun of it, I was using Firefox. So that eliminates a browser problem. And I right clicked... so it doesn't seem related to that.

What it does seem related to is whether there may have been any server activity recently. I have thrashed it half to death left and right click but now with OVERLAPPING downloads, and no problem. First download - it dies, after what could be a cache worth of bytes - about 300 odd kb. So start looking for some power saving on the part of the server which then drops the first connection that wakes it up from its snooze (and was turned on 2 or 3 weeks ago)...

Also, as that matches roughly when MS patches are applied in the month, if it runs MS software, check the patching logs, and look up each patch applied to see if any affect IP connections / web serving / power saving / etc.

And look in the log at Fri 2 Oct 2009 00h25m38s +/- 5s BST. Although, from what you say, I doubt that you will find anything as this isn't a server error or restart (if it were, it wouldn't work 5s later, rebooting would take at least 30s), just a dropped connection, and almost certainly isn't being logged.

PS It wasn't my ISP or the BT Exchange power saving on my ADSL connection as I has successfully downloaded something without any error from another web site a minute or two before.

PPS I'll move this to a technical Board when picked up.

[driversseat]

WT;
There is another possibility here. Perhaps unlikely, but your firewall may be interrupting things when you first start the download.

You may, in fact, have two firewalls on your system. One, (hardware) in the router, and a software firewall on your computer. Sometimes, there will be a momentary conflict between the two which can result in the situation you describe.

Be sure to check the logs in your firewalls to see if they have been the cause. If so, then it should be fairly easy to get them working together correctly.

[WT]

That's a very good point driversseat. I do indeed have two firewalls exactly as you describe - but they don't conflict. I know this because:
All other downloads work fine.
Sky's downloads worked fine with my set up until about 3 weeks ago.
My firewall set up has not changed in any way.


I looked at the hardware log anyway, but it doesn't cover connections, just startup, admin, DoS, scans and attempts to access blocked sites (there aren't any). The software log has no blocked packets near the relevant time.

Thanks for the suggesiton.

[skymouse]

Thanks for the further discussion. WT, I will mention your observations to the hosting company. Maybe it is related to having multiple TCP sessions to a single client, and some of them not closing properly (this would maybe have a similar effect as if your own PC were experiencing a kind of denial of service attack, depending on how your TCP stack handles things). In this case, they might (I would speculate) try to identify settings that are currently permitting you to open more connections than can actually be permited (by a different setting) to remain open.

No component of my web server (or its surrounding network) uses MS products (except maybe the odd one or two microsoft mouses in the accounts dept, probably!), so the recent update won't have affected things at thes erver end. However, an update may have subtly changed things for some visitors to the web sites in such a way as to expose or create a problem in some situations that maybe didn't happen before.

SM

[skymouse]

WT, I have pasted your entire message* verbatim to the web hosting technical staff handling this issue.

SM

* minus your email address and Wet Trousers nickname :)

[skymouse]

I've received the following two messages (from two different engineers who have both been testing the server) (pasted below). I am wondering if perhaps what is happening is that when you have a certain number of connections open simultaneously (which can happen as they may persist for a short time even after data has stopped being transferred), your anti-virus software is thinking you are suffering an attack from the web server (because it sees the multiple connections), and hence blocks one or more of the connections.

SM

Hello,

I would ask them to check for changes made to the browser after any
updates, as well as any programs that may be running (virus scanners,
etc.) which could cause an issue. Let us know if anything else is needed.

--
Regards,

----------------------------------------
Addyon Varga
Webair Internet Development, Inc.



Hello,

In some cases it can be due to a clients anti virus software; alot of
software proxies the connection between the computer and the internet, so
if there is an error with the virus scanner checking the file as it is
being downloaded, it can lead to truncated file downloads.

If you continue to see the issue please let us know, however we have not
been able to find anything on the server end to explain this.

Sincerely,

----------------------------------------
Chris Buckley
Webair Internet Development, Inc.

[WT]

Thanks, Sky.

I'll try stopping the automatic memory-resident scanning if I remember when I download the next videos - I can scan them manually by hand once downloaded. It will be interesting to see if that does avoid the issue.

If it looks promising, I will try combinations of temporarily disabling McAfee VirusScan and ZoneAlarm Suite anti-spyware to see if I can finger one of them as the culprit (BTW I don't have ZoneAlarm anti-virus installed as that would fight McAfee). Of course, they are updated nearly every day, so that could account for the change when the problem started.

I'll let you know...

[WT]

There is another '30 Sep' video there today, so I was able to test it straight away. Guess what?

The download failed in exactly the same way! :shock: Even with all memory resident scanning turned off. So I guess it isn't that. Actually thinking about it, it couldn't have been anyway - if that were the cause, it is unlikely to the point of utter incredulity that around 10 of your downloads would have been affected and none of about 30 or 40 others. The odds of such a skewed combination occurring randomly are such that it would take most of the universe lifetime before such a combination would be observed.

The 2 or your downloads that have succeeded first time without error are much more likely to correspond to when others were already downloading and the servers were active, as I supposed, compared to around 10 that failed first time.

So, I am afraid, it is very specific to your web servers or their configuration. Not a problem though, I just let the download abort, try it again and it is fine. Then subsequent downloads overlapped with it are fine.

The only other thing I can think of is that there is probably another factor, such as that my ISP has some timeout that is exceeded when your servers have to be woken up but not when they are already awake, as others don't seem to be suffering this issue.

[skymouse]

It's looking less and less like an application level issue, and more like a lower level one.

The timeout idea is an interesting one. However, I'm not altogether sure whether there'd be a big latency between a request for a recent served file and any other file. There may be a small difference due to memory cacheing at the server, but latency varies on the web for some many possible reasons that I wouldn't think your ISP's timeout would be low enough to catch the difference in this case. Unless of course the connection is in fact taking VERY much longer. I will mention your latest experience to the hosting company, in case we're seeing excessive latency since introducing these bigger files, due to insufficient RAM or something.

First, please confirm that I've udnerstood your recent observations - when a file has (or is surmised to have been) very recently requested by you or a (likely) other user, the problem occurs with less probability?

SM

[skymouse]

PS, if I've understood that last bit correctly, we could quite easily test that out if you have some spare time at some point to talk on the phone. You could try downloading an assortment of files either with or without me first downloading it myself a second or two earlier. If we see a difference, we've nailed it, and it's high latency caused by a performance issue on the server.

[WT]

In reply to your last but one message - you have understood perfectly. The 'lower' probability of error seems to be close to zero. The probability of error downloading a file for the first time is about 80%, irrespective of time of day, browser and whether my security software is running.

However, it does not seem related to WHICH file I download. If I first download one, all the rest are then OK! So it probably isn't due to individual file caching. Like I say, it's like something waking up, or a channel being opened or somesuch.

Happy to phone and see if we can pinpoint, and glad from your PM that the ISP is now taking it more seriously. I will probably next be downloading at the weekend, so will text you (if I remember) before I start. Whether or not you are available, I will log times to +- 5 seconds for log checking (assuming Americans can convert BST to EST ;) ). It would be very interesting to see if there is still an issue if you are downloading a file at the time and the server is active...

PS There isn't any latency noticeable to a human being (and I have a connection running at 6-7 Mbps typically), so any timeout being exceeded is at most a few hundreds of milliseconds which, as you say, makes that unlikely in a TCP/IP environment (or everything would fail quite often!). But I reckon there must be an extra factor somewhere.

It's also funny that I get a few hundred kb of the file - valid, so it plays for 2 seconds - before it aborts, so something must start serving the right thing before giving up. All in all, quite a mystery... :???:

[WT]

After investigating this for two months, the hosting company has got nowhere and is now asking for the data again! It is unlikely that this will be solved.

If anyone else experiences movie downloads failing just after they start, the simple work-around is to start the download again, which will then work.

Factors that appear relevant to this happening are:
It only happens to movie downloads, not zip files of photos nor separate test files (the page header informaiton is different).
Not having done a download for a day - most failures occur on the first download in 24 hours.
Having protection turned on in Internet Explorer and Firefox (and maybe other browsers) that checks the website back with a central server.
Possibly the function of the local router, perhaps whether it inspects the contents of traffic.