Reply to Thread

Post a reply to the thread: Web Deploy: Majority of installations fail (Source file not found)

Your Message

Click here to log in

What's the name of our main installation product (in lowercase letters)?

 

You may choose an icon for your message from this list

Additional Options

  • Will turn www.example.com into [URL]http://www.example.com[/URL].

Topic Review (Newest First)

  • 03-28-2019, 03:58 AM
    linder

    Re: Web Deploy: Majority of installations fail (Source file not found)

    Hello again,

    we'll set up a VM test environment here to simulate limited incoming or outgoing data transfers (similar to the scenario 1.3 MBytes/min "Session 96" and 3.4 MB/min "Session 111" in your 19-03-2019 LOG). So you'll see some downloads in the range of .00001 - .00010 in your next LOG files. We'll play a bit with timeouts. It's possible that we change "User-Agent: SB8" to "User-Agent: SB10" so you can filter our traffic.

    Update 1: for your information. At the moment we are IP 37.81.163.174 in your LOG to simulate low-bandwidth connections and timeouts. You'll see more tests during the next few days in your LOG.

    Update 2: I have attached a screenshot. This happens with Internet Explorer when a user tries to download the .00005 cluster file (about 10 MB) from a low-bandwidth connection (we have simulated "Session 96" with 0.17 Mbits/second). It simply times-out.

    Friedrich
  • 03-27-2019, 10:32 AM
    linder

    Re: Web Deploy: Majority of installations fail (Source file not found)

    Quote Originally Posted by instrumentally View Post
    Again, thank you for the time you've put into researching the issue. I'm impressed.
    You are very welcome

    Quote Originally Posted by instrumentally View Post
    Out of curiosity, what tool did you use to create the reports seen in your screen snapshots?
    LogViewPro from Kainet:
    https://www.softpedia.com/get/Intern...gViewPro.shtml

    Quote Originally Posted by instrumentally View Post
    Also, how are you determining the reputation of an IP address?
    I am using https://www.abuseipdb.com

    Quote Originally Posted by instrumentally View Post
    This change is to be made on the server, or within the Setupbuilder project?
    This is in the SetupBuilder project. Just use the #pragma item and set the HTTP_TIMEOUT value. Then recompile.

    By the way, there are two improvements in SetupBuilder 2019.2 Build 6187 (March 18, 2019). I think you are using a slightly older version. Perhaps this already makes a difference for low-bandwidth connections? Would be interesting to see the results.

    NEW : IDE: Add HTTP_RETRY #pragma to set the number of HTTP(s) download retries in case of a failed Web Install download action. The default internal retry value has been increased from 1 to 2.

    NEW : Installer: In case of a failed Web Install download action, the installer runtime re-initializes the WinSock component after waiting for 50 milliseconds.


    If possible, please keep me posted.

    Friedrich
  • 03-27-2019, 07:55 AM
    instrumentally

    Re: Web Deploy: Majority of installations fail (Source file not found)

    Again, thank you for the time you've put into researching the issue. I'm impressed.

    Out of curiosity, what tool did you use to create the reports seen in your screen snapshots?

    The IP belongs to Fortinet Technologies, Canada. It has been reported 15+ times for Hacking Brute Force, Web Bot Web activity, Web App Attach, etc.
    Also, how are you determining the reputation of an IP address?

    The default timeout for blocking receive calls in Web Installation HTTP actions is set to 10,000 milliseconds. You can increase this using the HTTP_TIMEOUT #pragma.
    This change is to be made on the server, or within the Setupbuilder project?
  • 03-27-2019, 07:31 AM
    linder

    Re: Web Deploy: Majority of installations fail (Source file not found)

    Quote Originally Posted by instrumentally View Post
    What we are seeing in the LOGs could be a combination of both failed user attempts and anti-virus scanners that are trying to analyze whether the EXE and associated files are trustworthy or not. As I mentioned towards the beginning of this thread, the EXE cannot be accessed through a web page linked to our home page. The EXE link was either in an email that we sent out, or in a HTML file on the server that has no links to any other page. So it cannot be that a standard search engine spider is behind the downloads. Would a search engine spider try to run a Web Deploy stub? That doesn't make sense. I can, however, see virus checkers performing sandbox tests. I can also see Google analyzing any links found in emails to EXEs that arrive in GMail inboxes.
    Well, I think this IP address 14.141.60.156 (which belongs to Tata Communications Limited, India) is prepared to handle it. This is from your server LOG. They download the Web Deploy stub and later execute it from another IP.

    2019-03-19 04:13:52 192.168.1.15 GET /demo/deployment/inword.exe - 80 - 14.141.60.156 python-requests/2.18.4 - 200 0 0 31
    2019-03-19 04:14:00 192.168.1.15 GET /demo/deployment/inword.exe - 80 - 14.141.60.156 python-requests/2.18.4 - 200 0 0 15
  • 03-27-2019, 07:28 AM
    linder

    Re: Web Deploy: Majority of installations fail (Source file not found)

    Quote Originally Posted by instrumentally View Post
    Although I do appreciate the labor you've put into solving this puzzle, I don't believe I would conclude that there were no human beings behind the downloads. I did indeed get complaints from a handful of users that they were unsuccessful in downloading the Web Deploy version.
    Okay, let's have a look on your server LOG in deep detail. On 19-03-2019 the SetupBuilder Web Installer downloaded cluster files in nine (9) different sessions.

    1. IP 207.102.138.40 (Session 3)

    From 04:19:58 - 04:21:03 this IP downloaded cluster files in the range of .00001 - .00053.

    The IP belongs to Fortinet Technologies, Canada. It has been reported 15+ times for Hacking Brute Force, Web Bot Web activity, Web App Attach, etc.

    2. IP 46.246.43.14 (Session 6)

    From 05:20:59 - 05:23:14 this IP downloaded cluster files in the range of .00001 - .00053.

    The IP reverses to "anon-43-14.vpn.ipredator.se". This should be enough information to guess what is behind this IP. Interesting to see that it downloaded again the same range of cluster files.

    3. IP 185.189.94.31 (Session 11)

    From 06:01:37 - 06:02:07 this IP downloaded cluster files in the range of .00001 - .00006.

    The IP belongs to AVAST Software s.r.o. - Czech Republic. This says it all.

    4. IP 67.137.36.66 (Session 35)

    From 10:49:29 - 10:50:38 this IP downloaded cluster files in the range of .00001 - .00053.

    The IP belongs to Integra Telecom Inc. It has been reported 14+ times for bad traffic, DDoS Attack Hacking, Email Spam Brute-Force, Fraud Orders, etc. And again the same .00001 - .00053 range of cluster files.

    5. IP 47.13.117.202 (Session 88)

    From 15:53:04 - 15:56:44 this IP executed eight (8) Web Installations to download the same .00001 cluster file eight (8) times.

    The IP belongs to spectrum.com. I don't think we need any more background information here.

    6. IP 47.13.117.202 (Session 94)

    From 16:23:35 - 16:23:50 this IP started again eight (8) Web Installations to download the same .00001 cluster file eight (8) times.

    The IP belongs to spectrum.com. See 5. above.

    7. IP 24.55.129.30 (Session 96)

    From 16:38:09 - 16:57:42 (in 19:33 minutes!) this IP downloaded the .00001 - .00005 cluster files.

    As far as I can see, this IP belongs to the "Penn View Bible Institute". This machine had a very slow Internet connection. In 19:33 minutes they downloaded ~25 MB - this translates to 1.3 MB/min. That Internet connection needs 3-4 hours to download all your cluster files. So it's very well possible the session timed-out on your cluster file .00005 (which is 9.88 MB in size). Note: the AVAST IP (see Session 11 above) downloaded cluster files in the range of .00001 - .00006 (30 MB) in 30 seconds.

    8. IP 104.242.15.134 (Session 111)

    From 18:27:37 - 18:35:03 (in 7:26 minutes) this IP downloaded .00001 - .00005 cluster files.

    In 7:36 minutes they downloaded ~25 MB - this translates to 3.4 MB/min. That Internet connection needs 2 hours to download all your cluster files. See 7. above.

    9. IP 47.13.117.202 (Session 160)

    From 23:56:44 - 23:57:41 this IP started again eight (8) Installations to download .00001 - .00005 cluster files.

    This IP belongs to spectrum.com. See 5. + 6. above.

    To sum it up: we have analyzed all nine (9) Web Deploy sessions from your 19-03-2019 LOG. Most likely, seven (7) sessions where automatically executed from within sandboxes. Two (2) sessions might have been timed out. Requests sent to the server—and responses from the server—can time out. This occurs when a low-bandwidth connection requires too much time to transmit the data. The default timeout for blocking receive calls in Web Installation HTTP actions is set to 10,000 milliseconds. You can increase this using the HTTP_TIMEOUT #pragma.

    We have executed seven (7) of your Web Installations and all seven sessions succeeded without any problem (from different Internet connections). And even from machines thousands of miles away from your server.

    Please don't get me wrong, and with all due respect, but I have to stop here. I have put quite a few resources into investigating this case. There is no "Source file not found" issue in Web Deploy. And it's simply not fair to say that your email campaign has turned out to be a disaster because (according to the LOG file) 8 out of 8 different users failed to complete the web deployment.
  • 03-26-2019, 07:45 PM
    instrumentally

    Re: Web Deploy: Majority of installations fail (Source file not found)

    There were no human beings behind the downloads.
    Although I do appreciate the labor you've put into solving this puzzle, I don't believe I would conclude that there were no human beings behind the downloads. I did indeed get complaints from a handful of users that they were unsuccessful in downloading the Web Deploy version.

    What we are seeing in the LOGs could be a combination of both failed user attempts and anti-virus scanners that are trying to analyze whether the EXE and associated files are trustworthy or not. As I mentioned towards the beginning of this thread, the EXE cannot be accessed through a web page linked to our home page. The EXE link was either in an email that we sent out, or in a HTML file on the server that has no links to any other page. So it cannot be that a standard search engine spider is behind the downloads. Would a search engine spider try to run a Web Deploy stub? That doesn't make sense. I can, however, see virus checkers performing sandbox tests. I can also see Google analyzing any links found in emails to EXEs that arrive in GMail inboxes.
  • 03-26-2019, 04:32 AM
    linder

    Re: Web Deploy: Majority of installations fail (Source file not found)

    Quote Originally Posted by instrumentally View Post
    You won't find any 404 return codes from the server with regards to the _0000#.bin files.
    Correct. After spending many hours on this and after analysing your LOG files, I came to the conclusion that there were NO 404 errors. All SetupBuilder Web Installation GET requests succeeded!

    Quote Originally Posted by instrumentally View Post
    So you are trying to tell me my problem is with my server returning error 404 codes and yet my server is not returning error 404 codes for the .bin files.
    Well, the good news for you is, that you do not have a server problem. And the good news for us is, there is no SetupBuilder Web Installer problem.

    Quote Originally Posted by instrumentally View Post
    On 2019-03-24, the last .bin file that the stub attempted to download was _00005.bin which returned code 200. The remaining 390+ other .bin files were never downloaded despite no 404 messages occurring.
    This download came from IP 185.220.70.152. It's listed several times in abuse and spam databases.

    Quote Originally Posted by instrumentally View Post
    The same thing happened for 2019-03-22. _00005.bin was the last .bin attempted, and the LOG file shows code 200 was returned, not 404.
    This download came from IP 178.62.228.82 and belongs to Digital Ocean in Amsterdam, The Netherlands. I bet it's some kind of spider-bot.

    Quote Originally Posted by instrumentally View Post
    For 2019-03-20, _00003.bin and _00054.bin were the last .bin files downloaded, again, with code 200.
    This download came from IP 66.102.6.183 and belongs to Google LLC.

    Quote Originally Posted by instrumentally View Post
    For 2019-03-19, _00001.bin, _00005.bin, _00053.bin and _00054.bin were the last .bin files downloaded, again, all stopping with code 200.
    This download came from IP 35.161.55.221 which belongs to Amazon.com Inc (amazonaws.com) and it's listed in abuse and spam databases.

    To cut a long story short, all mentioned "downloads" stopped with code 200 because "The Machines" simply terminated (killed) the sandbox executed Web Installations after a specific number of downloaded cluster files. There were no human beings behind the downloads. All Web Installs "stopped/terminated" intentionally.

    Friedrich
  • 03-26-2019, 12:33 AM
    linder

    Re: Web Deploy: Majority of installations fail (Source file not found)

    My guess is that ALL aborted/terminated Web Installations in your LOGs come from suspicious IP addresses and are automated downloads executed from within a sandbox.

    For example, the 185.220.70.152 IP in your server LOG which I already mentioned in one of my screenshots (I have attached it again). It has been reported 6 times (Abuse IPDB) and 7 times in Stop Spam.

    And please note the "/WP-login.php" attempts directly after the terminated Web Installs (on two different days). See attached screenshot. You said it's totally unrelated because it comes from another IP, but I really think it is related.

    IMO, it is time to close this "Source file not found" support thread, wouldn't you agree? SetupBuilder is not responsible for the "8 out of 8 different users failed to complete the web deployment" issue you mentioned in your original post. The automated (hacker/sniffer/whatever) tools started and terminated these Web Installations - the "failed" installation attempts you see in your server LOGs are not real user downloads.

    See: http://www.lindersoft.com/forums/sho...0181#post90181

    Friedrich
  • 03-26-2019, 12:13 AM
    linder

    Re: Web Deploy: Majority of installations fail (Source file not found)

    Perfect. Thanks for the confirmation. My "404" theory was based on your "Source file not found" statement.

    Friedrich
  • 03-26-2019, 12:11 AM
    linder

    Re: Web Deploy: Majority of installations fail (Source file not found)

    Quote Originally Posted by instrumentally View Post
    The wp-login.php request is coming from a different IP address that the stub EXE attempt to download the associated .bin files. That wp-login.php attempt isn't related to the SetupBuilder matter.
    Okay, but it is very often directly after a "terminated" Web Install. IMO, it is related.

    You started this support thread with "Web Deploy: Majority of installations fail (Source file not found)". So let me ask you, did you receive complains from (potential) customers? Did they send you screenshots from "Source file not found" error messages? Or did you start this support thread solely on the basis of the "aborted" Web installations LOG entries?

    You sent the LOG yesterday and before that I assumed your customers reported massive "Source file not found" errors. Based on the information you provided, I brought the "404 error" and "server does not respond" into play. But as far as I can see from your server LOGs, automated processes start your Web Installation in a sandbox (even from different IPs) and then terminate it. And you are absolutely right: there is not any SetupBuilder related error code (e.g. 404) in your LOGs because the SetupBuilder Web Installation never failed.

    Friedrich
This thread has more than 10 replies. Click here to review the whole thread.

Posting Permissions

  • You may post new threads
  • You may post replies
  • You may not post attachments
  • You may not edit your posts
  •