Server Upgrades

[Lycen]

Hey guys - a quick heads up that Steve is aware of this thread and has been fast at work seeking to resolve these issues. I promise that these snags won't be a permanent headache.

Oh sweet, good to know, thank you for keeping us updated Gary and best of luck to Steve

Thanks so much for posting about these problems, and for your patience. : ) Gary

[Bring4th_Steve]

LOL Lycen!

Sorry about the recent spat of server performance issues, everyone. I was trying a few changes to even further improve performance to the site and I accidentally wiped Bring4th off the face of the planet to where the domain name "Bring4th.org" was not recognized! (For the techies in the house, I'm embarrassed to divulge that I was testing out a distributed DNS system across multiple zones and it all backfired. Anything DNS-related can always take hours to change back, unfortunately.)

There's a lot that goes into running your own server; we're learning new things every day with our new VPS host! But when things are working, and when pages are caching as intended, Bring4th is proving to be incredibly fast!! Thankfully, we have the help of the web host provider, but I really took for granted all of the things that a "shared host" (think GoDaddy and HostGator) does for its customers versus a VPS host where they won't get involved unless you ask. It's literally like going from a "Novice" to "Expert" mode, where every single detail of a server is completely controllable and customizable. One sudden bad move, and a chain reaction of other bad effects unravel. The learning has been really good, though, and I've perfected our DNS settings so that we don't get the server errors anywhere close to the frequency we were before.

In fact, if I may ask... Has anyone experienced "any" server errors since July 26th?

Thanks,
Steve

[Lycen]

I am very happy to hear of your improvement in knowledge

In fact, if I may ask... Has anyone experienced "any" server errors since July 26th?

The site was just now sluggish took a few minutes to open any thread and threw up the usual errors. Yet after that things seem normal once more.

Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, webmaster@bring4th.org and inform them of the time the error occurred, and anything you might have done that may have caused the error.
More information about this error may be available in the server error log.
Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

or

Service Temporarily Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.

[Bring4th_Steve]

Hey Lycen,

I noticed that members who visit Bring4th from other countries tend to get the timeout screens.

In your case, I checked your IP and see that you are in a VERY remote location, and when I checked how many "hops" it takes from you to get to Texas, U.S., it's quite a few jumps before you arrive. I think that can be causing periodic delays.

I am currently looking at a couple of CDN services that keep content on servers in other countries, so that when Bring4th is accessed, the remote content can be served up first, saving a lot of time. The problem has to do with hotlinking, though. Many CDNs will put their data out on the "edge", but unless you intentionally authenticate each user, you have to keep the images and files open to the public, and that causes a lot of abuse and wasted money when people hotlink to your content.

So I'm exploring various options. And you might have seen the proposal that just went up in Olio for a site expansion. I will definitely be using some caching/edge services to get the new site going. We have a diverse audience now, and I don't like the idea of making anyone wait, when it's really not necessary.

Thanks for providing those details!
Steve

[Lycen]

Ah I cee, well I have no plans of moving right now so I am most appreciative that you are taking the time to give thought to this!
My pleasure, and thank you for the awesome service .)

[Jade]

In fact, if I may ask... Has anyone experienced "any" server errors since July 26th?

A few hours ago I was timing out, it was right at 10AM CDT which is the typical time that I've been having issues, so I'm guessing that's peak usage.

Not a big deal at all, it was for a short period, and I appreciate all your attention! Thank you Steve!

[Bring4th_Steve]

Thanks for letting me know, Guenivere! It's baffling to me how we signed up with a new service to get improved server performance, yet the thing that is slowing us down is not the server, but the network of virtual roads "leading" to our server. So frustrating! It is all a learning experience, however, and I've got some new ideas that I'm preparing to try out, which will hopefully put an end to the delay in messages.

Stay tuned! :-)
Steve

[Plenum]

Hi Steve,

the 'disppearing posts' issue seems to be rearing its head again. This is what I recevied just now as an email update:

"MarcRammer has just replied to a thread which you have subscribed to at Bring4th. This thread is titled what do you think about when you meditate?.

Here is an excerpt of the message:
------------------------------------------
I meditate at work while doing random tasks like vacuuming or sweeping. I just let go and rest in the silence. I also do random meditations sitting in a chair on my breaks. I'm still not completely silent in my mind, but I've been getting better.... (visit the thread to read more..)
------------------------------------------"

and yet, when I went to the thread, that post was not present.

so yes, is it a cacheing issue? or a glitch in the database?

others have also reported sporadic incidents of missing posts as well.

thanks man,

plenum

[reeay]

Plenum I had a similar problem (I can read marc's post right now). In my case I had set the option to view 50 posts per page. Selecting default option cleared up missing posts problem.

[Lycen]

Plenum I had a similar problem (I can read marc's post right now). In my case I had set the option to view 50 posts per page. Selecting default option cleared up missing posts problem.

I confirm this as well, just made use of it with success!

You can change this from "my dashboard" > "Edit your Profile" > "Edit Options" choose an other option from what you have at "Posts Per Page" if ya have this glitch. Did it and could see my post to tweak it. Seems like changing it from what one has set it to 'fixes' the problem. Meine was on default so I chose 50 posts instead.

Also by pressing "Preview Post" on a thread seems to show the missing posts. Without having to tweak the settings, good for none users! I thought one could preview posts when not logged in, it is not so *derp*

Edit: I was in err, corrected.

[reeay]

lol I am now having a problem viewing the latest post on the 'Confession' thread. When I click on link that would take me directly to the latest post, leads me to the top of the page (so I see the OP). Scrolling to the bottom of the page, the latest post is not there.

[zenmaster]

lol I am now having a problem viewing the latest post on the 'Confession' thread. When I click on link that would take me directly to the latest post, leads me to the top of the page (so I see the OP). Scrolling to the bottom of the page, the latest post is not there.

yep, and the second to latest is not accessible as well.

[Ankh]

Same here.

[xise]

same problem with the confession thread as above

[Horuseus]

More chance of a blind man solving the Rubiks cube than trying to send a PM these days .

Steve assuming you're using myBB which it does seem as though it does consider a database resync/rebuild via the Admin CP.

http://docs.mybb.com/Admin_CP_Database_Tools.html

[Bring4th_Austin]

I've lit the Steve signal, a silhouette of Steve's handsome face graces the night sky, he shall be here shortly to save the day.

[Bring4th_Steve]

Thanks for notifying us that the issue is happening. This remains as a bug with the forums until they release 2.0. The only way to get around the issue is to change the number of posts you see, which changes the way pagination occurs.

It is in the pagination where the latest post on a new page does not become visible until someone else replies to the thread. Only then does the previously invisible post resume normal visibility.

So for now, if you are notified of a message and do not see it, please change the amount of posts viewed at one time, as others in the forum have suggested. If there is a workaround to this issue prior to MyBB's 2.0 release, I am all too happy to apply the upgrade to fix the bug!

Sorry for the inconvenience... Never expect software to be infallible; it's always a moving target! :-)

Steve

[Bring4th_Steve]

I've lit the Steve signal, a silhouette of Steve's handsome face graces the night sky, he shall be here shortly to save the day.

Wow, I'd love to know who that Steve guy is! He sounds dreamy and so capable! -rolling my eyes- :-)

Thanks, Austin.. I wish I had the answer to the pagination bug, but the forum system has a remarkably large code base, and the limited team of developers who maintain the code reportedly have a laundry list of maintenance tasks, upgrade features, and other higher-priority bugs. They've acknowledged the issue in their forum, and claim that the 2.0 rewrite takes care of this. I don't know when that version is coming, but we might want to let them work the kinks out of it a little bit before we jump right into it.

I did refresh/recount/optimize all of the database tables, to see if there is a temporary reprieve. It would be helpful, from this post forward, if anyone can notify me that they received a new reply notification to one of their posts, and that they found the reply doesn't exist in the forums. That will help me get an idea of whether the database optimization worked. If it did work, and messages are now appearing again, I'll go ahead an increase the frequency in which the tables are optimized and rebuilt/recounted.

Thanks,
Steve

[Spaced]

I keep getting a little notification saying I have a new private message but there is no private message when I check. Or is there?

It keeps popping up and I keep clicking on it full of excitement that someone wants to reach out to me and my hopes are dashed time and time again

[Plenum]

Hiya Steve,

server seems to be getting pounded in the last 24 hours. Page loads can take up to 40 seconds, which is a crazy backload on the cpu.

dunno what the issue is; but something seems to be up.

thanks mate,

plen-meister

[Bring4th_Steve]

Well, today I discovered why the server had started freaking out... It turns out that deep within the bowels of our system, exist MASSIVE logs, recording every single URL that is visited, including each time chat is gets a live update from the server. Since this web host is fairly new, disk storage was not seen as a potential issue.

Over time, the data started piling up without anyone knowing the wiser, until the server finally shouted, "Ok! I've had enough! Going to sleep now!"

I've learned there's an application we can run called logrotate, which will automatically keep files from getting out of control. That should help us avoid running into this problem again.

I'm really sorry for the brief period of downtime!
Steve

[Marc]

I encountered a log error this morning with the site down.

[Plenum]

Well, today I discovered why the server had started freaking out... It turns out that deep within the bowels of our system, exist MASSIVE logs, recording every single URL that is visited, including each time chat is gets a live update from the server. Since this web host is fairly new, disk storage was not seen as a potential issue.

Over time, the data started piling up without anyone knowing the wiser, until the server finally shouted, "Ok! I've had enough! Going to sleep now!"

I've learned there's an application we can run called logrotate, which will automatically keep files from getting out of control. That should help us avoid running into this problem again.

I'm really sorry for the brief period of downtime!
Steve

Hi Steve!

things fixed yet? have you narrowed down the problem?

downtime in the last 48 hours has been like 70%

- -

perhaps we need to make you sign a 5 9's contract!

http://www.cyberciti.biz/faq/computer-ne...es-999999/

[Bring4th_Steve]

During the past few days, many of you have probably noticed that Bring4th has been down, or interrupting your sessions with one kind of error or another.

The reason for this is because the mail server is being attacked. It is attempting to send out over 5000 e-mails to a spam list, every second.

The complication in resolving this matter is that the server uses automated anti-spam tools to "fix" the issue. However, the tools are so aggressive that they will delete the clues to what is causing the error in the first place in order to get services running again.

When the mail server is cleared out, it turns out that the system logs take a beating, too. We have a limited account size, which is 200% larger than we need at the moment. Unfortunately, when services get out of control, the logs can become rather large, as well. I also learned that these large logs are also being backed up as part of the daily backup routine on our server, which means that any large log file we have essentially is doubled in size as a backup for it is made.

These issues are known, at least, and I'm currently working with the Host Provider to get control back of our disk space and log reporting. It is hoped that within the next few hours we can identify the exact script allowing for the mail exploits to take place.

Thanks for your patience!
Steve

[Bring4th_Steve]

Hey everyone,

For those of you who are technically-oriented and are interested in knowing why Bring4th has been suffering sporadic outages and performance issues, we FINALLY got the last of the wrinkles worked out!

It turns out that two scripts were being exploited. One from the chat room (which I had no idea even had mail services!), and one in the store where you "tell a friend" about a product through e-mail.

Once those two scripts were neutralized, our exim mail queue went from 6000+ e-mails waiting to serve as spam, to a little over 150, which is wayyyyy more acceptable for a site like ours. Here's the before and after from the command line:

root@host [/home/av/public_html]# exim -bpc
6372

6372 e-mails waiting to be sent out. Now we have:

root@host [/home/av/public_html]# exim -bpc
171

Much better. I'm going to continue monitoring this, and thankfully we have a solid web host who helped identify these threats.

The end result is that the site should "feel" like it is back to normal again. And once it is proving to be fine again, I will sign that 5 99999 contract, Plenum, just as long as there is a clause in there that if our performance falls below that, I maintain the right to nullify the contract. hehe

Again, thanks for the patience, everyone. Hopefully this exercise of hardening the site on the new web host will provide the smoother experience we were enjoying before the hackers turned their bots loose on us.

(Some of you may wonder how we even get on the radar for bots to be unleashed on us. The answer is that we are a big forum! And the bigger we are, the greater the target we become.)

Cheers,
Steve

[AnthroHeart]

Steve, I'm glad it's you and not me taking care of things. I am somewhat experienced in php, but nowhere near your level for maintaining such a large site.

[βαθμιαίος]

The home page isn't working right unless the visitor is logged in.

Before logging in:

---

Logged in:

---

After logging out:

[Bring4th_Steve]

Thanks for pointing this out, and for the handy screenshots! The issue has been resolved.

Steve

[Bring4th_Steve]

Steve, I'm glad it's you and not me taking care of things. I am somewhat experienced in php, but nowhere near your level for maintaining such a large site.

It can get hairy, err, furry, at times! But thank you, Gemini, for acknowledging the level of work that goes into maintaining the code. There's been enough issues over the years that this site makes a great way to stay in service to others! lol

Steve

[Horuseus]

Main site is infected with Malware. Seems restricted to the root site directory rather than the forum. My AV flagged up a trojan and a Sucuri scan also brings up the same:

-Web site: www.bring4th.org
-Status: Site infected with malware
-Web trust: Not Blacklisted

Malware found in the URL:
http://www.bring4th.org

Known javascript malware.
Details: http://labs.sucuri.net/db/malware/malwar...sanon7?v15

http://sitecheck.sucuri.net/results/www.bring4th.org

Edit: Reported this post as there's potential for B4 users without a half decent AV to fall foul of this. I had a similar issue with my own forum recently. Would advise reverting to an older backup and plugging/removing the exploited script as that is how this sort of thing usually occurs. Also suggest hardening your .htaccess to prevent such writes.