site just ... stops

![](" /> ~~Every once in a while, I'll load up my webpage and I'll notice the page will take a really, really long time t respond. No 404, or 500, or connection timeout or anything like that. It just does nothing. The image is what the my CPU graph looks like. It's like all my traffic just dropped off. The IO and bandwidth graphs look exactly the same.

I can ssh into my linode without any trouble or slowness. The apache log just shows no requests after the point in time where the graph goes to zero. The only way to "wake up" the server is to restart apache.

RAM useage during all of this is just about the same as it is any other time.

Does anyone have any ideas as to what could be causing this? Any ideas on what I can monitor both during normal operation and during these "black hole" periods so I can fix whatever is causing this? Whenever it does happen, I quickly restart apache rather than fiddle around with shuff because I obviously want the site to be working.

The site is running mod_wsgi with 2 django apps running. One is moderately busy (30,000 pageviews/month), while the other is pretty much idle (< 50 page views a month). I'm on a linode 360 with ubuntu 9.10 32bit.

edit: my logs show about 10 requests per minute up until [04/Feb/2010:18:25:26 -0500], then the next one is [04/Feb/2010:19:24:53 -0500], which is after I restarted apache.~~

8 Replies

Anything in your apache error log?

@mosdl:

Anything in your apache error log? nothing but your garden variety of deprecation warnings and various other stuff sprinkled about. No logs have anything between ~18:25 and ~19:24. Is there a log somewhere that lists when services get shut down? Maybe something just killed apache during that time?

I have exactly the same problem, but as nbv4 I cannot figure out what could be the problem.

How many Apache children do you have at the time? Are you hitting your maximum defined in httpd.conf?

Could you post the relevant parts of your config for mod_wsgi and the virtual hosts which use it?

I'm currently facing issues with mod_wsgi after migrating to Linode, I'm just wondering if there's some problem with WSGI daemon processes not being spawned properly.

How large are your Apache log files? Are they being rotated out properly?

@BarkerJr:

How many Apache children do you have at the time? Are you hitting your maximum defined in httpd.conf? how do I determine this?

> How large are your Apache log files? Are they being rotated out properly?

some of them are as high as 31MB, but they are somewhat regularily compressed. there are about 30 logs that are gzipped to the 100K - 1.5MB range

here is my wsgi file:

import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'proj.settings'
os.environ['MPLCONFIGDIR'] = '/var/cache/mpl'

import sys
sys.path = ['/srv/', '/srv/prof/'] + sys.path

import site
from settings_local import ENV_DIR
site.addsitedir(ENV_DIR + "/lib/python2.6/site-packages")

from django.core.handlers.wsgi import WSGIHandler
application = WSGIHandler()

here is my apache config, I have three that are more or less like this:

WSGIPythonHome /srv/BASELINE
WSGIRestrictStdout Off

 <virtualhost *:80="">ServerAdmin info@example.com
    ServerName example.com
    ServerAlias beta.example.com www.example.com

    RewriteEngine On
        RewriteCond %{HTTP_HOST} ^www\.(.+)$ [NC]
        RewriteRule ^(.*)$ http://%1/$1 [R=301,L]

    RewriteCond %{HTTP_HOST} ^beta\.(.+)$ [NC]
        RewriteRule ^(.*)$ http://%1$1 [R=301,L]

    WSGIDaemonProcess chris processes=2 maximum-requests=500 threads=5

    ErrorLog /var/log/apache2/beta_error.log

        # Possible values include: debug, info, notice, warn, error, crit,
        # alert, emerg.
        LogLevel warn

        CustomLog /var/log/apache2/beta_access.log combined

    WSGIProcessGroup chris
    WSGIScriptAlias / /srv/proj/wsgi.py

    Alias /fl-media /srv/proj/media
    Alias /ad-media /var/www/ad-media
    Alias /states /var/www/states

    Options    -Indexes FollowSymLinks</virtualhost> 

here is my mod_wsgi.conf:

 <ifmodule mod_wsgi.c="">#This config file is provided to give an overview of the directives,
    #which are only allowed in the 'server config' context.
    #For a detailed description of all avaiable directives please read
    #http://code.google.com/p/modwsgi/wiki/ConfigurationDirectives

    #WSGISocketPrefix: Configure directory to use for daemon sockets.
    #
    #Apache's DEFAULT_REL_RUNTIMEDIR should be the proper place for WSGI's
    #Socket. In case you want to mess with the permissions of the directory,
    #you need to define WSGISocketPrefix to an alternative directory.
    #See http://code.google.com/p/modwsgi/wiki/ConfigurationIssues for more
    #information

        #WSGISocketPrefix /var/run/apache2/wsgi

    #WSGIPythonOptimize: Enables basic Python optimisation features.
    #
    #Sets the level of Python compiler optimisations. The default is '0'
    #which means no optimisations are applied.
    #Setting the optimisation level to '1' or above will have the effect
    #of enabling basic Python optimisations and changes the filename
    #extension for compiled (bytecode) files from .pyc to .pyo.
    #When the optimisation level is set to '2', doc strings will not be
    #generated and retained. This will result in a smaller memory footprint,
    #but may cause some Python packages which interrogate doc strings in some
    #way to fail. 

        #WSGIPythonOptimize 0

    #WSGIPythonPath: Additional directories to search for Python modules,
    #                overriding the PYTHONPATH environment variable.
    #
    #Used to specify additional directories to search for Python modules.
    #If multiple directories are specified they should be separated by a ':'.

        #WSGIPythonPath directory|directory-1:directory-2:...

    #WSGIRestrictStdin: Enable restrictions on use of STDIN.
    #WSGIRestrictStdout: Enable restrictions on use of STDOUT.
    #WSGIRestrictSignal: Enable restrictions on use of signal().
    #
    #Well behaved WSGI applications neither should try to read/write from/to
    #STDIN/STDOUT, nor should they try to register signal handlers. If your
    #application needs an exception from this rule, you can disable the
    #restrictions here.

        #WSGIRestrictStdin On
        #WSGIRestrictStdout On
        #WSGIRestrictSignal On

    #WSGIAcceptMutex: Specify type of accept mutex used by daemon processes.
    #
    #The WSGIAcceptMutex directive sets the method that mod_wsgi will use to
    #serialize multiple daemon processes in a process group accepting requests
    #on a socket connection from the Apache child processes. If this directive
    #is not defined then the same type of mutex mechanism as used by Apache for
    #the main Apache child processes when accepting connections from a client
    #will be used. If set the method types are the same as for the Apache
    #AcceptMutex directive.

        #WSGIAcceptMutex default</ifmodule> 

and just the relvent parts of apache.conf:

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
 <ifmodule mpm_worker_module="">StartServers          1
    MinSpareThreads       3
    MaxSpareThreads       6 
    ThreadLimit          24
    ThreadsPerChild      25
    MaxClients           24
    MaxRequestsPerChild  3000</ifmodule> 

during one of it's locks, free -m returns this:

             total       used       free     shared    buffers     cached
Mem:           348        317         30          0         17        186
-/+ buffers/cache:        113        234
Swap:          255        156         99

right after I reset apache, it returns this:

             total       used       free     shared    buffers     cached
Mem:           348        309         38          0          7        149
-/+ buffers/cache:        153        195
Swap:          255         87        168

![](" /> This is what happens to my apache processes when one of these crashes occur. Whats odd is that it appears to have crashed at 5:30 in the morning, which is exactly the low point in my site's traffic rate.

Reply

Please enter an answer
Tips:

You can mention users to notify them: @username

You can use Markdown to format your question. For more examples see the Markdown Cheatsheet.

> I’m a blockquote.

I’m a blockquote.

[I'm a link] (https://www.google.com)

I'm a link

**I am bold** I am bold

*I am italicized* I am italicized

Community Code of Conduct