Nginx v Redis: Upstream Prematurely Closed Connection
My AWS EC2 instance, running Node, needs to pass data to my Redis cache. This has been working fine for months. But, for the passed few days, it's broken.
All requests either timeout after approx 60 seconds, or I get a 502 error.
I added extra logging to the node function. This revealed that it all breaks down when the service trying to talk to Redis.
I have rebooted the redis node several times, but that did not work. I rebuilt the Cloudformation stack several times, but that did not work. i have terminated and created new EC2 instances - same effect.
I have checked the following:
Security Groups checked: all good
IAM permissions checked: all good
IAM roles checked: all good
Redis endpoint URL: correct
Access key/secret checked: all good
I looked at the nginx logs and found this telling bit of info:
2019/01/11 09:24:33 [error] 3826#0: *6732 upstream prematurely closed connection while reading response header from upstream, client: , server: my.server.url,
request: “GET /path/of/request HTTP/1.1”, upstream: “http://ip.ad.dr.ess:PORT/path/of/request”, host: “my.server.url”
I have two nginx configs. The parent looks like this:
# For more information on configuration, see:
# * Official English Documentation: http://nginx.org/en/docs/
# * Official Russian Documentation: http://nginx.org/ru/docs/
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log;
#error_log /var/log/nginx/error.log notice;
#error_log /var/log/nginx/error.log info;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" - $request_time X-Forwarded-For=$http_x_forwarded_for Host=$host';
log_format KVP 'ip="$remote_addr" time="$time_local" request="$request" '
'status_code=$status request_time=$request_time '
'host="$http_host" '
'body_bytes_sent=$body_bytes_sent referrer="$http_referer" user_agent="$http_user_agent" remote_user="$remote_user"';
access_log /var/log/nginx/access.log main;
error_log /var/log/nginx/error.log;
#sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 300s;
#gzip on;
# Load modular configuration files from the /etc/path/to/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/path/to/*.conf;
index index.html index.htm;
}
And the following one is included:
proxy_read_timeout 300s;
limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
# A virtual host using mix of IP-, name-, and port-based configuration
#
upstream NAME_REMOVED {
server xxx.x.x.x:;
}
server {
listen 80;
server_name ;
real_ip_header X-Forwarded-For;
# internal addresses in TEST
set_real_ip_from xx.xxx.0.0/16;
# internal addresses in PRODUCTION
set_real_ip_from xx.xxx.0.0/16;
gzip on;
gzip_comp_level 6;
gzip_vary on;
gzip_min_length 1000;
gzip_proxied any;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript;
gzip_buffers 16 8k;
client_max_body_size 100M;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_read_timeout 300s;
proxy_send_timeout 300s;
proxy_connect_timeout 300s;
proxy_pass http://app;
proxy_redirect off;
limit_req zone=api burst=10;
}
location ~ /.git/ {
deny all;
}
}
The timeouts shown above were added by me to try to fix this issue, but didn't have the desired effect.
amazon-web-services amazon-ec2 redis elasticsearch
add a comment |
My AWS EC2 instance, running Node, needs to pass data to my Redis cache. This has been working fine for months. But, for the passed few days, it's broken.
All requests either timeout after approx 60 seconds, or I get a 502 error.
I added extra logging to the node function. This revealed that it all breaks down when the service trying to talk to Redis.
I have rebooted the redis node several times, but that did not work. I rebuilt the Cloudformation stack several times, but that did not work. i have terminated and created new EC2 instances - same effect.
I have checked the following:
Security Groups checked: all good
IAM permissions checked: all good
IAM roles checked: all good
Redis endpoint URL: correct
Access key/secret checked: all good
I looked at the nginx logs and found this telling bit of info:
2019/01/11 09:24:33 [error] 3826#0: *6732 upstream prematurely closed connection while reading response header from upstream, client: , server: my.server.url,
request: “GET /path/of/request HTTP/1.1”, upstream: “http://ip.ad.dr.ess:PORT/path/of/request”, host: “my.server.url”
I have two nginx configs. The parent looks like this:
# For more information on configuration, see:
# * Official English Documentation: http://nginx.org/en/docs/
# * Official Russian Documentation: http://nginx.org/ru/docs/
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log;
#error_log /var/log/nginx/error.log notice;
#error_log /var/log/nginx/error.log info;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" - $request_time X-Forwarded-For=$http_x_forwarded_for Host=$host';
log_format KVP 'ip="$remote_addr" time="$time_local" request="$request" '
'status_code=$status request_time=$request_time '
'host="$http_host" '
'body_bytes_sent=$body_bytes_sent referrer="$http_referer" user_agent="$http_user_agent" remote_user="$remote_user"';
access_log /var/log/nginx/access.log main;
error_log /var/log/nginx/error.log;
#sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 300s;
#gzip on;
# Load modular configuration files from the /etc/path/to/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/path/to/*.conf;
index index.html index.htm;
}
And the following one is included:
proxy_read_timeout 300s;
limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
# A virtual host using mix of IP-, name-, and port-based configuration
#
upstream NAME_REMOVED {
server xxx.x.x.x:;
}
server {
listen 80;
server_name ;
real_ip_header X-Forwarded-For;
# internal addresses in TEST
set_real_ip_from xx.xxx.0.0/16;
# internal addresses in PRODUCTION
set_real_ip_from xx.xxx.0.0/16;
gzip on;
gzip_comp_level 6;
gzip_vary on;
gzip_min_length 1000;
gzip_proxied any;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript;
gzip_buffers 16 8k;
client_max_body_size 100M;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_read_timeout 300s;
proxy_send_timeout 300s;
proxy_connect_timeout 300s;
proxy_pass http://app;
proxy_redirect off;
limit_req zone=api burst=10;
}
location ~ /.git/ {
deny all;
}
}
The timeouts shown above were added by me to try to fix this issue, but didn't have the desired effect.
amazon-web-services amazon-ec2 redis elasticsearch
add a comment |
My AWS EC2 instance, running Node, needs to pass data to my Redis cache. This has been working fine for months. But, for the passed few days, it's broken.
All requests either timeout after approx 60 seconds, or I get a 502 error.
I added extra logging to the node function. This revealed that it all breaks down when the service trying to talk to Redis.
I have rebooted the redis node several times, but that did not work. I rebuilt the Cloudformation stack several times, but that did not work. i have terminated and created new EC2 instances - same effect.
I have checked the following:
Security Groups checked: all good
IAM permissions checked: all good
IAM roles checked: all good
Redis endpoint URL: correct
Access key/secret checked: all good
I looked at the nginx logs and found this telling bit of info:
2019/01/11 09:24:33 [error] 3826#0: *6732 upstream prematurely closed connection while reading response header from upstream, client: , server: my.server.url,
request: “GET /path/of/request HTTP/1.1”, upstream: “http://ip.ad.dr.ess:PORT/path/of/request”, host: “my.server.url”
I have two nginx configs. The parent looks like this:
# For more information on configuration, see:
# * Official English Documentation: http://nginx.org/en/docs/
# * Official Russian Documentation: http://nginx.org/ru/docs/
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log;
#error_log /var/log/nginx/error.log notice;
#error_log /var/log/nginx/error.log info;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" - $request_time X-Forwarded-For=$http_x_forwarded_for Host=$host';
log_format KVP 'ip="$remote_addr" time="$time_local" request="$request" '
'status_code=$status request_time=$request_time '
'host="$http_host" '
'body_bytes_sent=$body_bytes_sent referrer="$http_referer" user_agent="$http_user_agent" remote_user="$remote_user"';
access_log /var/log/nginx/access.log main;
error_log /var/log/nginx/error.log;
#sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 300s;
#gzip on;
# Load modular configuration files from the /etc/path/to/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/path/to/*.conf;
index index.html index.htm;
}
And the following one is included:
proxy_read_timeout 300s;
limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
# A virtual host using mix of IP-, name-, and port-based configuration
#
upstream NAME_REMOVED {
server xxx.x.x.x:;
}
server {
listen 80;
server_name ;
real_ip_header X-Forwarded-For;
# internal addresses in TEST
set_real_ip_from xx.xxx.0.0/16;
# internal addresses in PRODUCTION
set_real_ip_from xx.xxx.0.0/16;
gzip on;
gzip_comp_level 6;
gzip_vary on;
gzip_min_length 1000;
gzip_proxied any;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript;
gzip_buffers 16 8k;
client_max_body_size 100M;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_read_timeout 300s;
proxy_send_timeout 300s;
proxy_connect_timeout 300s;
proxy_pass http://app;
proxy_redirect off;
limit_req zone=api burst=10;
}
location ~ /.git/ {
deny all;
}
}
The timeouts shown above were added by me to try to fix this issue, but didn't have the desired effect.
amazon-web-services amazon-ec2 redis elasticsearch
My AWS EC2 instance, running Node, needs to pass data to my Redis cache. This has been working fine for months. But, for the passed few days, it's broken.
All requests either timeout after approx 60 seconds, or I get a 502 error.
I added extra logging to the node function. This revealed that it all breaks down when the service trying to talk to Redis.
I have rebooted the redis node several times, but that did not work. I rebuilt the Cloudformation stack several times, but that did not work. i have terminated and created new EC2 instances - same effect.
I have checked the following:
Security Groups checked: all good
IAM permissions checked: all good
IAM roles checked: all good
Redis endpoint URL: correct
Access key/secret checked: all good
I looked at the nginx logs and found this telling bit of info:
2019/01/11 09:24:33 [error] 3826#0: *6732 upstream prematurely closed connection while reading response header from upstream, client: , server: my.server.url,
request: “GET /path/of/request HTTP/1.1”, upstream: “http://ip.ad.dr.ess:PORT/path/of/request”, host: “my.server.url”
I have two nginx configs. The parent looks like this:
# For more information on configuration, see:
# * Official English Documentation: http://nginx.org/en/docs/
# * Official Russian Documentation: http://nginx.org/ru/docs/
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log;
#error_log /var/log/nginx/error.log notice;
#error_log /var/log/nginx/error.log info;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" - $request_time X-Forwarded-For=$http_x_forwarded_for Host=$host';
log_format KVP 'ip="$remote_addr" time="$time_local" request="$request" '
'status_code=$status request_time=$request_time '
'host="$http_host" '
'body_bytes_sent=$body_bytes_sent referrer="$http_referer" user_agent="$http_user_agent" remote_user="$remote_user"';
access_log /var/log/nginx/access.log main;
error_log /var/log/nginx/error.log;
#sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 300s;
#gzip on;
# Load modular configuration files from the /etc/path/to/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/path/to/*.conf;
index index.html index.htm;
}
And the following one is included:
proxy_read_timeout 300s;
limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
# A virtual host using mix of IP-, name-, and port-based configuration
#
upstream NAME_REMOVED {
server xxx.x.x.x:;
}
server {
listen 80;
server_name ;
real_ip_header X-Forwarded-For;
# internal addresses in TEST
set_real_ip_from xx.xxx.0.0/16;
# internal addresses in PRODUCTION
set_real_ip_from xx.xxx.0.0/16;
gzip on;
gzip_comp_level 6;
gzip_vary on;
gzip_min_length 1000;
gzip_proxied any;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript application/javascript;
gzip_buffers 16 8k;
client_max_body_size 100M;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_read_timeout 300s;
proxy_send_timeout 300s;
proxy_connect_timeout 300s;
proxy_pass http://app;
proxy_redirect off;
limit_req zone=api burst=10;
}
location ~ /.git/ {
deny all;
}
}
The timeouts shown above were added by me to try to fix this issue, but didn't have the desired effect.
amazon-web-services amazon-ec2 redis elasticsearch
amazon-web-services amazon-ec2 redis elasticsearch
asked Jan 11 at 17:40
JezJez
61
61
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "3"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1393258%2fnginx-v-redis-upstream-prematurely-closed-connection%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Super User!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsuperuser.com%2fquestions%2f1393258%2fnginx-v-redis-upstream-prematurely-closed-connection%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown