ecomdev / ecomdev_varnish Goto Github PK
View Code? Open in Web Editor NEWAdvanced Varnish implementation for Magento
License: Open Software License 3.0
Advanced Varnish implementation for Magento
License: Open Software License 3.0
Ivan,
I have some questions regarding the VCL files.
Thanks
I believe this might be the difference between users with a PHP session and users without a PHP session.
Ivan,
It seems my I don't get my catalogsearch page out of cache.
Is this defined somewhere or should it be cached?
Thanks
Ivan,
What do you usually do in magento to combine your module/varnish with SSL, do you have experience with using Pound for magento? If yes, what are your thoughts about it?
Thanks
Hi,
I would like to know, how can I implement on layout the dynamic block with ajax?
Can you recommend a specific "best practice" Varnish config?
How to reproduce:
Use browser a, go to a product detail page, put it in cart (make sure its in cache).
Use browser b, go to a product detail page, put the same product in cart, result: empty shopping cart.
Has no effect on configurable products.
Hi @IvanChepurnyi,
Just found a bug in the extension, when the session expires, my top cart does not get refreshed, this is because the quote checksum cookie remains the same.
I fixed this by checking if a frontend cookie exists from my own file in the validate function like this:
EcomDev.Varnish.AjaxBlock.addMethods({
validate: function () {
if (!this.config.cookie) {
return true;
}
var currentValue = this.getCookieValue();
if (currentValue === false && !this.storage.getValue('cookie')) {
return true;
}
if(Mage.Cookies.get("frontend") === null){
return false;
}
return this.storage.getValue('cookie') === currentValue;
}
If you approve of this, I could make a pullrequest.
Ivan,
To get the url for the ajax calls you use the getAjaxReloadUrl function, but this function only returns the path (via the parse_url function). As a result the call is blocked on https pages.
What is the reason you don't use the full url?
Could I just replace this the following or would this cause problems in other parts of the module?
$this->_ajaxReloadUrl = $this->_getUrl('varnish/ajax/reload',array("_secure" => Mage::app()->getStore()->isCurrentlySecure()));
Thanks in advance.
Hello thanks for posting this module. I see you are taking advantage of local storage in some cases and it looks like the way you structure the ESI blocks is different than Turpentine. I'm curious if you have any info to share on this. Thanks.
We're currently using your module in a Magento project with Varnish but noticed that the frontend cookie of Magento was not showing up.
We located our problem in EcomDev/Varnish/etc/varnish/default.vcl near line 173 where cookies set by the backend will get unset. We've deleted this line in our setup to fix the problem.
But I'm wondering what the purpose is of this block of code because it's probably there for a reason. Could you please explain?
Please help to connect your module to varnish. I am noob with varnish.
My environment is vagrant with ubuntu 14, nginx, varnish (version 3), magento 1.14.
And what shell I do next?
I tried to apply my default.vcl by varnishadm commands:
varnishadm -T 127.0.0.1:6082 -S /etc/varnish/secret vcl.load {config_name} default.vcl
varnishadm -T 127.0.0.1:6082 -S /etc/varnish/secret vcl.use {config_name}
But it is obviously not enough.
Please, say what I am missing. Many thanks!
When installing a fresh Magento (through Console, or Web-GUI), the session handler of Magento is called when it's not ready yet.
# Time Memory Function Location
1 0.0004 359776 {main}( ) .../index.php:0
2 0.0051 375088 Mage::run( ) .../index.php:83
3 0.0091 488240 Mage_Core_Model_App->run( ) .../Mage.php:691
4 0.8097 936040 Mage_Core_Controller_Varien_Front->dispatch( ) .../App.php:365
5 0.8122 976792 Mage_Core_Controller_Varien_Router_Standard->match( ) .../Front.php:172
6 0.8505 1098328 Mage_Core_Controller_Varien_Action->dispatch( ) .../Standard.php:254
7 0.8657 1346536 Mage_Core_Controller_Varien_Action->postDispatch( ) .../Action.php:422
8 0.8657 1347056 Mage::dispatchEvent( ) .../Action.php:551
9 0.8657 1347056 Mage_Core_Model_App->dispatchEvent( ) .../Mage.php:456
10 0.8657 1352344 Mage_Core_Model_App->_callObserverMethod( ) .../App.php:1337
11 0.8657 1352344 EcomDev_Varnish_Model_Observer->controllerActionPostdispatch( ) .../App.php:1358
12 0.8658 1352344 EcomDev_Varnish_Model_Observer->_addCookies( ) .../Observer.php:234
13 0.8663 1353632 EcomDev_Varnish_Model_Message->apply( ) .../Observer.php:156
14 0.8672 1378096 EcomDev_Varnish_Model_Message->getStorageByMessageType( ) .../Message.php:176
15 0.8672 1378096 Mage::getSingleton( ) .../Message.php:99
16 0.8672 1378152 Mage::getModel( ) .../Mage.php:485
17 0.8672 1378152 Mage_Core_Model_Config->getModelInstance( ) .../Mage.php:471
18 0.8677 1389272 Mage_Customer_Model_Session->__construct( ) .../Config.php:1354
Issue could be resolved by adding a Mage::isInstalled() to the method isActive
/**
* Checks if extension is active
*
* @return mixed
*/
public function isActive()
{
return Mage::isInstalled() && Mage::getStoreConfig(self::XML_PATH_ACTIVE) ;
}
After a few minutes I get a 503 with the current config copied from the extensions folder. My backend keeps working w/o the Varnish so that is not the problem.
Hi,
I like this module, because I am thinking the code is much better than in turpentine.
However I have some problem to get the ESI functionality working.
The main functionality is installed ( from the chef varnish) and the Magento helper with "getIsEsiAllowed" returns true.
The full page cache i working, but even in the first call of a category_list page etc. the esi.phtml
is not called. Is this not included by default?
Is it possible to update the README.md with an working example from a magento demo store with sample data? Would be a great help.
Greetings Jan
Just to make sure ... my vcl
import std;
import header;
import cookie;
import ipcast;
import querystring;
include "devicedetect.vcl";
# Balancers
#probe healthcheck {
# .url = "/";
# .interval = 30s;
# .timeout = 0.3s;
# .window = 8;
# .threshold = 3;
# .initial = 3;
# .expected_response = 302;
#}
backend node1 {
.host = "127.0.0.1";
.port = "8080";
# .probe = healthcheck;
# .first_byte_timeout = 300s;
# .connect_timeout = 5s;
# .between_bytes_timeout = 2s;
}
backend admin {
.host = "127.0.0.1";
.port = "8080";
.first_byte_timeout = 6000s;
.connect_timeout = 1000s;
.between_bytes_timeout = 2s;
}
director balancer client {
{
.backend = node1;
.weight = 1;
}
}
# Acls
acl allow_refresh {
# "127.0.0.1";
# "localhost";
# "192.168.10.1";
"192.168.10.2";
}
acl is_local {
# "127.0.0.1";
# "localhost";
# "192.168.10.1";
"192.168.10.2";
}
acl allow_admin {
"127.0.0.1";
"localhost";
"192.168.10.1";
}
# Admin detect
sub detect_admin {
unset req.http.is-admin;
if (req.url ~ "^(/index.php)?/admin") {
set req.http.is-admin = "1";
}
}
# Custom functions
sub normalize_url {
# Some generic URL manipulation, useful for all templates that follow
# First remove the Google Analytics added parameters, useless for our backend
set req.url = querystring.regfilter(req.url, "utm_source|utm_medium|utm_campaign|gclid|cx|ie|cof|siteurl");
# Strip a trailing ? if it exists
if (req.url ~ "\?$") {
set req.url = regsub(req.url, "\?$", "");
} else {
set req.url = querystring.sort(req.url);
}
}
sub normalize_cookie {
cookie.parse(req.http.cookie);
# Some generic cookie manipulation, useful for all templates that follow
# Remove the "has_js" cookie
cookie.delete("has_js");
# Remove any Google Analytics based cookies
cookie.delete("_ga");
cookie.delete("__utma");
cookie.delete("__utmb");
cookie.delete("__utmc");
cookie.delete("__utmz");
cookie.delete("__utmx");
# Remove the AddThis cookies
cookie.delete("__atuvc");
set req.http.cookie = cookie.get_string();
# Are there cookies left with only spaces or that are empty?
if (req.http.cookie ~ "^\s*$") {
remove req.http.cookie;
}
}
sub normalize_gzip_ua {
# Normalize Accept-Encoding header
# straight from the manual: https://www.varnish-cache.org/docs/3.0/tutorial/vary.html
if (req.http.Accept-Encoding) {
if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
# No point in compressing these
remove req.http.Accept-Encoding;
} elsif (req.http.Accept-Encoding ~ "gzip") {
set req.http.Accept-Encoding = "gzip";
} elsif (req.http.Accept-Encoding ~ "deflate") {
set req.http.Accept-Encoding = "deflate";
} else {
# unkown algorithm
remove req.http.Accept-Encoding;
}
}
# Send Surrogate-Capability headers to announce ESI support to backend
set req.http.Surrogate-Capability = "key=ESI/1.0";
if (req.http.User-Agent ~ "(?i)(ads|google|bing|msn|yandex|baidu|ro|career|)bot" ||
req.http.User-Agent ~ "(?i)(baidu|jike|symantec)spider" ||
req.http.User-Agent ~ "(?i)(facebook|scanner)" ||
req.http.User-Agent ~ "(?i)(web)crawler") {
set req.http.X-UA-Device = "bot";
}
# Send Surrogate-Capability headers to announce ESI support to backend
set req.http.Surrogate-Capability = "key=ESI/1.0";
set req.http.User-Agent = req.http.User-Agent + " " + req.http.X-UA-Device;
}
sub normalize_customer_segment {
unset req.http.X-Cache-Segment;
if( cookie.isset("segment_checksum")) {
set req.http.X-Cache-Segment = cookie.get("segment_checksum");
if (req.http.X-Cache-Segment) {
set client.identity = client.identity + req.http.X-Cache-Segment;
}
}
}
sub normalize_ip_address {
if (req.http.X-Forwarded-For ) {
set req.http.X-Forwarded-For = regsub(req.http.X-Forwarded-For, "^(^[^,]+),?.*$", "\1");
if (ipcast.ip(req.http.X-Forwarded-For, "127.0.0.1") == "127.0.0.1" ) {
error 400 "Bad request";
}
if (client.ip !~ is_local) {
unset req.http.X-Forwarded-For;
}
}
}
# Handle the HTTP request received by the client
sub vcl_recv {
# shortcut for DFind requests
if (req.url ~ "^/w00tw00t") {
error 404 "Not Found";
}
call detect_admin;
set client.identity = req.http.User-Agent + " " + client.ip;
call normalize_url;
call normalize_cookie;
call normalize_customer_segment;
call normalize_ip_address;
call devicedetect;
call normalize_gzip_ua;
# Deny access to admin, if not in list of allowed ip
if (req.http.is-admin && client.ip !~ allow_admin) {
error 403 "Forbidden";
} elsif (req.http.is-admin) {
if (req.http.X-Forwarded-For && ipcast.ip(req.http.X-Forwarded-For, "127.0.0.1") !~ allow_admin) {
error 403 "Forbidden";
}
set req.backend = admin;
unset req.http.is-admin;
return (pass);
} else {
set req.backend = balancer;
}
if (req.http.X-Forwarded-Proto && client.ip !~ is_local) {
unset req.http.X-Forwarded-Proto;
}
if (req.restarts == 0) {
if (client.ip !~ is_local) {
set req.http.X-Forwarded-For = client.ip;
}
}
# Normalize the header, remove the port (in case you're testing this on various TCP ports)
set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");
# Accept pages purge from authorized browsers' CTRl+F5
if (req.http.Cache-Control ~ "no-cache" && client.ip ~ allow_refresh) {
if (!req.http.X-Forwarded-For || ipcast.ip(req.http.X-Forwarded-For, "127.0.0.1") ~ allow_refresh) {
# Forces current page to have a cache miss
set req.hash_always_miss = true;
std.log("Cache Miss because of IP Address");
}
}
# Only deal with "normal" types
if (req.request != "GET" &&
req.request != "HEAD" &&
req.request != "PUT" &&
req.request != "POST" &&
req.request != "TRACE" &&
req.request != "OPTIONS" &&
req.request != "PATCH" &&
req.request != "DELETE") {
/* Non-RFC2616 or CONNECT which is weird. */
return (pipe);
}
# Large static files should be piped, so they are delivered directly to the end-user without
# waiting for Varnish to fully read the file first.
# TODO: once the Varnish Streaming branch merges with the master branch, use streaming here to avoid locking.
if (req.url ~ "^[^?]*\.(mp[34]|rar|tar|tgz|gz|wav|zip)(\?.*)?$") {
return (pipe);
}
# Remove all cookies for static files
# Static files are cached by default
if (req.url ~ "^[^?]*\.(bmp|bz2|css|doc|eot|flv|gif|gz|ico|jpeg|jpg|js|less|pdf|png|rtf|swf|txt|woff|xml|css\.map)(\?.*)?$") {
unset req.http.Cookie;
return (lookup);
}
if (req.http.Authorization) {
# Not cacheable by default
return (pass);
}
if (req.request == "POST") {
return (pass);
}
return (lookup);
}
# The data on which the hashing will take place
sub vcl_hash {
hash_data(req.url);
if (req.http.host) {
hash_data(req.http.host);
} else {
hash_data(server.ip);
}
# hash device for request
if (req.http.X-UA-Device) {
hash_data(req.http.X-UA-Device);
}
if (req.http.X-Forwarded-Proto) {
hash_data(req.http.X-Forwarded-Proto);
}
if (req.http.X-Cache-Segment) {
hash_data(req.http.X-Cache-Segment);
}
if (req.http.X-Geo-Country) {
hash_data(req.http.X-Geo-Country);
}
return (hash);
}
# Handle the HTTP request coming from our backend
sub vcl_fetch {
# Parse ESI request and remove Surrogate-Control header
if (beresp.http.Surrogate-Control ~ "ESI/1.0") {
set beresp.do_esi = true;
}
# Enable gzip compression, if header for it is specified
if (beresp.http.X-Cache-Gzip) {
remove beresp.http.X-Cache-Gzip;
set beresp.do_gzip = true;
}
# Sometimes, a 301 or 302 redirect formed via Apache's mod_rewrite can mess with the HTTP port that is being passed along.
# This often happens with simple rewrite rules in a scenario where Varnish runs on :80 and Apache on :8080 on the same box.
# A redirect can then often redirect the end-user to a URL on :8080, where it should be :80.
# This may need finetuning on your setup.
#
# To prevent accidental replace, we only filter the 301/302 redirects for now.
if (beresp.status == 301 || beresp.status == 302) {
set beresp.http.Location = regsub(beresp.http.Location, ":[0-9]+", "");
return (hit_for_pass);
}
if (beresp.status == 404) {
unset beresp.http.Set-Cookie;
return (hit_for_pass);
}
set beresp.http.X-UA-Device = req.http.X-UA-Device;
if (!beresp.http.X-Cache-Segment) {
set beresp.http.X-Cache-Segment = req.http.X-Cache-Segment;
}
# Remove all cookies for static files and cache them for 1hour
if (req.url ~ "^[^?]*\.(bmp|bz2|css|doc|eot|flv|gif|gz|ico|jpeg|jpg|js|less|pdf|png|rtf|swf|txt|woff|xml|css\.map)(\?.*)?$") {
unset req.http.Cookie;
set beresp.ttl = 1h;
return (deliver);
}
if (beresp.http.X-Cache-Ttl) {
set beresp.ttl = std.duration(beresp.http.X-Cache-Ttl, 0s);
unset beresp.http.Set-Cookie;
} else {
return (hit_for_pass);
}
return (deliver);
}
# The routine when we deliver the HTTP request to the user
# Last chance to modify headers that are sent to the client
sub vcl_deliver {
if (obj.hits > 0) {
set resp.http.X-Cache = "cached";
} else {
set resp.http.X-Cache = "uncached";
}
if (resp.http.X-Cache-Segment && (!cookie.isset("segment_checksum") || cookie.get("segment_checksum") != resp.http.X-Cache-Segment)
&& !header.get(resp.http.Set-Cookie, "segment_checksum=")) {
header.append(resp.http.Set-Cookie, "segment_checksum=" + resp.http.X-Cache-Segment + "; Domain=." + req.http.Host + "; Path=/");
}
if (resp.http.X-Cache-Store && (!cookie.isset("store") || cookie.get("store") != resp.http.X-Cache-Store)
&& !header.get(resp.http.Set-Cookie, "store=")) {
header.append(resp.http.Set-Cookie, "store=" + resp.http.X-Cache-Store + "; Domain=." + req.http.Host + "; Path=/; HttpOnly");
}
# Remove some headers: Apache version & OS
if (!resp.http.X-Debug) {
remove resp.http.X-Cache;
remove resp.http.X-UA-Device;
remove resp.http.X-Cache-Segment;
remove resp.http.X-Secure;
remove resp.http.X-Powered-By;
remove resp.http.Server;
remove resp.http.Via;
remove resp.http.Link;
remove resp.http.X-Header-Additional;
}
return (deliver);
}
Hi Ivan,
To get the correct block as a response in the ajax calls you need to define your block in the layout handle, otherwise you get an empty response. Is there a way to make this work with blocks that are not included in every page?
Is it correct that in order to make an element dynamic on a cached page you to create your own cookie in custom code and change the value of it everytime you change the content of you dynamic block?
Thanks in advance
Jens
When you delete something in cart, Magento uses http_referer.
When you add something to the cart before it, the last url hit was the holepunch url, this means that the http_referer is this ajax call url. This gives a blank page.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.