varnish web cache服务

 ??- varnish ????????- 充分利用epoll机制（能显著提高程序在大量并发连接中只有少量活跃的情况下的系统CPU利用率），并发量大，单连接资源较轻 ????- squid ????????- 在较大并发下，稳点性好，老当益壮

 ????- Manager Process ????????- 管理进程，相当于nginx的主控进程，不处理用户请求 ????- Cacher Process ????????- 线程Storage：完成缓存存储管理 ????????- 线程Log/Stats：日志记录----->存入共享内存Shared Memory Log中 ????????- 线程Worker threads：真正处理用户请求，通过线程池来定义，最大并发（线程池*线程池最大并发） ????- shared memory log ????????- varnishlog：读取日志文件，保存在磁盘中 ????????- varnishstat：读取统计数据，计数器 ????????- VCL配置接口：varnish配置语言 ????- varnishadm：让varnish加载新配置文件 ????- VCC Process：varnish的c编译器

 ?yum -y install varnish

 配置文件： ????- /etc/varnish/varnish.params（/etc/sysconfig/varnishd）：配置varnish服务进程的工作特性（监听地址和端口，缓存机制等） ????- /etc/varnish/default.vcl：配置各Child/Cache线程的工作特性 主程序： ????- /usr/sbin/varnishd CLI interface： ????- /usr/bin/varnishadmin：通过此管理工具，完成与Manager Process的交互，进而控制varnish的工作特性 Share Memory Log交互工具： ????- /usr/bin/varnishhist：日志历史 ????- /usr/binvarnishlog：记录详细log（请求报文首部，响应报文首部等） ????- /usr/bin/varnishcsa：格式化记录日志 ????- /usr/bin/varnishstat：日志统计 ????- /usr/bin/varnishtop：日志排序分析 测试工具程序： ????- /usr/bin/varnishtest VCL配置文件重载程序： ????- /usr/sbin/varnish_reload_vcl：此程序会编译配置文件 Systemd Unit File： ????- /usr/lib/systemd/system/varnish.service：varnish服务 ????- /usr/lib/systemd/system/varnishlog.service：原始记录日志（保存磁在盘上） ????- /usr/lib/systemd/system/varnishncsa.service：ncsa格式日志（保存磁在盘上）

 - malloc[,size]：内存存储，[,size]用于定义空间大小，重启后所有缓存项失效 - file[,path[,size[,granularity]]]：文件存储，黑盒，重启后所有缓存项失效 - persistent,path,size：文件存储，黑盒，重启后所有缓存项有效（试验阶段）

- 程序选项：/etc/varnish/varnish.params文件; ????- -a address[:port][,address[:port][...]：默认为6081端口; ????- -T address[:port]：默认为6082端口; ????- -s [name=]type[,options]：定义缓存存储机制； ????- -u user ????- -g group ????- -f config：VCL配置文件； ????- -F：运行于前台； - 运行时参数：/etc/varnish/varnish.params文件， DEAMON_OPTS ????- DAEMON_OPTS="-p thread_pool_min=5 -p thread_pool_max=500 -p thread_pool_timeout=300" ????- -p param=value：设定运行参数及其值，可重复使用多次 ????- -r param[,param...]：设定指定的参数为只读状态

# varnish_reload_vcl

# varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082 #登录管理程序

help [<command>] 获取帮助ping [<timestamp>] 测试服务器auth <response>quit 退出clibannerstatus 显示状态start 启动stop 停止vcl.load <configname> <filename> 加载VCL配置文件vcl.inline <configname> <quoted_VCLstring>vcl.use <configname> 激活VCL配置文件vcl.discard <configname> 删除VCL配置vcl.list 列出VCL配置param.show [-l] [<param>] 列出当前运行的参数param.set <param> <value> 运行参数临时调整panic.showpanic.clearstorage.list 列出数据存储信息vcl.show [-v] <configname> 列出VCL详细配置backend.list [<backend_expression>] 列出后端服务器backend.set_health <backend_expression> <state>ban <field> <operator> <arg> [&& <field> <oper> <arg>]...ban.list

RELOAD_VCL=1VARNISH_VCL_CONF=/etc/varnish/default.vcl ?#指定加载VCL配置文件VARNISH_LISTEN_ADDRESS=192.168.1.5 ?#服务监听的地址VARNISH_LISTEN_PORT=6081 ?#默认监听端口VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1 ?#管理服务监听的地址VARNISH_ADMIN_LISTEN_PORT=6082 #管理服务监听的端口VARNISH_SECRET_FILE=/etc/varnish/secret ?#连接秘钥VARNISH_STORAGE="malloc,256M" ?#用内存提供保存缓存,大小为256MVARNISH_USER=varnish ?#用户身份VARNISH_GROUP=varnish #组身份DAEMON_OPTS="-p thread_pool_min=5 -p thread_pool_max=500 -p thread_pool_timeout=300" #指定进程的运行参数

- 一个请求进入，varnish收下这个请求，判断是否对这个请求做处理 ???- 访问控制，直接拒绝 ???- 请求方法不识别，直接找后端服务器响应 ???- 正常请求，进入下一步- 查询此请求方法是否能缓存 ???- 不能缓存（如post，put），交给vcl_fetcg，由vcl_fetch投递给后端服务器响应 ???- 能缓存，进入下一步- vcl_hash基于hash查找缓存中是否有对应资源 ???- 如果命中，从本地缓存中直接响应给客户端 ???- 若未命中，通过vcl_fetch到后端服务器取回资源，然后先缓存，再响应给客户端

- vcl_recv收到请求，查找vcl_hash ???- 若命中（传递值hit）,交由vcl_hit ???????- hit命中，直接从缓存中响应，交由vcl_deliver投递给客户端 ???????- vcl_hash -(hit)-> vcl_hit --> vcl_deliver ???- 未命中（传递值miss），交由vcl_miss ???????- 交由vcl_backend_fetch请求后端服务器 ???????- vcl_hash -(miss)-> vcl_miss --> vcl_backend_fetch --> vcl_backend_response --> vcl_deliver ???- 若要删除缓存项（传递值purge）,交由vcl_purge ???????- 交由vcl_synh管理缓存，删除对应缓存 ???????- vcl_hash -(purge)-> vcl_purge --> vcl_synth ???- 若不能理解请求（传递值pipe）,交由vcl_pipe，请求被直接送至后端服务器 ???????- vcl_hash -(pipe)-> vcl_pipe ???- 并发连接超出（传递值busy），进入waiting状态，会等待重新请求查询缓存 ???- 传递值（pass，hit-for-pass），交由vcl_pass ???- vcl_hit和vcl_miss也能交由给pass- 两个特殊引擎： ???- vcl_init：在处理任何请求之前要执行的vcl代码：主要用于初始化vMODS ???- vcl_fini：所有的请求都已经结束，在vcl配置被丢弃时调用，主要用于清理vMODS

varnish> vcl.show -v boot ?#在客户端cli工具中查看sub vcl_recv { ???if (req.method == "PRI") { ?#如果客户端的请求方法是PRI，不支持SPDY或HTTP/2.0 ???????return (synth(405)); ?#则构建一个405的包响应给客户端 ???} ???if (req.method != "GET" && ?#如果客户端的请求方法不是GET ?????req.method != "HEAD" && ?#并且不是HEAD ?????req.method != "PUT" && ?#并且不是PUT ?????req.method != "POST" && ?#并且不是... ?????req.method != "TRACE" && ?????req.method != "OPTIONS" && ?????req.method != "DELETE") { ???????return (pipe); ?#即，不是标准HTTP请求方法的交给pipe（管道） ???} ???if (req.method != "GET" && req.method != "HEAD") { ?#请求方法不是GET和HEAD的 ???????return (pass); ?#交给pass处理，也就是除了GAT和HEAD方法其他的无法缓存 ???} ???if (req.http.Authorization || req.http.Cookie) { ?#http的请求首部包含Authorization（认证）或Cookie，即个人专有信息 ???????return (pass); ?#交给pass处理，因为这些带有个人信息的数据无法缓存 ???} ???return (hash); ?#以上的规则都没有做处理的请求交给hash做处理，剩下的是可以查询缓存的请求了}sub vcl_pipesub vcl_passsub vcl_hashsub vcl_purgesub vcl_hitsub vcl_misssub vcl_deliversub vcl_synthsub vcl_backend_fetchsub vcl_backend_responsesub vcl_backend_errorsub vcl_initsub vcl_fini

 - 文件开始要注明vcl版本号：vcl 4.0； - //,#,/*,*/为注释 - 子例行Subroutines使用sub关键字；例如sub_recv {...}; - 不支持循环，但支持条件语法，支持內建变量（受限于引擎） - 使用一个keyword基于return函数终止当前状态引擎，并决定交给哪一个状态引擎 - “域”专用配置，在一个状态引擎中的配置只对当前状态引擎有效

 - 每一个请求被单独处理 - 请求和请求间任何时间都是隔离的 - 各状态引擎有相关性，通过return连接 - 內建VCL code一直有效，并附加在自建的代码之后（vcl.show -v boot）

 - sub subroutine { ????????... ????} - if CONDITION { ????????... ????} else { ????????????... ????} - return(), hash_data()

regsub(str,regex,sub):字符串为str，根据正则regex模式匹配，把匹配到的内容替换为sub，只替换一次regsuball(str,regex,sub)：和regsub相同，替换所有ban(boolean expression)：符合表达式的都清理hash_data(input)：对input做hash计算synthetic(str)hash_data()：指明哈希计算的数据；减少差异，以提升命中率regsub(str,regex,sub)：把str中被regex第一次匹配到字符串替换为sub；主要用于URL Rewriteregsuball(str,regex,sub)：把str中被regex每一次匹配到字符串均替换为subreturn()ban(expression)ban_url(regex)：Bans所有的其URL可以被此处的regex匹配到的缓存对象synth(status,"STRING")：生成响应报文

call subroutine：调用子例行程序return(action):指明下一个动作newset：设定变量的值unset：取消变量的值

==,！=,~,>,>=,<,<= ????逻辑操作符：&&，||，！ ????变量赋值：=

# vim /etc/varnish/varnish.paramsVARNISH_LISTEN_PORT=6081 #监听端口默认是监听在本机的所有低智商的端口# vim /etc/varnish/default.vclbackend default { ???.host = "192.168.130.10"; #后端服务器的地址 ???.port = "80"; ?#后端服务器的端口号}sub vcl_deliver { # ???if (obj.hits>0) { ???????set resp.http.X-Cache = "HIT via" + " " + server.ip; ???} else { ???????set resp.http.X-Cache = "MISS from " + server.ip; ???}}# systemctl restart varnish ?#谨慎重启varnish服务，会导致之前的缓存失效#yum install httpd #在后端安装
#systemctl start http #启动服务
# echo X-Cache > /var/www/html/index.html #在后端服务器上添加页面# for i in {1..5};do curl -I -s 192.168.130.8:6081 | grep "X-Cache"; done #在客户端访问X-Cache: MISS from192.168.130.8X-Cache: HTT via 192.168.130.8X-Cache: HTT via 192.168.130.8X-Cache: HTT via 192.168.130.8X-Cache: HTT via 192.168.130.8

# vim /etc/varnish/default.vclsub vcl_recv { ???if (req.url ~ "(?i)^/(login|admin)") { ?#"?i"表示忽略大小写，匹配到url中带有login或admin的不查询缓存 ???????return(pass); ???}}# varnish_reload_vcl# for i in {1..5};do curl -I -s 192.168.130.8:6081/login | grep "X-Cache"; done #在客户端访问/login全部MISSX-Cache: MISS from192.168.130.8X-Cache: MISS from192.168.130.8X-Cache: MISS from192.168.130.8X-Cache: MISS from192.168.130.8X-Cache: MISS from192.168.130.8# for i in {1..5};do curl -I -s 192.168.130.8:6081/admin | grep "X-Cache"; done #在客户端访问/admin全部MISSX-Cache: MISS from192.168.130.8X-Cache: MISS from192.168.130.8X-Cache: MISS from192.168.130.8X-Cache: MISS from192.168.130.8X-Cache: MISS from192.168.130.8# for i in {1..5};do curl -I -s 192.168.130.8:6081/ | grep "X-Cache"; done #在客户端访问其他页面正常缓存查询X-Cache: MISS from192.168.130.8X-Cache: HTT via 192.168.130.8X-Cache: HTT via 192.168.130.8X-Cache: HTT via 192.168.130.8X-Cache: HTT via 192.168.130.8

sub vcl_backend_response { ???if (beresp.http.cache-control !~ "s-maxage") { ???????if (bereq.url ~ "(?i)\.(jpg|jpeg|png|gif|css|js)$") { ???????????unset beresp.http.Set-Cookie; ???????????set beresp.ttl = 3600s; ???????} ???}}

# vim /etc/varnish/default.vclsub vcl_recv { ???if (req.restarts == 0) { ?#匹配没有被重写的URL请求，即第一次请求 ???????if (req.http.X-Forwarded-For) { ?#变量存在并且有值则为真 ???????????set req.http.X-Forwarded-For = req.http.X-Forwarded-For + "," + client.ip; ?#将真正的client.ip添加到此变量中，用","隔开 ???????} else { ???????????set req.http.X-Forwarded-For = client.ip; ?#如果变量不存在或值为空，则直接将client.ip赋值与 ???????} ???}}# varnishadm -S /etc/varnish/secret -T 127.0.0.1:6082varnish> vcl.load conf1 /etc/varnish/default.vclvarnish> vcl.use conf1varnish> vcl.list ??available ??????0 bootavailable ??????0 reload_2018-07-14T09:55:58active ?????????0 conf1 ?#当前正在使用的配置[root@web ~]# vim /etc/httpd/conf/httpd.confLogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined[root@web ~]# systemctl restart httpd[root@client ~]# for i in {1..5}; do curl -I -s http://192.168.130.8:6081/admin |grep "X-Cache"; done #在客户端访问X-Cache: MISS from192.168.130.8X-Cache: HTT via 192.168.130.8X-Cache: HTT via 192.168.130.8X-Cache: HTT via 192.168.130.8X-Cache: HTT via 192.168.130.8[root@web ~]# tail /var/log/httpd/access_log 192.168.130.8 - - [15/Jul/2018:15:25:03 +0800] "HEAD /login HTTP/1.1" 404 - "-" "curl/7.29.0"192.168.130.8 - - [15/Jul/2018:15:25:03 +0800] "HEAD /login HTTP/1.1" 404 - "-" "curl/7.29.0"192.168.130.8 - - [15/Jul/2018:15:25:03 +0800] "HEAD /login HTTP/1.1" 404 - "-" "curl/7.29.0"192.168.130.8 - - [15/Jul/2018:15:25:13 +0800] "HEAD /admin HTTP/1.1" 404 - "-" "curl/7.29.0"192.168.130.8 - - [15/Jul/2018:15:25:13 +0800] "HEAD /admin HTTP/1.1" 404 - "-" "curl/7.29.0"192.168.130.8 - - [15/Jul/2018:15:25:13 +0800] "HEAD /admin HTTP/1.1" 404 - "-" "curl/7.29.0"192.168.130.8 - - [15/Jul/2018:15:25:13 +0800] "HEAD /admin HTTP/1.1" 404 - "-" "curl/7.29.0"192.168.130.8 - - [15/Jul/2018:15:25:13 +0800] "HEAD /admin HTTP/1.1" 404 - "-" "curl/7.29.0"192.168.130.8 - - [15/Jul/2018:15:27:09 +0800] "GET / HTTP/1.1" 200 8 "-" "curl/7.29.0"192.168.130.7,192.168.130.7 - - [15/Jul/2018:15:58:58 +0800] "GET /admin HTTP/1.1" 404 203 "-" "curl/7.29.0"#拿到了真正客户端IP，而不是之前的varnish服务器的IP

sub vcl_recv { ???if(req.http.User-Agent ~ "curl") { ???????return(synth(403)); ???}}

sub vcl_purge { ???return (synth(200,"Purged"));}

sub vcl_recv { ???if (req.method == "PURGE") { ???????return(purge); ???} ???...}

# vim /etc/varnish/default.vclacl purgers { ???"127.0.0.0"/8; ???"192.168.0.0"/24;}sub vcl_recv { ???if (req.method == "PURGE") { ???????if (!client.ip ~ purgers) { ???????????return(synth(405,"Purging not allowed for " + client.ip)); ???????} ???????return(purge); ???}}varnish> vcl.load conf3 /etc/varnish/default.vclvarnish> vcl.use conf3[root@client ~]# curl -I http://192.168.130.8：6081/HTTP/1.1 200 OKDate: Sun, 15 Jul 2018 08:53:15 GMTServer: Apache/2.4.6 (CentOS)Last-Modified: Sun, 15 Jul 2018 07:58:29 GMTETag: "8-57105141f79f4"Content-Length: 8Content-Type: text/html; charset=UTF-8X-Varnish: 32833 61Age: 12Via: 1.1 varnish-v4X-Cache: HTT via 192.168.130.8 #从缓存中获取Connection: keep-alive[root@client ~]# curl -I -X "PURGE" ?http://192.168.130.8：6081/ #清除缓存HTTP/1.1 405 Purging not allowed for 192.168.130.7 #客户端IP Date: Sun, 15 Jul 2018 08:53:22 GMTServer: VarnishX-Varnish: 32831Content-Type: text/html; charset=utf-8Retry-After: 5Content-Length: 333Connection: keep-alive[root@client ~]# curl -I http://192.168.130.8：6081/ HTTP/1.1 200 OKDate: Sun, 15 Jul 2018 08:56:23 GMTServer: Apache/2.4.6 (CentOS)Last-Modified: Sun, 15 Jul 2018 07:58:29 GMTETag: "8-57105141f79f4"Content-Length: 8Content-Type: text/html; charset=UTF-8X-Varnish: 63Age: 0Via: 1.1 varnish-v4X-Cache: MISS from192.168.130.8 #不从缓存中获取Connection: keep-alive

varnish> ban req.url ~ (?i)^/javascripts

sub vcl_recv { ???if (req.method == "BAN") { ???????ban("req.http.host == " + req.http.host + " && req.url == " + req.url); ?#将规则拼接起来传递给ban函数 ???????return(synth(200, "Ban added")); ???}}# curl -I -X "BAN" http://192.168.130.8：6081/javascripts/

backend default { ???.host = "172.20.81.10"; ???.port = "80";}backend appsrv { ???.host = "172.20.81.11"; ???.port = "80";}sub vcl_recv { ???if (req.url ~ "(?i)\.php$") { ???????set req.backend_hint = appsrv; ???} else { ???????set req.backend_hint = default; ???}}

import directors;backend srv1 { ???.host = "192.168.130.10"; ???.port = "80";}backend srv2 { ???.host = "192.168.130.11"; ???.port = "80";}sub vcl_init { ???new websrvs = directors.round_robin(); ?#round_robin()调度算法，不支持加权 ???websrvs.add_backend(srv1); ???websrvs.add_backend(srv2);}sub vcl_recv { ???set req.backend_hint = websrvs.backend();}

sub vcl_init { ???new h = directors.hash(); ???h.add_backend(one, 1); ???h.add_backend(two, 1);}sub vcl_recv { ???set req.backend_hint = h.backend(req.http.cookie);}

sub vcl_init { ???new websrvs = directors.random(); ???websrvs.add_backend(srv1, 1); ???websrvs.add_backend(srv2, 2);}

.probe：定义健康状态检测方法；.url：检测时要请求的URL，默认为”/";.request：发出的具体请求；.request ="GET /.healthtest.html HTTP/1.1""Host: www.dongfei.tech""Connection: close".window：基于最近的多少次检查来判断其健康状态；.threshold：最近.window中定义的这么次检查中至有.threshhold定义的次数是成功的；.interval：检测频度；.timeout：超时时长；.expected_response：期望的响应码，默认为200；

import directors;probe http_chk { ???????.url = "/index.html"; ???????.interval = 2s; ???????.timeout = 2s; ???????.window = 10; ?#最近10次检查 ???????.threshold = 7; ?#有7次成功则为健康主机}backend srv1 { ???????.host = "192.168.130.10"; ???????.port = "80"; ???????.probe = http_chk;}backend srv2 { ???????.host = "192.168.130.11"; ???????.port = "80"; ???????.probe = http_chk;}sub vcl_init { ???????new websrvs = directors.random(); ???????websrvs.add_backend(srv1, 1); ???????websrvs.add_backend(srv2, 2);}sub vcl_recv { ???????set req.backend_hint = websrvs.backend();}varnish> backend.list #查看后端主机健康状态信息 ????Backend name ??????????????????Refs ??Admin ?????Probesrv1(192.168.0.9,,80) ?????????3 ?????probe ?????Healthy 10/10srv2(192.168.0.10,,80) ????????3 ?????probe ?????Healthy 10/10varnish> backend.set_health srv1 sick|healthy|auto ?#手动标记主机状态 down|up|probe

backend BE_NAME { ???... ???.connect_timeout = 0.5s; ?#连接超时时间 ???.first_byte_timeout = 20s; ?#第一个字节20s不响应则为超时 ???.between_bytes_timeout = 5s; ?#第一个字节和第二个字节间隔超时时间 ???.max_connections = 50; ?#最大连接数}

thread_pools：工作线程数，最好小于或等于CPU核心数量thread_pool_max：每线程池的最大线程数thread_pool_min：最大空闲线程数thread_pool_timeout：空闲超过多长时间被清除thread_pool_add_delay：生成线程之前等待的时间thread_pool_destroy_delay：清除超出最大空闲线程数的线程之前等待的时间

# varnishstat -1 -f MAIN ?#指定查看MAIN段的信息# varnishstat -1 -f MAIN.cache_hit -f MAIN.cache_miss ?#显示指定参数的当前统计数据MAIN.cache_hit ?????????????47 ????????0.00 Cache hitsMAIN.cache_miss ????????????89 ????????0.01 Cache misses

# systemctl start varnishlog.service