特点：
- HTTP叫超文本传输协议，基于请求/响应模式的！
- HTTP是无状态协议。
为了方便认识http的请求和响应协议，用一段py来抓包分析下
import socketimport timedef handle_request(client): ???time.sleep(10) ???buf = client.recv(1024) ???print(buf.decode(‘utf-8‘)) ???client.send(bytes("HTTP/1.1 200 OK\r\n\r\n",encoding=‘utf-8‘)) ???client.send(bytes(‘Hello World‘, encoding=‘utf-8‘))def main(): ???sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) ???sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) ???sock.bind((‘192.168.1.102‘, 8000)) ???sock.listen(2) ???while True: ???????connection, address = sock.accept() ???????handle_request(connection) ???????connection.close()if __name__ == ‘__main__‘: ???main()请求协议
get请求协议的格式如下：
请求首行； ?// 请求方式 请求路径 协议和版本，例如：GET /index.html HTTP/1.1
请求头信息；// 请求头名称:请求头内容，即为key:value格式，例如：Host:localhost
空行； ????// 用来与请求体分隔开
请求体。 ??// GET没有请求体，只有POST有请求体。
get抓包分析
- GET / HTTP/1.1 ???????????????????
#请求主机ip:port- Host: 192.168.1.102:8000 ???
#客户端支持的链接方式，保持一段时间链接- Connection: keep-alive ??????
#表明客户端不愿意接受缓存请求，它需要的是最即时的资源。- Pragma: no-cache ??????????????
#没有缓存- Cache-Control: no-cache ??
#更加支持用https- Upgrade-Insecure-Requests: 1 ?
#与浏览器和OS相关的信息。有些网站会显示用户的系统版本和浏览器版本信息，这都是通过获取User-Agent头信息而来的；- User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36 ???????
- Accept: ?
#告诉服务器，当前客户端可以接收的文档类型，其实这里包含了/，就表示什么都可以接收；- text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8 ?
#告诉服务器，当前客户端可以接收的文档类型，其实这里包含了/，就表示什么都可以接收；- Accept-Encoding: gzip, deflate
#当前客户端支持的语言，可以在浏览器的工具?选项中找到语言相关信息；- Accept-Language: zh-CN,zh;q=0.9,en;q=0.8,cy;q=0.7
get请求特点：
- 没有请求体
- 数据必须在1K之内！
- GET请求数据会暴露在浏览器的地址栏中
post请求
#python 发送一个post请求import requeststest = {‘key1‘: ‘value1‘, ‘key2‘: ‘value2‘}ret = requests.post("http://192.168.1.102:8000", data=test)print(ret)print(ret.text)抓包分析
#post请求
- POST / HTTP/1.1 ????
- Host: 192.168.1.102:8000
- User-Agent: python-requests/2.18.4
- Accept-Encoding: gzip, deflate
- Accept: /
- Connection: keep-alive
#请求body长度
- Content-Length: 23
- Content-Type: application/x-www-form-urlencoded
#请求体 ?
- key1=value1&key2=value2
post请求特点
- 数据不会出现在地址栏中
- 数据的大小没有上限
- 有请求体
- 请求体中如果存在中文，会使用URL编码！
响应
- HTTP/1.1 200 OK
Hello World
响应协议的格式如下：
响应首行；
响应头信息；
空行；
响应体。
响应协议说明
HTTP/1.1 200 OK：响应协议为HTTP1.1，状态码为200，表示请求成功，OK是对状态码的解释；
Server:WSGIServer/0.2 CPython/3.5.2：服务器的版本信息；
Content-Type: text/html;charset=UTF-8：响应体使用的编码为UTF-8；
Content-Length: 724：响应体为724字节；
Set-Cookie: JSESSIONID=C97E2B4C55553EAB46079A4F263435A4; Path=/hello：响应给客户端的Cookie；
Date: Wed, 25 Sep 2012 04:15:03 GMT：响应的时间，这可能会有8小时的时区差；
tcp三次握手四次断开说明
服务端代码
import socketserver = socket.socket(family=socket.AF_INET, type=socket.SOCK_STREAM)back_log = 5buffer_size = 1024ip_port = (‘192.168.1.102‘, 8000)server.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)server.bind(ip_port)server.listen(back_log)print(‘waiting.....‘)doing = 1while doing > 0: ???conn, addr = server.accept() ???while doing > 0: ???????data = conn.recv(buffer_size).decode(‘utf-8‘) ???????if not data : ???????????break ???????print(‘recv:‘,data) ???????print(‘test:‘) ???????if data == ‘exit‘: ???????????conn.close() ???????????break ???????if data == ‘exitall‘: ???????????doing=0 ???????????break ???????else: ???????????if data: ???????????????conn.send(data.upper().encode(‘utf-8‘))server.close()客户端代码
import socketclient = socket.socket(family=socket.AF_INET, type=socket.SOCK_STREAM)buffer_size = 1024ip_port = (‘192.168.1.102‘, 8000)client.connect(ip_port)while True: ???msg = input(‘input>>>\n‘).strip() ???if not msg: ???????continue ???client.send(bytes(msg,encoding=‘utf-8‘)) ??????????data = client.recv(buffer_size).decode(encoding=‘utf-8‘) ???print(data) ???if msg == ‘exit‘: ???????breakclient.close()1、开启一个监听在8000端口的tpc进程
[root@ns1 conf.d]# ss -antp|grep 8000LISTEN ????0(当前等待的已经是ESTAB的连接数量，排除正在通信的) ?????5(表示接收连接最大数) ?????192.168.1.102:8000 ????????????????????*:* ??????????????????users:(("python3.6",pid=29528,fd=3))我们用watch来看服务器的连接情况
2、连接一个客户端，进程观察
其实这个阶段有很多状态变化，服务端会从SYN_RECV到ESTAB
Every 1.0s: ss -antp|grep 8000 ?????????????????????????????????????????????????????????????????????????????????????????Sat May 12 10:36:44 2018LISTEN ????0 ?????5 ?192.168.1.102:8000 ????????????????????*:* ??????????????????users:(("python3.6",pid=39882,fd=3))ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:58132 ??????????????users:(("python3.6",pid=39882,fd=4))2.1断开连接，进程观察
客户端断开，TIME-WAIT是等待服务器断开
Every 1.0s: ss -antp|grep 8000 ?????????????????????????????????????????????????????????????????????????????????????????Sat May 12 10:41:39 2018LISTEN ????0 ?????5 ?192.168.1.102:8000 ????????????????????*:* ??????????????????users:(("python3.6",pid=41041,fd=3))TIME-WAIT ?0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:59438几秒之后客户端就确认断开了 ?这里为什么会有延迟，是因为有可能服务器端发送到客户端的数据还没有发完，所以服务器要确认
Every 1.0s: ss -antp|grep 8000 ?????????????????????????????????????????????????????????????????????????????????????????Sat May 12 10:43:50 2018LISTEN ????0 ?????5 ?192.168.1.102:8000 ????????????????????*:* ??????????????????users:(("python3.6",pid=41041,fd=3))这里还有一种情况,出现close-wait，这种情况是因为客户端断开连接，而服务器还连接着到客户端的连接，有可能是服务器的bug（正常情况这个阶段是很快被服务器确认的）
Every 1.0s: ss -antp|grep 8000 ?????????????????????????????????????????????????????????????????????????????????????????Sat May 12 10:47:46 2018LISTEN ????0 ?????5 ?192.168.1.102:8000 ????????????????????*:* ??????????????????users:(("python3.6",pid=41041,fd=3))CLOSE-WAIT 0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:61056 ??????????????users:(("python3.6",pid=41041,fd=4))另外一组测试，测试多个连接
3、先连接3个客户端，进程观察
一共3个ESTAB连接 1给非阻塞，2个阻塞
Every 1.0s: ss -antp|grep 8000 ?????????????????????????????????????????????????????????????????????????????????????????Sat May 12 10:51:28 2018LISTEN ????2 ?????5 ?192.168.1.102:8000 ????????????????????*:* ??????????????????users:(("python3.6",pid=43041,fd=3))ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62088ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62079ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62070 ??????????????users:(("python3.6",pid=43041,fd=4))3.1 在连接3个客户端，一共6个，进程观察
6个建立ESTAB连接，1个非阻塞，5个阻塞
Every 1.0s: ss -antp|grep 8000 ?????????????????????????????????????????????????????????????????????????????????????????Sat May 12 10:52:02 2018LISTEN ????5 ?????5 ?192.168.1.102:8000 ????????????????????*:* ??????????????????users:(("python3.6",pid=43041,fd=3))ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62249ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62088ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62233ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62242ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62079ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62070 ??????????????users:(("python3.6",pid=43041,fd=4))3.2 继续添加3个客户端，一共9个连接 ，进程观察
一共9个连接，1个非阻塞ESTAB，6个阻塞ESTAB，2个等待服务器回应SYN-RECV
Every 1.0s: ss -antp|grep 8000 ?????????????????????????????????????????????????????????????????????????????????????????Sat May 12 10:52:24 2018LISTEN ????6 ?????5 ?192.168.1.102:8000 ????????????????????*:* ??????????????????users:(("python3.6",pid=43041,fd=3))SYN-RECV ??0 ?????0 ?192.168.1.102%if355331399:8000 ??????????????192.168.1.101:62369SYN-RECV ??0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62364ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62249ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62088ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62233ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62355ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62242ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62079ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62070 ??????????????users:(("python3.6",pid=43041,fd=4))几秒之后,服务器一直没有响应客户端的请求，服务器断开连接
Every 1.0s: ss -antp|grep 8000 ?????????????????????????????????????????????????????????????????????????????????????????Sat May 12 10:53:33 2018LISTEN ????6 ?????5 ?192.168.1.102:8000 ????????????????????*:* ??????????????????users:(("python3.6",pid=43041,fd=3))ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62249ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62088ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62233ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62355ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62242ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62079ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62070 ??????????????users:(("python3.6",pid=43041,fd=4))3.3 客户端断开第一个连接，就是非阻塞那个， 进行观察
1个等待服务器确认断开，1个非阻塞，5个阻塞。之前的62070非阻塞断开，第二个阻塞状态转为非阻塞
Every 1.0s: ss -antp|grep 8000 ?????????????????????????????????????????????????????????????????????????????????????????Sat May 12 10:54:25 2018LISTEN ????5 ?????5 ?192.168.1.102:8000 ????????????????????*:* ??????????????????users:(("python3.6",pid=43041,fd=3))ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62249ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62088ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62233ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62355ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62242ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62079 ??????????????users:(("python3.6",pid=43041,fd=4))TIME-WAIT ?0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:620703.4继续断开2个连接，观察
2个等待确认断开，1个非阻塞，3个阻塞
Every 1.0s: ss -antp|grep 8000 ?????????????????????????????????????????????????????????????????????????????????????????Sat May 12 10:56:25 2018LISTEN ????3 ?????5 ?192.168.1.102:8000 ????????????????????*:* ??????????????????users:(("python3.6",pid=43041,fd=3))ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62249TIME-WAIT ?0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62088ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62233 ??????????????users:(("python3.6",pid=43041,fd=4))ESTAB ?????4 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62355ESTAB ?????0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62242TIME-WAIT ?0 ?????0 ?192.168.1.102:8000 ??????????????192.168.1.101:62079- 由此分析三次握手四次断开：
1、在三次握手中，客户端先发起请求，syn seq请求信号
2、服务器接收到信号转为SYN-RECV
3、服务器查看半连接池，如果还有数量可以接收，就给客户端确认，并且发出服务器请求信号，状态还是SYN-RECV
4、客户端接收到服务器的同意，建立客户端到服务器的连接（DDOS攻击，这个阶段不给确认），客户端同意服务器的连接，然后确认信息
5、服务器接收到客户端的确认，建立服务器到客户端连接，状态由SYN-RECV到ESTAB
6、三次握手成功，期间双方可以友好交流，每次交流都是发出请求，确认收到。
7、客户端发送断开请求，状态转为close_wait
8、服务器确认同意，客户端到服务器的请求断开
9、服务器确认可以关闭请求，发送请求
10、客户端确认，断开服务器到客户端连接
常见IO模型介绍
介绍IO模型因为配置web服务高并发，提供一些基础理解。
还有是进程线程的一些概念，大概就是进程之前数据相互独立，切换进程比切换线程消耗大很多，线程是能共享进程资源，这里不做详细解释了
1、阻塞IO ?上述服务端socekt就是阻塞io
2、非阻塞IO
import socketimport timeserver = socket.socket(family=socket.AF_INET, type=socket.SOCK_STREAM)buffer_size = 1024ip_port = (‘127.0.0.1‘, 8000)server.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)server.bind(ip_port)server.listen(5)server.setblocking(False) ??#非阻塞print(‘waiting.....‘)doing = 1while doing > 0: ???try: ???????conn, addr = server.accept() ???????while doing > 0: ???????????try: ???????????????data = conn.recv(buffer_size).decode(‘utf-8‘) ????????????????if not data : ???????????????????break ???????????????print(‘recv:‘,data) ???????????????print(‘test:‘) ???????????????if data == ‘exit‘: ???????????????????conn.close() ????????????????if data == ‘exitall‘: ???????????????????doing = 0 ???????????????????break ???????????????else: ???????????????????if data: ???????????????????????conn.send(data.upper().encode(‘utf-8‘)) ???????????except Exception as e: ???????????????time.sleep(1) ???????????????print(e) ???except Exception as e: ???????time.sleep(1) ???????print(e)server.close()3、 IO 多路复用-select
用户进程调用select，内核负责所有select添加的sokect的状态，当任何一个socket中的数据准备好了，select就会返回，返回所有select添加的socket。这个时候用户进程再调用read操作，将数据从kernel拷贝到用户进程。IO多路复select在按找时间空间划分cpu，就已经实现来单线程高并发。缺点也很明显，水平出发消耗比较高，非活跃socket本因无须操作，select还依赖文件描述符，每台服务器能打开的文件描述符资源也是有限的。
服务端代码
import socketimport selectsk=socket.socket()sk.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)sk.bind(("127.0.0.1",8008))sk.listen(5)inputs=[sk,]while True: ???r, w, e = select.select(inputs, [], [], 5) ???for i in r: ???????if i is sk: ?# 看上述r描述 ???????????conn, add = i.accept() ?# 把r进来的socket接收到 ?不接收会一直存在内核 ???????????inputs.append(conn) ???????else: ???????????data_byte = i.recv(1024) ???????????print(str(data_byte, ‘utf-8‘)) ???????????inp = input(‘回答%s号客户>>>‘ % inputs.index(i)) ???????????i.sendall(bytes(inp, ‘utf-8‘)) ???print(‘>>>>>>‘)客户端代码
import socketsk=socket.socket()sk.connect(("127.0.0.1",8000))while 1: ???inp=input(">>").strip() ???sk.send(inp.encode("utf8")) ???data=sk.recv(1024) ???print(data.decode("utf8"))IO多路复用-poll-epoll
poll 本质跟select区别不大，只是取消来文件描述符的限制
epoll在linux内核2.6之后出现，同时支持水平触发跟边缘触发，边缘触发意思就是返回文件描述符发生变化的socket，大大减少来服务器的开销，内核复制数据到用户空间使用mmap内存映射技术，省掉了文件描述符在系统调用时复制的开销，另一个本质的改进在于epoll采用基于事件的就绪通知方式，一旦基于某个文件描述符就绪时，内核会采用类似callback的回调机制，迅速激活这个文件描述符。
事件通知方式实现方式
from concurrent.futures import ProcessPoolExecutorimport requestsdef task(url): ???response = requests.get(url) ???return responsedef done(future,*args,**kwargs): ???response = future.result() ???print(response.status_code,response.content)pool = ProcessPoolExecutor(7)url_list = [ ???‘https://www.taobao.com‘, ???‘https://www.sina.com‘, ???‘https://www.baidu.com‘,]for url in url_list: ???print(url) ???v = pool.submit(task,url) ???v.add_done_callback(done)print(‘all‘)pool.shutdown(wait=True)epoll的实现在python中特别简单，selectors模块的EpollSelector
服务端
import selectorsimport socketsel = selectors.EpollSelector()def accept(sock, mask): ???conn, addr = sock.accept() ?# Should be ready ???print(‘accepted‘, conn, ‘from‘, addr) ???conn.setblocking(False) ???sel.register(conn, selectors.EVENT_READ, read)def read(conn, mask): ???data = conn.recv(1000) ?# Should be ready ???if data: ???????print(‘echoing‘, repr(data), ‘to‘, conn) ???????conn.send(data) ?# Hope it won‘t block ???else: ???????print(‘closing‘, conn) ???????sel.unregister(conn) ???????conn.close()sock = socket.socket()sock.bind((‘localhost‘, 1234))sock.listen(100)sock.setblocking(False)sel.register(sock, selectors.EVENT_READ, accept)while True: ???events = sel.select() ???for key, mask in events: ???????callback = key.data ???????callback(key.fileobj, mask)客户端
import socketclient = socket.socket()client.connect((‘localhost‘, 9000))while True: ???cmd = input(‘>>> ‘).strip() ???if len(cmd) == 0 : continue ???client.send(cmd.encode(‘utf-8‘)) ???data = client.recv(1024) ???print(data.decode())client.close()Linux web服务前言
原文地址：http://blog.51cto.com/marvin89/2115474