pandas.read_csv——分块读取大文件

  • read_csv中有个参数chunksize,通过指定一个chunksize分块大小来读取文件,返回的是一个可迭代的对象TextFileReader,IO Tools 举例如下:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
In [138]: reader = pd.read_table('tmp.sv', sep='|', chunksize=4)

In [139]: reader
Out[139]: <pandas.io.parsers.TextFileReader at 0x120d2f290>

In [140]: for chunk in reader:
.....: print(chunk)
.....:
Unnamed: 0 0 1 2 3
0 0 0.469112 -0.282863 -1.509059 -1.135632
1 1 1.212112 -0.173215 0.119209 -1.044236
2 2 -0.861849 -2.104569 -0.494929 1.071804
3 3 0.721555 -0.706771 -1.039575 0.271860
Unnamed: 0 0 1 2 3
0 4 -0.424972 0.567020 0.276232 -1.087401
1 5 -0.673690 0.113648 -1.478427 0.524988
2 6 0.404705 0.577046 -1.715002 -1.039268
3 7 -0.370647 -1.157892 -1.344312 0.844885
Unnamed: 0 0 1 2 3
0 8 1.075770 -0.10905 1.643563 -1.469388
1 9 0.357021 -0.67460 -1.776904 -0.968914
  • 指定iterator=True 也可以返回一个可迭代对象TextFileReader :
1
2
3
4
5
6
7
8
9
10
In [141]: reader = pd.read_table('tmp.sv', sep='|', iterator=True)

In [142]: reader.get_chunk(5)
Out[142]:
Unnamed: 0 0 1 2 3
0 0 0.469112 -0.282863 -1.509059 -1.135632
1 1 1.212112 -0.173215 0.119209 -1.044236
2 2 -0.861849 -2.104569 -0.494929 1.071804
3 3 0.721555 -0.706771 -1.039575 0.271860
4 4 -0.424972 0.567020 0.276232 -1.087401

  • 我需要打开的数据集是个csv文件,大小为3.7G,并且对于数据一无所知,所以首先打开前5行观察数据的类型,列标签等等:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
chunks = pd.read_csv('train.csv',iterator = True)
chunk = chunks.get_chunk(5)
print chunk
'''
date_time site_name posa_continent user_location_country \
0 2014-08-11 07:46:59 2 3 66
1 2014-08-11 08:22:12 2 3 66
2 2014-08-11 08:24:33 2 3 66
3 2014-08-09 18:05:16 2 3 66
4 2014-08-09 18:08:18 2 3 66

user_location_region user_location_city orig_destination_distance \
0 348 48862 2234.2641
1 348 48862 2234.2641
2 348 48862 2234.2641
3 442 35390 913.1932
4 442 35390 913.6259

user_id is_mobile is_package ... srch_children_cnt \
0 12 0 1 ... 0
1 12 0 1 ... 0
2 12 0 0 ... 0
3 93 0 0 ... 0
4 93 0 0 ... 0

srch_rm_cnt srch_destination_id srch_destination_type_id is_booking cnt \
0 1 8250 1 0 3
1 1 8250 1 1 1
2 1 8250 1 0 1
3 1 14984 1 0 1
4 1 14984 1 0 1

hotel_continent hotel_country hotel_market hotel_cluster
0 2 50 628 1
1 2 50 628 1
2 2 50 628 1
3 2 50 1457 80
4 2 50 1457 21

[5 rows x 24 columns]
'''
-------------End of the articleThank you for reading-------------
  • Author of this article:zfish
  • Link to this article: archives/f867afb8.html
  • Copyright Notice: All articles in this blog, except for special statements, please indicate the source!
0%