a3nm's blog

Wikimedia projects by dump size, and average compressed article size

— updated

I was looking at the list of Wikipedias, and I thought that after all, the number of articles is a very bad measure of the size of a Wikipedia, because articles can have various sizes. More specifically, some Wikipedias have bots which create stubs about all sorts of villages and other geographic locations (to take the random example of the Volapük Wikipedia), and this makes them seem bigger than they really are.

It seems fairer to compare the dump size of the wiki as a measure of the total quantity of text, or rather the compressed dump size to eliminate redundancy. Surprisingly, I didn't manage to find this information in a usable chart, so here we go... Note that I'm speaking about the dumps from the Wikimedia dump service, and more specifically the pages-articles.xml.bz2 dump. You might want to look precisely what this measures (it does not include images or page history, for instance).

curl -s 'http://dumps.wikimedia.org/backup-index.html' |
  grep '<li>' |
  grep '<a href' |
  cut -d '"' -f 2 |
  cut -d '/' -f 1 |
  sort |
  uniq |
  while read p; do
    echo -n "$p "
    curl -s "http://dumps.wikimedia.org/$p/latest/" |
      grep 'pages-articles.xml.bz2"' |
      head -1 |
      cut -d '>' -f 9 |
      cut -d '<' -f 1
    sleep 1
  done |
  awk '
    /K$/ {printf "%s %d\n", $1, $2*1024}
    /M$/ {printf "%s %d\n", $1, $2*1024*1024} 
    /G$/ {printf "%s %d\n", $1, $2*1024*1024*1024}' |
  sort -k2,2nr

Yes, I know, this is really ugly and will break the day Wikimedia changes the layout of the dump service pages in the slightest way, but it works for now. It no longer does. Here is the current result:

enwiki 7301444403
dewiki 2040109465
frwiki 1503238553
commonswiki 1288490188
jawiki 1288490188
itwiki 1181116006
eswiki 1073741824
enwikisource 1041026252
ruwiki 966367641
plwiki 823866163
ptwiki 697093324
nlwiki 647495680
frwikisource 604294348
zhwiki 538548633
svwiki 371195904
cawiki 361863577
huwiki 357878988
zhwikisource 335439462
fiwiki 321493401
ukwiki 311531929
cswiki 297166438
dewikisource 295908147
nowiki 295698432
hewiki 250295091
trwiki 209924915
kowiki 208456908
enwiktionary 203318886
ruwikisource 201431449
viwiki 181403648
arwiki 178782208
rowiki 167981875
dawiki 155923251
idwiki 153826099
srwiki 149631795
bgwiki 147115212
eswikisource 126877696
fawiki 124256256
hrwiki 124256256
arwikisource 122788249
eowiki 118174515
frwiktionary 116706508
skwiki 114294784
thwiki 111463628
ltwiki 103704166
elwiki 102655590
slwiki 101711872
enwikibooks 88814387
itwikisource 88394956
mswiki 82522931
glwiki 82313216
hewikisource 80530636
etwiki 76860620
euwiki 75916902
simplewiki 69310873
shwiki 67423436
hiwiki 66794291
mkwiki 64067993
nnwiki 61236838
plwikisource 60293120
en_labswikimedia 58195968
metawiki 55469670
enwikiquote 54525952
kawiki 50226790
zhwiktionary 50226790
lvwiki 49073356
ptwikisource 47500492
tawiki 47290777
tewiki 45193625
slwikisource 44459622
ruwiktionary 43096473
azwiki 41418752
lawiki 40684748
dewiktionary 40160460
flaggedrevs_labswikimedia 39950745
readerfeedback_labswikimedia 39950745
specieswiki 38692454
bswiki 37224448
knwiki 37224448
dewikibooks 36595302
be_x_oldwiki 34603008
mlwiki 32820428
sqwiki 32505856
brwiki 31771852
plwiktionary 31142707
enwikinews 30513561
bnwiki 30303846
enwikiversity 30094131
iswiki 29150412
kowikisource 28206694
sourceswiki 28206694
cswikisource 27787264
svwikisource 27472691
bewiki 26738688
bgwiktionary 26528972
afwiki 26109542
cywiki 26004684
lbwiki 25794969
huwikisource 25165824
tlwiki 24117248
anwiki 23488102
nowikisource 23278387
ptwiktionary 22858956
lawikisource 22334668
ocwiki 22229811
mrwiki 22124953
srwikinews 21390950
mgwiktionary 20866662
viwiktionary 20552089
fiwiktionary 18874368
elwikisource 18664652
itwikiquote 18454937
vowiki 18350080
rowikisource 18245222
fywiki 18035507
huwiktionary 18035507
thwikisource 17930649
alswiki 17511219
ltwiktionary 17511219
swwiki 17196646
hrwikisource 17091788
jvwiki 17091788
zh_yuewiki 16882073
ndswiki 16672358
siwikibooks 16252928
frwikibooks 16148070
guwiki 16148070
incubatorwiki 16148070
eswiktionary 16043212
urwiki 15938355
nostalgiawiki 15728640
srwikisource 15623782
hywiki 15414067
siwiki 15309209
nlwikisource 14994636
itwikibooks 14889779
trwiktionary 14365491
fawikisource 13946060
astwiki 13841203
elwiktionary 13736345
kowiktionary 13736345
warwiki 13421772
nlwiktionary 13316915
fiwikisource 13212057
gawiki 13107200
mediawikiwiki 13002342
kkwiki 12687769
iowiktionary 12582912
jawiktionary 12268339
eswikibooks 12058624
plwikiquote 12058624
trwikisource 11953766
itwiktionary 11744051
dewikinews 11639193
lmowiki 11324620
tawiktionary 10800332
huwikibooks 10695475
frwikinews 10590617
scnwiki 10590617
jawikisource 10380902
barwiki 10276044
svwiktionary 10276044
iowiki 10171187
mywiki 10066329
iawiktionary 9961472
ukwikisource 9961472
suwiki 9856614
mnwiki 9751756
viwikisource 9542041
cvwiki 9437184
dewikiversity 9437184
quwiki 9437184
ptwikibooks 9332326
newwiki 9017753
yowiki 9017753
arzwiki 8912896
kuwiki 8912896
pmswiki 8912896
plwikinews 8808038
nlwikibooks 8598323
ttwiki 8493465
yiwiki 8388608
tawikisource 8283750
vecwiki 8178892
liwiki 8074035
newiki 7969177
cebwiki 7759462
itwikinews 7759462
plwikibooks 7654604
frwikiversity 7549747
kuwiktionary 7549747
bnwikisource 7444889
dawikisource 7444889
etwiktionary 7235174
mywiktionary 7235174
pswiki 7235174
tewikisource 7235174
eswikiquote 7130316
pamwiki 7025459
scowiki 6920601
uzwiki 6920601
eswikinews 6815744
ruwikiquote 6815744
de_labswikimedia 6710886
zh_classicalwiki 6710886
wawiki 6606028
ruwikibooks 6501171
idwikisource 6396313
jawikibooks 6186598
oswiki 6081740
dewikiquote 5976883
napwiki 5872025
ptwikinews 5872025
zh_min_nanwiki 5872025
hywikisource 5767168
sahwiki 5767168
htwiki 5662310
mtwiki 5662310
sqwikibooks 5662310
cawikisource 5557452
iawiki 5557452
nowiktionary 5557452
bpywiki 5452595
gvwiki 5452595
mwlwiki 5452595
strategywiki 5452595
hsbwiki 5347737
nds_nlwiki 5347737
rowiktionary 5347737
hewikibooks 5138022
ptwikiquote 5138022
vlswiki 5138022
gdwiki 5033164
tgwiki 4928307
foundationwiki 4823449
knwiktionary 4823449
pnbwiki 4823449
kmwiki 4718592
mlwiktionary 4613734
frwikiquote 4508876
jawikinews 4508876
tewiktionary 4404019
betawikiversity 4299161
cswiktionary 4299161
roa_tarawiki 4299161
ukwiktionary 4299161
bat_smgwiki 4194304
fowiki 4194304
itwikiversity 4089446
mlwikisource 4089446
nahwiki 3984588
bswikisource 3879731
amwiki 3774873
azwikisource 3774873
arwiktionary 3670016
ckbwiki 3670016
zhwikinews 3670016
angwiki 3565158
dvwiki 3565158
ganwiki 3565158
mgwiki 3565158
cawiktionary 3460300
ruwikiversity 3460300
eowiktionary 3355443
hifwiki 3355443
hewiktionary 3250585
kshwiki 3250585
eswikiversity 3145728
fiwikibooks 3145728
szlwiki 3145728
wuuwiki 3145728
cewiki 3040870
glwiktionary 3040870
liwiktionary 3040870
lowiktionary 3040870
ruwikinews 2936012
hewikiquote 2831155
rmwiki 2831155
sawikisource 2831155
simplewiktionary 2831155
ugwiki 2831155
vecwikisource 2831155
cowiki 2726297
arwikibooks 2621440
bowiki 2621440
zhwikibooks 2516582
bclwiki 2411724
cswikiversity 2411724
furwiki 2411724
huwikiquote 2411724
idwiktionary 2411724
iswikisource 2411724
kywiki 2411724
map_bmswiki 2411724
stqwiki 2411724
diqwiki 2306867
fiu_vrowiki 2306867
hrwiktionary 2306867
krcwiki 2306867
mkwikisource 2306867
ptwikiversity 2306867
afwiktionary 2202009
bawiki 2202009
extwiki 2202009
hiwiktionary 2202009
iswiktionary 2202009
lawiktionary 2202009
ocwiktionary 2202009
roa_rupwiki 2202009
sawikibooks 2202009
scwiki 2202009
tkwiki 2202009
fawikiquote 2097152
ilowiki 2097152
ladwiki 2097152
mhrwiki 2097152
nrmwiki 2097152
viwikibooks 2097152
bgwikisource 1992294
brwikisource 1992294
cawikibooks 1992294
dsbwiki 1992294
fawiktionary 1992294
pawiki 1992294
sewiki 1992294
sowiki 1992294
thwiktionary 1992294
wawiktionary 1992294
yiwikisource 1992294
zhwikiquote 1992294
cswikibooks 1887436
fawikibooks 1887436
mkwikibooks 1887436
mrjwiki 1887436
orwiki 1887436
bgwikiquote 1782579
emlwiki 1782579
eowikisource 1782579
lijwiki 1782579
mznwiki 1782579
scnwiktionary 1782579
skwikiquote 1782579
trwikiquote 1782579
wowiki 1782579
crhwiki 1677721
euwiktionary 1677721
hsbwiktionary 1677721
kwwiki 1677721
novwiki 1677721
plwikimedia 1677721
svwikibooks 1677721
bjnwiki 1572864
frpwiki 1572864
gnwiki 1572864
kowikibooks 1572864
kvwiki 1572864
miwiki 1572864
nlwikiquote 1572864
nvwiki 1572864
ruewiki 1572864
udmwiki 1572864
brwiktionary 1468006
csbwiki 1468006
cswikiquote 1468006
cywikisource 1468006
lnwiki 1468006
svwikinews 1468006
testwiki 1468006
xalwiki 1468006
bswikiquote 1363148
chrwiki 1363148
jbowiki 1363148
kmwiktionary 1363148
sawiki 1363148
tpiwiki 1363148
trwikibooks 1363148
cbk_zamwiki 1258291
eowikibooks 1258291
fiwikinews 1258291
frrwiki 1258291
idwikibooks 1258291
koiwiki 1258291
mrwikibooks 1258291
nowikibooks 1258291
pcdwiki 1258291
rwwiki 1258291
simplewikiquote 1258291
skwikibooks 1258291
sqwiktionary 1258291
elwikiquote 1153433
fawikinews 1153433
glkwiki 1153433
glwikisource 1153433
hakwiki 1153433
iswikibooks 1153433
mswiktionary 1153433
papwiki 1153433
ukwikimedia 1153433
arwikinews 1048576
azwiktionary 1048576
bugwiki 1048576
dawiktionary 1048576
fywiktionary 1048576
gagwiki 1048576
huwikinews 1048576
iawikibooks 1048576
iewiki 1048576
kbdwiki 1048576
klwiki 1048576
lowiki 1048576
nlwikimedia 1048576
rowikibooks 1048576
rowikinews 1048576
sdwiki 1048576
tawikinews 1048576
ukwikibooks 1016729
kawikibooks 1005363
cswikinews 996352
cvwikibooks 996352
dawikibooks 975667
elwikiversity 962048
swwiktionary 955596
zeawiki 955494
aywiki 954470
bgwikinews 949760
myvwiki 949452
arcwiki 946688
astwiktionary 943718
etwikisource 943718
fiwikiquote 943718
pdcwiki 943718
srwiktionary 943718
slwiktionary 937267
fiwikiversity 933888
vowiktionary 933785
hewikinews 931840
kabwiki 922931
ltwikiquote 922419
kawiktionary 915660
azwikibooks 894873
nlwikinews 894259
cywiktionary 889753
acewiki 889651
hrwikibooks 884224
thwikibooks 878080
kaawiki 869376
glwikibooks 841318
wikimania2007wiki 840192
outreachwiki 832512
jawikiquote 818995
test2wiki 818176
elwikibooks 813772
pflwiki 811315
wikimania2009wiki 805171
wikimania2011wiki 803123
ltwikisource 801382
svwikiquote 788787
jawikiversity 780492
igwiki 776192
ttwikibooks 775680
bhwiki 771379
ukwikinews 757248
cawikinews 754176
abwiki 744140
aswiki 736153
mrwiktionary 732569
skwikisource 725299
mdfwiki 721510
cuwiki 720486
nowikinews 718233
zh_min_nanwiktionary 714547
ltwikibooks 713318
hywiktionary 707174
bgwikibooks 700211
pagwiki 698675
sahwikisource 688640
arwikiquote 687001
slwikiquote 685158
trwikinews 683008
tetwiki 678809
srnwiki 663142
cawikiquote 660582
wikimania2008wiki 646041
towiki 644710
liwikisource 638873
ukwikiquote 621772
nowikiquote 617984
hawwiki 608870
srwikibooks 597299
ltgwiki 594841
rmywiki 590745
pntwiki 583270
lvwiktionary 565248
kowikiquote 565043
wikimania2010wiki 564019
gawiktionary 554496
lbewiki 544870
sawiktionary 516096
elwikinews 514150
sswiki 511180
nawiki 504012
cdowiki 499097
srwikiquote 498380
bmwiki 496025
brwikimedia 492748
tenwiki 489267
kkwiktionary 480665
eowikiquote 475750
avwiki 471654
slwikibooks 471449
eewiki 467046
omwiki 464691
hrwikiquote 464179
hywikiquote 459980
wikimania2006wiki 450867
piwiki 449331
kawikiquote 447488
dawikiquote 441241
kywikibooks 439603
hiwikibooks 439091
rowikiquote 430387
hawiki 429158
svwikiversity 425881
zh_min_nanwikisource 424448
tswiki 421580
viwikiquote 419840
angwiktionary 414310
tawikibooks 413081
idwikiquote 412262
tlwiktionary 411955
smwiki 405401
tywiki 404172
pihwiki 386457
sdwiktionary 382771
bxrwiki 379596
tlwikibooks 374476
mowiki 373555
sqwikinews 367718
simplewikibooks 363520
kgwiki 363315
cowiktionary 352665
iuwiki 350105
yiwiktionary 344985
ttwiktionary 342323
gotwiki 340889
ikwiki 340480
nowikimedia 338329
tkwiktionary 337408
mswikibooks 335667
glwikiquote 332288
newiktionary 329523
sewikimedia 328601
urwiktionary 326860
biwiki 324812
bewiktionary 322048
siwiktionary 320102
usabilitywiki 319078
amwiktionary 318259
skwiktionary 315596
angwikibooks 304025
thwikiquote 294297
mlwikibooks 293273
hiwikiquote 287744
iewiktionary 278630
bnwikibooks 277401
nnwikiquote 276070
etwikiquote 275046
xhwiki 269209
mlwikiquote 266035
bswikinews 264908
lbwiktionary 264806
kswiki 262041
kowikinews 261427
anwiktionary 261324
sdwikinews 261017
wowiktionary 258867
bswiktionary 257843
sgwiki 257024
zawiki 257024
knwikisource 246784
urwikibooks 235929
wikimania2005wiki 234803
kywiktionary 233881
zuwiki 232652
gnwiktionary 227328
etwikibooks 224665
ffwiki 221696
bnwiktionary 214220
nnwiktionary 213606
crwiki 211968
azwikiquote 208793
dzwiki 207974
csbwiktionary 206131
nywiki 205414
thwikinews 205312
cywikiquote 204492
tumwiki 202956
ndswiktionary 198041
eowikinews 195686
stwiki 194867
mkwiktionary 192204
tiwiki 189644
kmwikibooks 188620
iswikiquote 186368
rnwiki 185958
newikibooks 184320
tnwiki 182784
ptwikimedia 182169
vewiki 180224
kuwikiquote 177971
shwiktionary 175206
euwikiquote 174796
lgwiki 169062
chwiki 166297
vowikibooks 160768
fjwiki 154419
klwiktionary 146636
jvwiktionary 146227
uawikimedia 142131
guwiktionary 140800
bewikiquote 132300
snwiki 130457
nahwiktionary 130150
htwikisource 129331
pswikibooks 127590
pswiktionary 125440
ocwikibooks 124108
akwiki 120934
mnwiktionary 120115
mrwikiquote 120012
sqwikiquote 119910
tewikiquote 119296
gdwiktionary 117657
fowiktionary 117555
ugwiktionary 116121
rwwiktionary 111513
twwiki 109260
bswikibooks 107929
tgwiktionary 107008
quwiktionary 104345
hywikibooks 103731
kiwiki 101580
liwikiquote 101273
lawikibooks 100454
brwikiquote 99328
euwikibooks 99225
kuwikibooks 99123
kwwiktionary 97382
afwikiquote 95129
gvwiktionary 93900
tawikiquote 90009
lawikiquote 87449
stwiktionary 85196
mtwiktionary 84889
ruwikimedia 83763
rswikimedia 81510
miwiktionary 80384
tpiwiktionary 79462
ngwiki 79155
chrwiktionary 78233
zuwiktionary 75059
fywikibooks 73932
omwiktionary 73932
bewikibooks 73830
tewikibooks 73113
swwikibooks 70963
iuwiktionary 69939
lvwikibooks 69529
tkwikibooks 67276
sowiktionary 66252
fowikisource 64716
chywiki 63897
tiwiktionary 63692
tgwikibooks 62464
suwiktionary 61235
alswiktionary 60928
cowikimedia 60723
iewikibooks 60723
lnwiktionary 59596
pawiktionary 59596
fiwikimedia 58880
sswiktionary 58265
tswiktionary 58265
cowikibooks 57548
uzwiktionary 57344
nawiktionary 56422
pawikibooks 55398
roa_rupwiktionary 54886
ikwiktionary 54169
mkwikimedia 53964
smwiktionary 53350
hawiktionary 52633
afwikibooks 51916
pa_uswikimedia 51609
chwikimedia 51097
knwikiquote 50688
sgwiktionary 49971
tnwiktionary 46796
guwikiquote 46592
dvwiktionary 46284
kswiktionary 46284
liwikibooks 45568
mowiktionary 45363
fjwiktionary 45056
mgwikibooks 43212
aswiktionary 40448
angwikiquote 38912
trwikimedia 38809
bhwiktionary 38297
abwiktionary 37478
dzwiktionary 37273
orwiktionary 37068
snwiktionary 37068
yowiktionary 36864
biwiktionary 36761
twwiktionary 36761
towiktionary 36249
xhwiktionary 35840
lbwikiquote 35737
bowiktionary 34713
cywikibooks 33894
urwikiquote 33484
aywiktionary 31641
nzwikimedia 28057
uzwikiquote 27545
aawiki 27340
astwikiquote 26931
amwikiquote 26828
suwikibooks 25600
zawiktionary 24985
kywikiquote 23552
advisorywiki 22835
cowikiquote 22630
aawiktionary 21504
astwikibooks 20275
wikimania2012wiki 20275
zh_min_nanwikibooks 19148
iiwiki 18329
zh_min_nanwikiquote 18124
akwikibooks 17920
qualitywiki 17817
jbowiktionary 17408
mhwiki 16384
etwikimedia 14540
chowiki 14028
dkwikimedia 11776
kkwikiquote 11673
kkwikibooks 9420
mnwikibooks 9318
scwiktionary 9113
wowikiquote 8601
knwikibooks 8089
angwikisource 7884
bmwikibooks 7680
ndswikibooks 7680
quwikibooks 7680
nawikibooks 7475
zawikibooks 7270
howiki 7065
xhwikibooks 6860
suwikiquote 6656
ugwikibooks 6553
miwikibooks 6451
kswikibooks 6348
zuwikibooks 6246
guwikibooks 6041
kswikiquote 5836
tkwikiquote 5529
uzwikibooks 5529
chwikibooks 5222
kjwiki 5222
lbwikibooks 5222
lnwikibooks 5222
ugwikiquote 5222
gnwikibooks 5120
gawikibooks 5017
gawikiquote 4915
gotwikibooks 4812
bmwiktionary 4710
hzwiki 4608
nahwikibooks 4505
sewikibooks 4505
bowikibooks 4403
kwwikiquote 4300
rmwikibooks 4300
mhwiktionary 4096
ttwikiquote 4096
alswikibooks 3993
biwikibooks 3993
rmwiktionary 3788
yowikibooks 3788
vowikiquote 3686
avwiktionary 3584
bmwikiquote 3584
aawikibooks 3379
aswikibooks 3276
piwiktionary 3276
rnwiktionary 3276
bawikibooks 3174
crwiktionary 3174
wawikibooks 3072
crwikiquote 2969
mywikibooks 2969
muswiki 2867
nawikiquote 2867
akwiktionary 2764
alswikiquote 2662
krwiki 2560
zawikiquote 2252
aywikibooks 2150
chwiktionary 1945
vewikimedia 1945
quwikiquote 1843
ndswikiquote 1740
krwikiquote 1126

I won't comment much: the biggest wikis are mostly the same (though in a slightly different order), but the Volapük Wikipedia, to continue using this example, is quite lower in the rankings. Actually, the right way to look at this would be to plot the dump size over the number of articles. Here is the result, restricted to the first 80 Wikipedias (not Wikimedia projects) sorted by the number of articles. (If the graph is not correctly displayed below, you can get it here.)

Plot

The Volapük Wikipedia is indeed an outlier, with a few other cases.

comments welcome at a3nm<REMOVETHIS>@a3nm.net