08.31.06
How To: Install Wikipedia For Offline Access
In the old days (say, around 1990) a must-have application when buying a computer was an encyclopedia on a CD-ROM. Hello, Grolier’s and Encarta! No more would you need a shelf full of books to look up interesting facts! When I bought an iBook, it came with a copy of World Book, which I thought quite an entertaining addition.
These days, such an addition is no longer the norm, thanks to the Internet. An incredible amount of information can be gleaned online with a quick search. However, a project started a few years ago has quickly risen to become a great resource for user-provided information on a wide variety of topics. I speak, of course, of Wikipedia. While initially just a quick repository for user feedback, it’s quickly become a resource worthy of comparison to more established sources, such as The Encyclopedia Britannica, even if its veracity may be in question.
I have a laptop, but don’t always have an Internet connection, but wondered, why can’t I have an offline copy of Wikipedia? As it turns out, I can. Now, if I’m on the road and want to look up something quickly, I don’t even have to find a hotspot — I can just turn on my laptop, pull up a browser, and find the answer. This article shows you how I did it.
Overview
Wikipedia runs on the open source software MediaWiki. This in turn runs on top of MySQL and PHP, as well as possibly Linux and Apache. My laptop runs Windows XP Professional SP2 Tablet PC Edition, so running Linux and Apache just wasn’t going to happen. Fortunately, there is a WAMP project (Windows - Apache - MySQL - PHP), which did all the hard work of that installation for me. So, all I’d have to do is:
- Install WAMP.
- Install MediaWiki.
- Download and install a pages dump of Wikipedia.
These instructions should in theory work for any Windows XP SP2 machine. However, your results may vary. I take no responsibility if you try this yourself! Some anticipated caveats:
- You need Administrator privileges. You’re installing software, as well as creating services, so you need the privileges.
- You need disk space. The full English Wikipedia will take a over 10 gigabytes when uncompressed into the database.
- You need NTFS. Because of this, the database files themselves may grow to larger than 2 GB. If you’re using FAT32, you’re out of luck.
- You’re installing a new service. By default, the server installs without remote access, and hopefully, you leave your firewall in place. However, you are still installing new services on your machine, which means they have the potential for exploitation.
- No pictures included. These instructions do not cover the images in Wikipedia.
That said, let’s get on with the show!
Install WAMP.
Go to the Wampserver site and download the latest WAMP distribution (in my case, 1.6.4). Double-click the executable to run, and the defaults will pretty much be what you want. (E.g., install to C:\wamp\, create a Start Menu group, do not auto-start, set DocumentRoot to www, and Launch immediately.)
A Windows Security Alert will probably pop up and ask if you want to keep blocking Apache HTTP Server. You want to select “Keep Blocking” for this question.
Now, in your systray on the lower right side you should see a little dashboard icon, with a lock on it. It should be white, and when you mouse over it, it should say “WAMP5 - All services running - server Offline”. (When they say “offline” here, they actually mean that it’s restricting access to localhost — it’s actually online, technically.
To verify that it’s working, open up a web browser, and point it at http://127.0.0.1/. If the installation was successful, you should see a page that looks like the following:
That’s it for WAMP!
Install MediaWiki.
First, we’ll set up a MySQL user for Wiki. To do so, make sure WAMP is running. (If not, go to Start->Programs->WampServer->Start Wampserver.) Then, go to phpMyAdmin. Click on “Privileges”, then “Add a new User”. Use the following values:
- User name:
wikiuser - Host: Select “Local” from the dropdown
- Password: Select “Use text field” from the dropdown, and pick a password of your choice
- Generate Password: Click the “Generate” button
- Global privileges: Leave all unchecked
Scroll to the bottom and click “Go”, and it should successfully create a user. On this confirmation page, you should have a screen to edit the user if you scroll down. Do so, to the section marked “Database-specific privileges”. Set the dropdown to “Use text field”, and enter “wikidb”. Click “Go”.
You should be presented with a new page for Database-specific privileges. Click the “Check All” link to check all the boxes, and click “Go”.
Download the latest stable release of MediaWiki. At the time of this writing, that was version 1.7.1. It’s a .tar.gz file, so you’ll need a program to expand it — I recommend the shareware program WinRAR. When you unpack this, you’ll create a folder named mediawiki-1.7.1. Rename this to wikipedia, and move it to c:\wamp\www\.
If you now visit http://127.0.0.1/wikipedia/, you should get a splash page saying to “setup the wiki” first. Follow that link, and you should get a “Site config” page. I used these values for this form:
- Wiki name:
Wikipedia Offline - (Admin) Password: custom password
- Database host:
localhost - Database name:
wikidb - DB username:
wikiuser - DB password: same password used when creating MySQL user
The other defaults were fine. Once done, I went in Windows Explorer to C:\wamp\www\wikipedia\config\, and moved the file LocalSettings.php up one directory to C:\wamp\www\wikipedia\.
Another check of http://127.0.0.1/wikipedia/ should state that “MediaWiki has been successfully installed.”
Download and install a pages dump of Wikipedia.
You can download a copy of the English Wikipedia pages from http://download.wikimedia.org/enwiki/latest/. However, you should check this page, for the entry for “enwiki” first, to make sure the dump completed successfully. The file you will want is named enwiki-latest-pages-articles.xml.bz2. This contains all the article pages, but none of the revisions or history. You just want the articles, right? As of this writing, that file is around 1.5 GB, compressed.
If you don’t already, you should make sure you have Java installed. If you don’t, you can get it from http://java.sun.com/j2se/1.5.0/download.jsp. I usually just open a command window and type java and hit enter, and see if it just hangs. If it does, it’s probably installed, and I hit cntrl-C to cancel.
You’ll also need MWDumper. Download mwdumper.jar from http://download.wikimedia.org/tools/. Put this file and the wiki dump file in the same directory, say, c:\tmp\.
You’ll need to edit MySQL’s config file to increase the max_allowed_packet size. If you don’t, the import will most likely choke around the 49,000 article mark. This is quite annoying, because it kills the rest of the import. While you’re add it, you might as well change the innodb_log_file_size, which should modestly increase the import speed. To do so, go to c:\wamp\mysql\, right-click on my.ini, and select Open. This will open up the ini file in a text editor. Find the line innodb_log_file_size, and set this to 512M (was 10M in mine). Scroll to the bottom, and add the following line:
set-variable=max_allowed_packet=32M
Remember that little dashboard with the lock in your systray? Left-click on it, and a menu should pop up. Select MySQL->Stop Service. Wait a few seconds, then left-click on it again, and select MySQL->Start / Resume Service.
Before you import, you’re going to need to delete data in MySQL from the default installation. Otherwise, you’ll get errors about a dupe right at the start, and then none of the rows will import. Left-click on the dashboard with a lock in your systray, go to “MySQL”, and select “MySQL console”. You’ll be asked for a password, which by default is blank, so just hit enter. Enter in the following commands into the console:
use wikidb; delete from page; delete from revision; delete from text; quit
This will delete all pages in the wiki.
Open a command window by going to Start->Run, and typing in cmd. Type c: to change to the c: drive, and then cd c:\tmp\ to change to the directory where you put mwdumper.jar and the wiki dump file. You’re ready to do the import, but beware — this will take a long time. It’s best to start the process, then leave for a few hours. When you’re ready, type the following:
java -jar mwdumper.jar --format=sql:1.5 enwiki-latest-pages-articles.xml.bz2 | c:\wamp\mysql\bin\mysql -u wikiuser -p wikidb
This will begin the import process, and as noted, this will take a long time. There are over three million pages to process, so don’t expect it to finish right away. On a reasonably fast single processor machine (*not* my laptop), it took me over 24 hours.
Usage
Using Wikipedia Offline is pretty straightforward. If you haven’t already, start WAMP. (If you see the dashboard with a lock icon in your systray, and it’s white, then it’s running. If not, go to Start->Programs->WampServer->Start Wampserver.) Then, just fire up a web browser and browse to http://127.0.0.1/wikipedia/. If all goes well, it should be accessible just like Wikipedia, searches and all.
Anticipated Questions
- Why do this?
I’m not always connected to the Internet, and think Wikipedia is a great resource. Now I can take it wherever I want. I suppose if I were paranoid about Wikipedia tracking my searches, then I could do this and do all the searches I wanted offline. Doing it this way also seemed like a fun tech project. - Is this legal?
Sure! Wikipedia offers all of their data for use by interested parties. All of the software involved is open source, except for Windows. - Where are the pictures?
You can download a dump of the English Wikipedia images from here. Wikipedia doesn’t package these with the dump for two reasons: 1) the images might be copyrighted, so they don’t want to distribute them; and 2) the dump file would be huge. As of this writing, the dump file is about 75 GB, which was larger than the hard drive on my laptop. - Isn’t it overkill to install full MediaWiki?
Yes, but it’s not nearly as much effort as you might think. Plus, with WAMP, I can experiment with other types of LAMP-based software. You can always build static pages if you’d prefer something a little more lightweight. - Won’t the data fall out of date?
Yes, but I’m doing this more to just have a quick reference, rather than something that’s kept constantly up to date. In that sense, it’s similar to those encyclopedia CD-ROMs! Besides, the way the dumps are handled, you’re guaranteed to be slightly out of date. If you really need to be that current, you should probably be going online. - How can updates be done?
I’m presuming I can just go in mysql and delete from the ‘page’, ‘revision’, and ‘text’ tables; download a new dump; and re-import using mwdumper.jar. I haven’t actually tried this, though. - Can I use these instructions to run a wiki web site?
There are a few problems with this. First, the WAMP folk note that “WAMP5 is not meant to be a production server.” Also, running a web site takes a fair bit of security knowledge to prevent hacking, so you’ll get yourself in trouble if you just use it to publish on the ‘net. Finally, you can’t republish the contents of Wikipedia as your own site. So, technically, you could use the first few parts to set up wiki, but it’s not a good idea. You’re better off getting a proper web host that has a one-click install of wiki, such as DreamHost. - How do I uninstall?
If you want to trash the whole thing, go to “Add or Remove Programs” in the Control Panel, and select WAMP5. Remove it, then be sure to deleteC:\wamp\as well.
ronsahtol said,
November 6, 2006 at 11:25 am
Help! Thanks to your very helpful instructions, but I got stuck at some very important step. I have already installed wampserver, downloaded the huge dump file, downloaded the mwdumper.jar file, installed JRE; everything was going so fine and I was so giddy up in anticipation for the Wikipedia that’s going to grace my hard drive–up until this stage:
“Before you import, you’re going to need to delete data in MySQL … You’ll be asked for a password, which by default is blank, so just hit enter…”
I couldn’t get to hit enter and do the next step, because the console window just disappeared! I couldn’t quite catch what was written on the screen before that window went away, but I guess it probably was “Error 2003″ or something.
When I looked at the system tray and hovered the mouse pointer over the wamp icon,there was this message “1 of 2 services running.” When I went to http://127.0.0.1/, I saw this message “MySQL not launched or bad phpmyadmin config.” When I clicked “Wikipedia” under “Your projects,” I got this page instead “Wikipedia Offline has a problem…(Can’t contact the database server: Can’t connect to MySQL server on ‘localhost’ (10061)).
Please help, oh you kind one. Let me say that I am a complete idiot with computers, so advise accordingly.
Thank you very much!
Techie said,
November 6, 2006 at 12:20 pm
Hi ronsahtol,
Wow, it’s an ambitious project installing wikipedia locally.
Based on your error messages, it sounds like for some reason MySQL isn’t starting. (You can’t do the import if MySQL isn’t running, and that would also be why Wiki isn’t showing up.) Check C:\wamp\logs\mysql_error.log for why it’s dying.
ben said,
December 4, 2006 at 2:21 am
im a nerd, and i finally got it
i did not expereience that error
i did experience some java errors but i fixed it
the new line should read
java -jar mwdumper.jar –format=sql:1.5 enwiki-latest-pages-articles.xml mysql -u wikiuser -p wikidb
it increases in speed as its adding so far 2.1 mil pages in 20 minutes 2325 pages per second now, it was 600 per second at beginning
Techie said,
December 4, 2006 at 10:37 am
Hey ben,
Great tip. So basically, you unbzipped the xml dump first? I’m surprised that uncompressing on the fly had that big an impact!
lance said,
December 31, 2006 at 6:29 pm
Im having the exact same problem as the first person who wrote a comment(ronsahtol )..can nayone please HELP [email redacted] thanks
Download wikipedia, request & instructions :) - TechEnclave said,
February 8, 2007 at 12:52 am
[...] Download wikipedia, request & instructions
Hi all, Can somebody download http://download.wikimedia.org/enwiki…ticles.xml.bz2 for I want to have a local wikipedia repositery as given here Blinded By Tech How To: Install Wikipedia For Offline Access I’m actually hoping for somebody to make/host a .torrent out of this so people like me can download it without waiting for hrs. In the long-run having wikipedia on the desktop ————————- Advertisement ————————- __________________ Poorest Man’s Rig This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivs 2.5 License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/2.5/ or send a letter to Creative Commons, 543 Howard Street, 5th Floor, San Francisco, California, 94105, USA. [...]
shirish said,
February 8, 2007 at 4:15 am
Hi all,
First of all thnx & congrats for doing this. I’m trying to do this on windows box. Some things the latest wikimedia is mediawiki-1.9.1 which has slightly things to http://127.0.0.1/wikipedia/mediawiki-1.9.1/ instead of http://127.0.0.1/wikipedia/ . Also with Java SE 6 coming up would it be http://java.sun.com/javase/6/ or something else. Looking forward to updates from u or any differences in things. Thnx in advance
WAMP with MediaWiki said,
February 11, 2007 at 12:55 am
For a WAMP Server, there is also the Web-Developer Server Suite. And it comes with MediaWiki installed.
Brendan said,
April 29, 2007 at 4:01 pm
Hi,
I managed to get everything installed, but now am having some weird display issues on many of the article pages. I see things like “{{#ifexpr:{{#if:l{{#time:UItoday}}>” … etc, and a huge chunk of the text winds up in the left column. I’m wondering if anyone else has had this problem, or knows what I might be able to do to fix it — I’d really rather not have to reimport the dump, since it took forever and a day on this computer!
Thanks!
shlomo said,
May 14, 2007 at 6:09 am
The mwdumper.jar I downloaded recently was not configured correctly to run with the -jar java option. And I bunzip’d the archive first, for speedy importing. This is the import command-line I used:
java -classpath mwdumper.jar org.mediawiki.dumper.Dumper –format=sql:1.5 enwiki-20070402-pages-articles.xml | mysql -u root -p wp
Timbo said,
May 31, 2007 at 7:14 pm
I have a solution and a problem.
ronsahtol and lance. This is what you have to do. Go to c:\wamp\mysql\data and delete ib_logfile0 and ib_logfile1 (both those files would have been 10Mb). Now left-click the dashboard then MySQL then Start/Resume Service. Wait 30 seconds and the two log files should reappear, but this time 512Mb (like the setting you changed before in my.ini).
Now, here’s my question: I got to page 1,144,000 and then things really sped up. Unfortunately this was not a good thing, but the indication of a bad thing. It seems that mwdumper was doing it’s part, but mySQL failed to import any articles above 1,144,000. No error messages. I checked mysql error log and there was nothing. I’ve got access to about 60% of Wikipedia, which is nice, but even key articles are missing.
What can I do?
Timbo said,
June 6, 2007 at 11:01 pm
OK. I solved my problem. This is what worked on my laptop with 512MB RAM: I went into my.ini and (changed/added) these settings to:
innodb_buffer_pool_size=256M
innodb_log_file_size=256M
max_allowed_packet = 64M
innodb_thread_concurrency=1
I wasn’t patient enough to find out what made it work, but I think it was the last one that did it (innodb_thread_concurrency=1).
Using the dump from 2007-04-02, MySQL’s data file grew to nearly 18GB and it took more than 50 hours!!
Brendan, you’ll be happy to hear that you will not have to import the dump again. The solution to the strange tag problem that Brendan was having is to use some extensions.
Go to http://svn.wikimedia.org/svnroot/mediawiki/trunk/extensions/ParserFunctions
Download both of these files and put them in a new directory called ParserFunctions in your extensions directory.
* Expr.php
* ParserFunctions.php
Then put the following at the end of your LocalSettings.php:
require_once( “$IP/extensions/ParserFunctions/ParserFunctions.php” );
There are many more useful extensions in http://svn.wikimedia.org/svnroot/mediawiki/trunk/extensions/ but I’m still not sure which ones are more useful. For each of these you download, use a similar installation technique as you used for ParserFunctions. Try installing Citation and Cite.
This information was sourced from http://meta.wikimedia.org/wiki/ParserFunctions#Installation
I still haven’t got it to work 100%. Probably need some other extension or something. The article on Australia is pretty messed up, with the left link panel being right down the page.
I’m still trying to get mathematical equations to work as well. Any questions, email [redacted. Ed. Note: Don't put email addresses in posts, for your own good. ;)]
Timbo said,
June 6, 2007 at 11:06 pm
By the way, after you change those 4 setting in my.ini, don’t forget to left-click the dashboard/MySQL/Stop Service then go to c:\wamp\mysql\data and delete ib_logfile0 and ib_logfile1. Now left-click the dashboard then MySQL then Start/Resume Service. Wait 30 seconds and the two log files should reappear, but this time 256Mb (like the setting you changed before in my.ini).
Timbo said,
June 7, 2007 at 1:46 am
To display equations you may use this technique: http://www.mediawiki.org/wiki/Manual:Running_MediaWiki_on_Windows#Alternative_Solution
It worked fine the first time!!! (unlike everything else this week)
Timbo said,
June 7, 2007 at 2:53 am
If you’re going to open some page which with lots of equations (like List_of_integrals_of_trigonometric_functions), you’ll need to increase your maximum execution time to more than 30 seconds because rendering that many equations will take a minute or two!
Open c:\wamp\php\php.ini and change
max_execution_time =180
Do the same in c:\wamp\apache2\php.ini
Now go to the dashboard, left-click, Apache->Stop Service. Wait a few seconds and choose dashboard->Apache->Start/Resume Service.
Test it: http://localhost/wikipedia/index.php/List_of_integrals_of_trigonometric_functions
Christian said,
June 26, 2007 at 1:35 pm
Hi there!
Thank you very much for your information. It helped me tremendously in getting a working local installation of wikipedia (I’m doing this for research, but it’s fun too
I’d like to call attention to one piece of advice for reducing the bottlenecking produced by mwdumper having to wait for mysql operations to complete:
http://www.chuma.org/2006/10/19/real-mobile-wikipedia/
If you scroll down to where the command ’split’ is shown, you will find a way to make the overall operation faster.
Hope that’s useful to someone
Robin said,
August 6, 2007 at 4:41 pm
Hello!
Thanks for this very helpful page, but I got stuck where it says « If you now visit http://127.0.0.1/wikipedia/, you should get a splash page saying to “setup the wiki” first. Follow that link, and you should get a “Site config” page. »
At http://127.0.0.1/wikipedia/ there are only links “Parent Directory” and “mediawiki-1.10.0.tar.gz”
I hope someone can help me
Techie said,
August 6, 2007 at 4:48 pm
Robin, It sounds like your unpacking didn’t go correctly. Check to make sure you see files in C:\wamp\www\wikipedia\ (and not just the mediawiki-1.10.0.tar.gz archive). You may need to re-unpack the archive.
Robin said,
August 7, 2007 at 6:06 am
Which files do I have to see in C:\wamp\www\wikipedia\ ? I see only the mediawiki-1.10.0.tar.gz file.
michael said,
August 7, 2007 at 11:36 am
Thanks. The Wikipedia works well. I would like that somebody helps me in something:
When I go to the Main_page of the Wikipedia offline and do click in some category (telecommunications for example) do not appear the articles of this category.
How to do to watch those articles (as it happens in wikipedia online)?
Techie said,
August 7, 2007 at 12:31 pm
Robin, the mediawiki-1.10.0.tar.gz file is an archive of the files that should be in c:\wamp\www\wikipedia\ . You need to unpack this file and put the contents in that directory. You’ll need a program like WinRAR to unpack this file.
Robin said,
August 7, 2007 at 12:51 pm
OK, it does now say “Set up the wiki first”
Thank you very much, Techie
Robin said,
August 7, 2007 at 1:14 pm
Sorry to bother you again, but if I fill in what I supose to fill in, and I click on “install”, I get a big red error with “Something’s not quite right yet; make sure everything below is filled out correctly.”
Robin said,
August 7, 2007 at 1:53 pm
I have tried something, and it works! I have my wiki!
smudge said,
August 14, 2007 at 8:44 pm
Not having internet access at home, I’m eagerly looking forward to having wikipedia offline.
I followed the above installation procedures including the (necessary) error resolution by Timbo but when I went to import the dump I got this:
M:\wamp\pediadump>java -classpath mwdumper.jar org.mediawiki.dumper.Dumper –format=sql:1.5 enwiki-latest-pages-articles.xml | mysql -u root -p wikidb
Enter password: Exception in thread “main” java.io.FileNotFoundException: -ûform
at=sql:1.5 (The system cannot find the file specified)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(Unknown Source)
at java.io.FileInputStream.(Unknown Source)
at org.mediawiki.dumper.Tools.openInputFile(Unknown Source)
at org.mediawiki.dumper.Dumper.main(Unknown Source)
(I used root instead wikiuser because wikiuser resulted in a permissions error.)
I tried what Ben posted and received this:
M:\wamp\pediadump>java -jar mwdumper.jar -format=sql:1.5 enwiki-latest-pages-articles.xml mysql -u wikiuser -p wikidb
Exception in thread “main” java.io.FileNotFoundException: ûformat=sql:1.5 (The s
ystem cannot find the file specified)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(Unknown Source)
at java.io.FileInputStream.(Unknown Source)
at org.mediawiki.dumper.Tools.openInputFile(Unknown Source)
at org.mediawiki.dumper.Dumper.main(Unknown Source)
I tried with Java 1.5 and 1.6 to no avail. As I know zippo about Java and MySQL I’m at a loss as what to do next.
(note:/pediadump is the folder I put the decompressed wikipedia dump and mwdumper.jar into)
Also I had to put the mysql.exe file into the /pediadump folder to eliminate the following:
M:\wamp\pediadump>java -classpath mwdumper.jar org.mediawiki.dumper.Dumper –format=sql:1.5 enwiki-latest-pages-articles.xml | mysql -u root -p wikidb
Exception in thread “main” java.io.FileNotFoundException: -ûformat=sql:1.5 (The
system cannot find the file specified)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(Unknown Source)
at java.io.FileInputStream.(Unknown Source)
at org.mediawiki.dumper.Tools.openInputFile(Unknown Source)
at org.mediawiki.dumper.Dumper.main(Unknown Source)
‘mysql’ is not recognized as an internal or external command,
operable program or batch file.
The ‘–format=sql:1.5′ seems to be the problem because when I removed it the following occurred:
M:\wamp\pediadump>java -classpath mwdumper.jar org.mediawiki.dumper.Dumper enwiki-latest-pages-articles.xml | mysql -u root -p wikidb
Enter password:
ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use n
ear ‘
smudge said,
August 14, 2007 at 8:47 pm
Here is the rest of my post that was cut off.
ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use n
ear ‘
smudge said,
August 14, 2007 at 8:49 pm
Sorry, I keep getting cut off.
near ‘
smudge said,
August 14, 2007 at 8:50 pm
This is getting crazy.
ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use n
ear ‘?xml version=”1.0″ encoding=”utf-8″ ?
smudge said,
August 14, 2007 at 8:51 pm
O.K. I think I found the rest of my post…
ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use n
ear ‘
mediawiki xmlns=”http://www.mediawiki.o’ at line 1
1,000 pages (326.584/sec), 1,000 revs (326.584/sec)
2,000 pages (486.736/sec), 2,000 revs (486.736/sec)
3,000 pages (563.063/sec), 3,000 revs (563.063/sec)
I stopped it at this point but prior when I let it proceed it went all the way to over 5 million pages gradually increasing speed. Afterwards when I went to http://127.0.0.1/wikipedia/. It said ‘There is currently no text in this page, you can search for this page title in other pages or edit this page.’
I’m assuming mwdumper did its thing but MySql did not, hence the empty page.
smudge said,
August 21, 2007 at 5:59 pm
Good news! After much trial and error (mostly error) I got it working. What I found that made it functional was to move -format=sql:1.5 to the other side of the file name like this:
java -classpath mwdumper.jar org.mediawiki.dumper.Dumper enwiki-latest-pages-articles.xml –format=sql:1.5 | mysql -u root -p wikidb
Although I thought with using the settings mentioned above in Timbo’s post and uncompressing the dump first I would have quick results but the page import rate slowly went down to 40/sec (the final speed at the end of importing). I don’t know how Ben got such a fast import rate!
Just FYI the August ‘07 (I think that was the latest one I downloaded) Wikipedia dump was over 2Gb compressed, over 11Gb uncompressed and over 20Gb after importing with over 5.4 million pages. It took over 30 hours to import and resulted in over 2.1 million articles. I used an external hard drive and installed/imported everything on it.
Now I just have to get the parsers and rendering working out. Timbo’s right the article on Australia is horrible!
For any interested this link in wikimedia’s wiki-meta may help http://meta.wikimedia.org/wiki/Data_dumps
Has anybody tried to successfully get Wiktionary and/or another Wiki project alongside the Wikipedia setup. My goal is to have a simple reference suite. So far I keep getting the following error when importing:
Exception in thread “main” java.lang.IllegalArgumentException: I
nput already set; can’t set to -format=sql:1.5
at org.mediawiki.dumper.Dumper.main(Unknown Source)
Also I apologize for my previous posts. The ‘greater than’ and ‘less than’ signs were causing the truncating and since I had no preview page I didn’t know that this was happening until the the comment was posted.
Mark Breen said,
August 31, 2007 at 8:50 am
After a lot of experimentation this is how i got it to work. note the two hyphens before the word format.
java -jar mwdumper.jar –format=sql:1.5 enwiki-latest-pages-articles.xml | mysql -u wikiuser -p wikidb
I spotted this practice on this page http://www.mediawiki.org/wiki/MWDumper
165,000 pages imported only 5.3million to go
Mark Breen said,
August 31, 2007 at 8:54 am
This website must format two hyphens next to one another as a long hyphen. Thats why the code wasnt working for me. the relevant section of the code should be mwdumper.jar HYPHENHYPHENformat=sql:1.5
Federico said,
September 5, 2007 at 4:38 pm
Great article!! Just what I was looking for, after reading the OLPC will include a reduced version of Wikipedia in the laptops and I wondered how could that be done.
The anAlyst said,
October 9, 2007 at 11:09 pm
[...] busy yet .. installing the fresh dump of wikipedia for the college LAN. After a few failures by using xml2sql,etc, Now am using mwdumper and directly putting the data into mysql. Here is one of the best pages that I found for installing wikipedia on a local computer. PS: Am working on Fedora 7. Windows user may find this useful. Other than this, there is also webaroo for windows. I have even tested this with windows Vista. The webaroo uses about 5.5 GB of disk space. Whereas the actual wikipedia dump for august is about 12 GB in xml format and 2.6 GB approx in bz2 format. The latest dumps can be downloaded from here. As to what you should download from there, read this. [...]
lee said,
November 3, 2007 at 2:03 am
I cant get the mwdumper to work. When I run it i get a mysql is not a recognized as an internal or external command error. I tried all the versions listed and still cant get it to work. Thanks in advance.
Quentin de Metz said,
November 4, 2007 at 11:17 am
I just read this and started to convert my extracted .xml dump, and the speed wasn’t great. At the beginning the rate was 170 pages/second, but after an hour or so it had dropped down to 45 pages/second. I got kinda bored, and closed the command prompt. I just restarted the process minutes ago, typing in the same command:
C:\tmp>java -jar mwdumper.jar –format=sql:1.5 enwiki-latest-pages-articles.xml | C:\Wamp\Mysql\bin\mysql like it says up there (not the two hyphens before format), and the speed is amazing. It started at 600 pages/second, and kept going faster. Right now it’s at 4000 pages/second at 4.6 million pages already (in 20 minutes). Weird, isn’t it ? My computer is new with a dual core processor 2×1.9 GHz and I have 2 Gigs of RAM. I don’t know if the conversion is working. I’ll post again when it’s over (like 10 minutes from now). I sure hope it works, I don’t feel like waiting for 48 hours.
Quentin de Metz said,
November 4, 2007 at 11:19 am
oops wrong command,
I typed in:
java -jar mwdumper.jar format=sql:1.5 enwiki-latest-pages-articles.xml | C:\Wamp\Mysql\bin\mysql -u wikiuser -p wikidb
and I just reached 5.6 million pages….
Quentin de Metz said,
November 4, 2007 at 11:24 am
done but it doesn’t work, i still get the
“There is currently no text in this page, you can search for this page title in other pages or edit this page.” message when accessing 127.0.0.1/wikipedia
Will try and figure it out
heimish said,
November 23, 2007 at 3:19 am
I have been having problems converting the dump to sql, but I finally figured out what’s wrong, so I wanted to share it with you all.
If you are getting this error:
main” java.io.FileNotFoundException: -ûformat=sql:1.5 (The system cannot find the file specified) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.(Unknown Source) at java.io.FileInputStream.(Unknown Source) at org.mediawiki.dumper.Tools.openInputFile(Unknown Source) at org.mediawiki.dumper.Dumper.main(Unknown Source)
than you probably copied and pasted the instructions into you command window.
solution? when you paste it, just delete the hyphens and retype it manually. It seems that the paste operation will mess up your life by providing a different character than intended. So, just delete the hyphens and type them again. Presto. You wiki is getting installed.
heimish said,
November 23, 2007 at 3:33 am
Smudge,
I think your problem was fixed because eventually you ended up typing the command instead of pasting it.
Is that possible?
goofy said,
December 8, 2007 at 2:46 am
Why I use less than one hour to dump it?
But the page show no content in this page,why?
priya said,
December 10, 2007 at 3:37 am
hey i too got that 2003 error
for that in mysql i ignored this step and now it is working:
innodb_log_file_size, and set this to 512M (was 10M in mine).
Geana said,
December 19, 2007 at 1:03 am
Thank you very much for this instructions and to the people in the comments, they helped me get this installed. It took about 60hs and took up a chunk of my hard drive. By the way, is it ok to delete the xml file? It’s pretty big.
I’m having some trouble using it, I was wondering if any of you had them and knew how to fix them.
I can click on the articles featured in the front page, but I can’t search or browse for other articles, they don’t come up even if I type up the title, I can only follow links and thus access very few articles.
It’s not just the search function, it seems that anything that requires more than just showing the article doesn’t work. Like, if I click on ‘featured article’, or pretty much any of the links on the left, they won’t work.
Also I changed the maximum execution time in php.ini but it still times out at 30s and articles take a while to come up. In addition, articles look pretty messy, lots of symbols sometimes cover the whole screen, and the boxes in the right are not well aligned (they randomly intersect the main text).
It took a lot of trying and error to get it installed, specially importing the dump file. So, maybe that messed up my installation? I would be very grateful if anyone could point me in the right direction in fix this.
Thank you very much in advance for any insight into this, and thanks again for posting the instructions in the first place.
Mark said,
January 19, 2008 at 10:52 pm
“Finally, you can’t republish the contents of Wikipedia as your own site.”
Actually, you can. That’s the whole point behind the GFDL. This is exactly how Wikipedia became such a popular site in the first place - other sites started popping up mirroring Wikipedia content. This was possible precisely because of the GFDL.
Yicheng Zheng said,
January 24, 2008 at 12:37 am
***** Mwdumper.jar — Readme.txt *****
— Now, everything is OK! —
A quick little tool for extracting sets of pages from a MediaWiki dump file.
Can read MediaWiki XML export dumps (version 0.3, minus uploads), perform
optional filtering, and output back to XML or to SQL statements to add things
directly to a database in 1.4 or 1.5 schema.
Still very much under construction.
MIT-style license like our other Java/C# tools; boilerplate to be added.
Contains code from the Apache Commons Compress project for cross-platform
bzip2 input/output support (Apache License 2.0).
If strange XML errors are encountered under Java 1.4, try 1.5:
* http://java.sun.com/j2se/1.5.0/download.jsp
* http://www.apple.com/downloads/macosx/apple/java2se50release1.html
USAGE:
Sample command line for a direct database import:
java -jar mwdumper.jar –format=sql:1.5 pages_full.xml.bz2 |
mysql -u -p
You can also do complex filtering to produce multiple output files:
java -jar mwdumper.jar \
–output=bzip2:pages_public.xml.bz2 \
–format=xml \
–filter=notalk \
–filter=namespace:\!NS_USER \
–filter=latest \
–output=bzip2:pages_current.xml.bz2 \
–format=xml \
–filter=latest \
–output=gzip:pages_full_1.5.sql.gz \
–format=sql:1.5 \
–output=gzip:pages_full_1.4.sql.gz \
–format=sql:1.4 \
pages_full.xml.gz
A bare parameter will be interpreted as a file to read XML input from;
if none is given or “-” input will be read from stdin. Input files with
“.gz” or “.bz2″ extensions will be decompressed as gzip and bzip2 streams,
respectively.
Internal decompression of 7-zip .7z files is not yet supported; you can
pipe such files through p7zip’s 7za:
7za e -so pages_full.xml.7z |
java -jar mwdumper –format=sql:1.5 |
mysql -u -p
Defaults if no parameters are given:
* read uncompressed XML from stdin
* write uncompressed XML to stdout
* no filtering
Output sinks:
–output=stdout
Send uncompressed XML or SQL output to stdout for piping.
(May have charset issues.) This is the default if no output
is specified.
–output=file:
Write uncompressed output to a file.
–output=gzip:
Write compressed output to a file.
–output=bzip2:
Write compressed output to a file.
–output=mysql:
Valid only for SQL format output; opens a connection to the
MySQL server and sends commands to it directly.
This will look something like:
mysql://localhost/databasename?user=&password=
Output formats:
–format=xml
Output back to MediaWiki’s XML export format; use this for
filtering dumps for limited import. Output should be idempotent.
–format=sql:1.4
SQL statements formatted for bulk import in MediaWiki 1.4’s schema.
–format=sql:1.5
SQL statements formatted for bulk import in MediaWiki 1.5’s schema.
Both SQL schema versions currently require that the table structure
be already set up in an empty database; use maintenance/tables.sql
from the MediaWiki distribution.
Filter actions:
–filter=latest
Skips all but the last revision listed for each page.
FIXME: currently this pays no attention to the timestamp or
revision number, but simply the order of items in the dump.
This may or may not be strictly correct.
–filter=list:
Excludes all pages whose titles do not appear in the given file.
Use one title per line; blanks and lines starting with # are
ignored. Talk and subject pages of given titles are both matched.
–filter=exactlist:
As above, but does not try to match associated talk/subject pages.
–filter=namespace:[!]
Includes only pages in (or not in, with “!”) the given namespaces.
You can use the NS_* constant names or the raw numeric keys.
–filter=notalk
Excludes all talk pages from output (including custom namespaces)
–filter=titlematch:
Excludes all pages whose titles do not match the regex.
Misc options:
–progress=
Change progress reporting interval from the default 1000 revisions.
–quiet
Don’t send any progress output to stderr.
PERFORMANCE TIPS:
To speed up importing into a database, you might try:
* Java’s -server option may significantly increase performance on some
versions of Sun’s JVM for large files. (Not all installations will
have this available.)
* Increase MySQL’s innodb_log_file_size. The default is as little as 5mb, but
you can improve performance dramatically by increasing this to reduce
the number of disk writes. (See the my-huge.cnf sample config.)
* If you don’t need it, disable the binary log (log-bin option) during the
import. On a standalone machine this is just wasteful, writing a second
copy of every query that you’ll never use.
* Various other wacky tips in the MySQL reference manual at
http://dev.mysql.com/mysql/en/innodb-tuning.html
TODO:
* Add some more junit tests
* Include table initialization in SQL output
* Allow use of table prefixes in SQL output
* Ensure that titles and other bits are validated correctly.
* Test XML input for robustness
* Provide filter to strip ID numbers
* is technically optional; live without it and use default namespaces
* GUI frontend(s)
* Port to Python?
Change history (abbreviated):
2005-10-25: Switched SqlWriter.sqlEscape back to less memory-hungry StringBuffer
2005-10-24: Fixed SQL output in non-UTF-8 locales
2005-10-21: Applied more speedup patches from Folke
2005-10-11: SQL direct connection, GUI work begins
2005-10-10: Applied speedup patches from Folke Behrens
2005-10-05: Use bulk inserts in SQL mode
2005-09-29: Converted from C# to Java
2005-08-27: Initial extraction code
Yicheng Zheng said,
January 24, 2008 at 12:44 am
******* Mwdumper.jar — Readme.txt *******
— Sorry! There’s something converted unexcepted. —
— Focus at: –format=sql:1.5 —
— Not be: - format=sql:1.5 —
A quick little tool for extracting sets of pages from a MediaWiki dump file.
Can read MediaWiki XML export dumps (version 0.3, minus uploads), perform
optional filtering, and output back to XML or to SQL statements to add things
directly to a database in 1.4 or 1.5 schema.
Still very much under construction.
MIT-style license like our other Java/C# tools; boilerplate to be added.
Contains code from the Apache Commons Compress project for cross-platform
bzip2 input/output support (Apache License 2.0).
If strange XML errors are encountered under Java 1.4, try 1.5:
* http://java.sun.com/j2se/1.5.0/download.jsp
* http://www.apple.com/downloads/macosx/apple/java2se50release1.html
USAGE:
Sample command line for a direct database import:
java -jar mwdumper.jar –format=sql:1.5 pages_full.xml.bz2 |
mysql -u -p
You can also do complex filtering to produce multiple output files:
java -jar mwdumper.jar \
–output=bzip2:pages_public.xml.bz2 \
–format=xml \
–filter=notalk \
–filter=namespace:\!NS_USER \
–filter=latest \
–output=bzip2:pages_current.xml.bz2 \
–format=xml \
–filter=latest \
–output=gzip:pages_full_1.5.sql.gz \
–format=sql:1.5 \
–output=gzip:pages_full_1.4.sql.gz \
–format=sql:1.4 \
pages_full.xml.gz
A bare parameter will be interpreted as a file to read XML input from;
if none is given or “-” input will be read from stdin. Input files with
“.gz” or “.bz2″ extensions will be decompressed as gzip and bzip2 streams,
respectively.
Internal decompression of 7-zip .7z files is not yet supported; you can
pipe such files through p7zip’s 7za:
7za e -so pages_full.xml.7z |
java -jar mwdumper –format=sql:1.5 |
mysql -u -p
Defaults if no parameters are given:
* read uncompressed XML from stdin
* write uncompressed XML to stdout
* no filtering
Output sinks:
–output=stdout
Send uncompressed XML or SQL output to stdout for piping.
(May have charset issues.) This is the default if no output
is specified.
–output=file:
Write uncompressed output to a file.
–output=gzip:
Write compressed output to a file.
–output=bzip2:
Write compressed output to a file.
–output=mysql:
Valid only for SQL format output; opens a connection to the
MySQL server and sends commands to it directly.
This will look something like:
mysql://localhost/databasename?user=&password=
Output formats:
–format=xml
Output back to MediaWiki’s XML export format; use this for
filtering dumps for limited import. Output should be idempotent.
–format=sql:1.4
SQL statements formatted for bulk import in MediaWiki 1.4’s schema.
–format=sql:1.5
SQL statements formatted for bulk import in MediaWiki 1.5’s schema.
Both SQL schema versions currently require that the table structure
be already set up in an empty database; use maintenance/tables.sql
from the MediaWiki distribution.
Filter actions:
–filter=latest
Skips all but the last revision listed for each page.
FIXME: currently this pays no attention to the timestamp or
revision number, but simply the order of items in the dump.
This may or may not be strictly correct.
–filter=list:
Excludes all pages whose titles do not appear in the given file.
Use one title per line; blanks and lines starting with # are
ignored. Talk and subject pages of given titles are both matched.
–filter=exactlist:
As above, but does not try to match associated talk/subject pages.
–filter=namespace:[!]
Includes only pages in (or not in, with “!”) the given namespaces.
You can use the NS_* constant names or the raw numeric keys.
–filter=notalk
Excludes all talk pages from output (including custom namespaces)
–filter=titlematch:
Excludes all pages whose titles do not match the regex.
Misc options:
–progress=
Change progress reporting interval from the default 1000 revisions.
–quiet
Don’t send any progress output to stderr.
PERFORMANCE TIPS:
To speed up importing into a database, you might try:
* Java’s -server option may significantly increase performance on some
versions of Sun’s JVM for large files. (Not all installations will
have this available.)
* Increase MySQL’s innodb_log_file_size. The default is as little as 5mb, but
you can improve performance dramatically by increasing this to reduce
the number of disk writes. (See the my-huge.cnf sample config.)
* If you don’t need it, disable the binary log (log-bin option) during the
import. On a standalone machine this is just wasteful, writing a second
copy of every query that you’ll never use.
* Various other wacky tips in the MySQL reference manual at
http://dev.mysql.com/mysql/en/innodb-tuning.html
TODO:
* Add some more junit tests
* Include table initialization in SQL output
* Allow use of table prefixes in SQL output
* Ensure that titles and other bits are validated correctly.
* Test XML input for robustness
* Provide filter to strip ID numbers
* is technically optional; live without it and use default namespaces
* GUI frontend(s)
* Port to Python?
Change history (abbreviated):
2005-10-25: Switched SqlWriter.sqlEscape back to less memory-hungry StringBuffer
2005-10-24: Fixed SQL output in non-UTF-8 locales
2005-10-21: Applied more speedup patches from Folke
2005-10-11: SQL direct connection, GUI work begins
2005-10-10: Applied speedup patches from Folke Behrens
2005-10-05: Use bulk inserts in SQL mode
2005-09-29: Converted from C# to Java
2005-08-27: Initial extraction code
Karthikeyan said,
January 29, 2008 at 2:04 pm
everything went fine except when i tried 2 change the innodb_log_file_size. it was 10mb in my my.ini file and when i changed it to 512mb, mysql simply wudn’t restart. However i tried retaining the old values and continued with the import. It imported about 9000 pages and it threw some garbage text on the screen and the command finished..any ideas on what could have gone wrong? i’m using vista with 1gb of ram in my laptop.
Prashanth said,
March 12, 2008 at 12:31 pm
thanks for the nice tutorial. everything is okay and my offiline copy of wikipedia is up and running . for people who get any io error while using mwdumper, i’d advice you to use java 1.50. I was having similar problems when I was using Java 1.42.11.. The second problem that I faced and not reported here is that after loading the dump successfully into the mysql tables, the index page of the wikipedia would not open. After this, I installed the following extensions: parserfunction, cite, wikihiero. Next, I changed the maximum execution time setting in php.ini. ( If you change the php.ini settings and the settings are not being reflected, then check if there are any other php..ini files in other folders, it was there in my case)
Tony said,
March 29, 2008 at 9:00 pm
I have a quad core processor but it started out at 153 pages/sec, peaked at 177 2000 pages later then started to fall and has continued doing so. Almost none of my processor is being used. Any ideas how to speed this process up? The command i used was:
java -classpath mwdumper.jar org.mediawiki.dumper.Dumper enwiki-latest-pages-articles.xml –format=sql:1.5 | C;\wamp\bin\mysql\mysql5.0.51a\bin\mysql -u root -p wikidb
(thats 2 hyphens preceding format)
darren said,
April 12, 2008 at 8:22 am
I’ve followed the instructions all the way up to the point where we import, i type in the command and get a 1062 error about a duplicate at key 1… I’m a total newbie at this. I have the latest java install, wikipedia dump, wamp and mediawiki… at the end of the import process nothing seems to have changed, my hard-drive has the same amount of space on it and my mediawiki is still the same as it was before the process. Any ideas on where I could be going so fantastically wrong, heh?
Tony said,
April 13, 2008 at 7:11 pm
I think you forgot to do the delete from page, etc step
darren said,
April 15, 2008 at 7:46 am
heehee! awesome! Thanks for the help
I also had a bit of a problem with parse errors, but everything seems to be working fine now!
darren said,
April 15, 2008 at 6:26 pm
hi again, I’ve been enjoying playing around with my wiki today (heh), but I have a question. I downloaded enwiki-20080312-pages-articles.xml.bz2 to my laptop. The offline wikipedia works great, I’ve been installing all the little add-ons to get it to how it looks online. So, my question related to images, were they in the dump I downloaded and imported, or will i have to download another file to get the images. If the latter, how big is the file when imported. If the former, is it a case of me just popping into wiki and manually downloading images when i need them?
Thanks for any and all help!
Tony said,
April 18, 2008 at 3:18 pm
I’m not 100% sure but I think it’s a separate file. I’m thinking its around 75GB. Don’t quote me on that though.
Michael said,
April 19, 2008 at 11:25 am
Parsing the dump went fine without any errors. However when i look at the various articles i get some normal text, but most of it is like this. Looking at for example the “space shuttle endeavour” article i get (sample):
Space Shuttle Endeavour (Orbiter Vehicle Designation: OV-105) is one of the three currently operational spacecraft in the Space Shuttle fleet of NASA, the space agency of the United States. {{
1. if: {{#if: http://www.nasa.gov/centers/kennedy/shuttleoperations/orbiters/endeavour-info.html | {{#if: Space Shuttle Overview: Endeavour (OV-105) |1}}}}
||You must specify title = and url = when using {{cite web}}.
{{#if: ||}} }}{{
1. if:
| {{#if: {{#if: | {{#if: |1}}}}
||You must specify archiveurl = and archivedate = when using {{cite web}}.
{{#if: ||}} }} }}{{#if:
| {{#if:
| [[{{{authorlink}}}|{{#if:
| {{{last}}}{{#if: | , {{{first}}} }}
| {{{author}}}
}}]]
| {{#if:
| {{{last}}}{{#if: | , {{{first}}} }}
| {{{author}}}
}}
}}
}}{{#if:
| {{#if: | ; {{{coauthors}}} }}
}}{{#if: |
{{#if:
| ({{#ifeq:{{#time:Y-m-d|{{{date}}}}}|{{{date}}}|[[{{{date}}}]]|{{{date}}}}})
| {{#if:
| {{#if:
| ({{{month}}} {{{year}}})
| ({{{year}}})
}}
}}
|}}
{{#if:
| ({{#ifeq:{{#time:Y-m-d|{{{date}}}}}|{{{date}}}|[[{{{date}}}]]|{{{date}}}}})
| {{#if:
| {{#if:
| ({{{month}}} {{{year}}})
| ({{{year}}})
}}
}}
|}}
Does anyone know perhaps if this is the result of some kind of setting? I haven’t got a clue what to do about it.
Michael said,
April 22, 2008 at 7:56 am
Never mind, it is solved.
Mathieu said,
April 28, 2008 at 1:59 pm
how did you solve it? i am having the same issue..
mat
Tony said,
May 3, 2008 at 10:55 am
Make sure you have the newest version of MediaWiki (1.12 as of writing this) and install the extensions Cite, Citation, ParserFunctions, and wikihiero. I have these installed and it works perfectly so far.
Tony said,
May 9, 2008 at 3:08 pm
Alright not quite perfect. tables dont work. anybody know what extensions fix this?
Bama Vince said,
May 16, 2008 at 5:07 pm
It work on the second try…THANKS!
Bama Vince said,
May 16, 2008 at 5:10 pm
Team, is there a way you can upload word docs into the wiki?
Desmos said,
May 25, 2008 at 3:10 pm
When I try to use mwdumper for a wikia dump I get the following error after reading several hundred pages. I am using xampp
“exception in thread main … XML document structures must start and end within the same entity …
error #1064 - You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ‘Content-Type: text/html’ at line 1
Thanks in advance
Megha said,
June 19, 2008 at 12:24 am
I want to setup my online wikipedia on my domain. Please guide me how will i do it. I have already installed XAMPP and wiki and these are working fine offline. But now i want to do the smae online. Can anyone guide me?????
Offline Wikipedia « Tinder Blog said,
November 21, 2008 at 11:49 am
[...] (the Wikipedia wiki software) can be downloaded and installed on a computer configured as an AMP server (Apache, MySQL, PHP). You can then import a Wikipedia [...]
Sagar said,
January 21, 2009 at 3:59 am
Hi…! This helped me a lot. But I want to install wikipedia on IIS server. Can anyone give me the information. Thanks in Advance.