How to Download Data Using Kaggle API, Solving OSError: Could Not Find kaggle.json.
Overview
The data on Kaggle can be substantial in size, making it somewhat risky to rely solely on a web browser’s download functionality. Therefore, an API that allows for the stable download of high-capacity data is provided, and the code at the very top of the above screenshot pertains to it.
By entering pip install kaggle in the terminal to install kaggle and then inputting
kaggle competitions download -c rsna-miccai-brain-tumor-radiogenomic-classification
the example data will be downloaded to the working directory.
Error
In truth, it won’t work.
PS D:\> kaggle competitions download -c rsna-miccai-brain-tumor-radiogenomic-classification
Traceback (most recent call last):
File "c:\users\rmsms\appdata\local\programs\python\python39\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\users\rmsms\appdata\local\programs\python\python39\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\rmsms\AppData\Local\Programs\Python\Python39\Scripts\kaggle.exe\__main__.py", line 4, in <module>
File "c:\users\rmsms\appdata\local\programs\python\python39\lib\site-packages\kaggle\__init__.py", line 23, in <module>
api.authenticate()
File "c:\users\rmsms\appdata\local\programs\python\python39\lib\site-packages\kaggle\api\kaggle_api_extended.py", line 164, in authenticate
raise IOError('Could not find {}. Make sure it\'s located in'
OSError: Could not find kaggle.json. Make sure it's located in C:\Users\rmsms\.kaggle. Or use the environment method.
Cause
To put it simply, you won’t be able to download it unless you are registered on Kaggle, and it doesn’t work because the user information is missing. The kaggle.json file containing user information cannot be installed with pip, so you need to download it directly from Kaggle and place it in a specific path.
Solution
Step 1.
Find the Account tab from https://www.kaggle.com/username/account or the user profile.
Step 2. Downloading the kaggle.json File
On that page, by searching for API, you can find a button to generate an API token as shown above. By clicking it, you can download the kaggle.json file. To avoid confusion, it is downloaded to the default download folder in this guide.
Step 3. Copying to a Specific Path
It’s crucial to run the powershell or terminal with administrator rights.
Like the above, move to the downloads folder where kaggle.json is located, and the command to copy the kaggle.json file comprises the following two lines.
cd C:\Users\rmsms\Downloads
cp kaggle.json ~/.kaggle/kaggle.json
The terminal is also opened with administrator rights because permission might be needed for copying.
Step 4. Attempt to Download
kaggle competitions download -c rsna-miccai-brain-tumor-radiogenomic-classification
By re-entering the above command, you can confirm that the data is successfully downloaded as follows.
Although it appeared to only download 12.3GB, whereas the site mentioned 127.45GB, there’s no need to be alarmed since it’s due to the high compression rate. Decompressing it will reveal the originally extensive data.
